Why do two runs of the same training give different models?

Because the landscape has many valleys and the descent depends on starting point and steps, so different runs settle in different minima.

What is a plateau in training?

A flat region of the surface where the downhill signal nearly vanishes and progress stalls until the walk escapes.

What makes a minimum wide or sharp?

Curvature: wide minima tolerate small changes cheaply, sharp ones do not, and wide ones usually generalize better.

AI Glossary / AI and machine learning / Advanced

Loss surface

Definition

A loss surface is the landscape of a model's error over all possible parameter settings: the hills and valleys training must descend, whose shape decides how hard learning will be.

What it means

Every choice of the model's parameters produces an error score, and the set of all those scores forms a landscape. Training is travel across that landscape, always downhill, hoping the valley it finds is a deep one.

For simple models the landscape is a smooth bowl with one bottom, and any sensible descent finds it. For deep networks it is a vast terrain of valleys, ridges and plateaus in millions of dimensions, and where you end depends on where you start and how you walk.

The landscape's geometry explains training's moods: plateaus are the flat regions where progress stalls for ages, sharp valleys are the minima that generalize poorly, and wide basins are the forgiving minima where small changes cost little.

Practical levers are landscape engineering: the learning rate sets step size across the terrain, momentum rolls through small bumps, and restarts explore whether a better valley exists elsewhere.

For a business, the loss surface is the honest answer to why did retraining give a different model: the landscape has many valleys, and training found a different one this time, which is normal, expected and manageable.

How it works

Accept the terrain: multiple minima are normal for complex models, and the goal is a good valley, not the valley.

Choose step sizes for the terrain: large enough to cross plateaus, small enough not to leap across valleys.

Use momentum and modern optimizers to roll through the small bumps that stop naive descent.

Restart from different initial points when the answer matters, and compare the valleys found.

Evaluate on held-out data, because the lowest point on the training landscape is not necessarily the best spot in the real world.

Three practical examples

Two training runs of the same model reaching different minima with equal validation scores, explaining a puzzling diff in behavior.
A team escaping a long plateau by raising the learning rate briefly, breaking through to a lower basin.
A research note showing the deployed model sits in a wide basin, stable under small data shifts, unlike a sharper rival.

Think of it this way

Think of hiking down an unknown mountain range in fog. You descend by feel, you reach a valley floor, and you cannot know whether the next ridge hides a deeper valley without climbing to look. Training is that hike, and the loss surface is the range.

The limit: the map is drawn from the training data. The real world's terrain differs a little, and the lowest point on your map may sit on a slope in theirs, which is why held-out data is the only compass that matters.

Worked example

two runs, two valleys

Suppose a bank retrains its triage model and gets a different model with a near-identical score. The figures are illustrative.

Predictions differ on 4 per cent of cases, and a review asks which run is right.

Validation on the newest month favors the new model slightly, but the honest finding is that both valleys are good, and the difference is landscape luck.

The team adopts a stable-seed policy for regulated retrains, so future differences are traceable to data rather than to the roll of the terrain.

Illustrative case study

A fictional trading firm obsessed over reaching the global minimum of its training loss, burning compute on ever-finer optimization of the training set.

The extra decimal places on training error bought nothing in live trading, because the minimum of the training landscape was not the minimum of the market's.

The invented lesson: the lowest valley on your map is still just your map. Optimize for the territory, measured on held-out data, not for the map's bottom.

Common misconceptions and mistakes

There is one true minimum to find. Complex models have many good minima; the search is for a good one, and identical scores from different valleys are normal.
A lower training loss means a better model. It means a deeper point in the training landscape, which often correlates with worse real-world behavior past a point.
The surface is fixed for a given model. Data, architecture and settings all reshape the terrain; the landscape belongs to the whole configuration.

Business use

Do not chase the global minimum: chase validation performance, which is the closest you can stand to the real terrain.

Keep seeds and configurations recorded, so differences between runs are explainable rather than mysterious.

Prefer wide-basin solutions when stability matters: ask how much the score moves under small data or parameter shifts.

Check your understanding

Why can the lowest point of the training loss surface be the wrong goal?

Answer: Because the surface is drawn from training data, so its deepest valley may be memorization, and the best real-world answer is found by judging candidates on held-out data instead.

Related terms

Learn next

AI for Business Owners explains how to compare two models that score the same but behave differently: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]

Sources

[S2] Google, Machine Learning Glossary. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Loss Of Situational Awareness Bias Next: Lossless compression