What does momentum do in training?

It carries a running average of past gradients into each update, smoothing the path and speeding convergence.

What is the typical momentum value?

Around 0.9, meaning ninety per cent of the previous velocity survives into each step.

What problem does it solve?

Zigzagging progress across valleys and stalling on shallow slopes, both common on real loss surfaces.

Yes: too much, or paired with too large a learning rate, overshoots minima and can make training diverge.

Is momentum used outside deep learning?

Rarely by name, but the same running-average idea appears in exponential smoothing and other forecasting workhorses.

AI Glossary / AI and machine learning / Intermediate

Momentum

Definition

Momentum is a training technique that lets an optimizer keep moving in its recent direction: each update blends the new gradient with a running average of past ones, smoothing the path and speeding progress.

What it means

Gradient descent follows the local slope, which on real loss landscapes is a bumpy, zigzagging route. Momentum adds inertia: consistent directions build speed, contradictory slopes cancel, and the optimizer rolls instead of jittering.

The effect is easiest to see in valleys where the gradient points mostly sideways, oscillating across the valley while drifting slowly along it. Momentum dampens the oscillation and amplifies the drift, cutting training time.

The technique carries one dial, the momentum coefficient, commonly 0.9: how much of the past velocity survives each step. Higher means more inertia, smoother paths and more overshoot risk.

Momentum also helps past small bumps and shallow dips, rolling over wrinkles that would trap a memoryless optimizer, which is part of why it became the default in deep learning.

For a business, momentum is one of the settings behind training claims: stable loss curves, fast convergence and occasional overshoot are all momentum stories, and the setting belongs in every run record.

The same averaging idea returns inside adaptive optimizers, which keep running averages of gradients and their squares; momentum is the ancestor of the family.

How it works

Start with the common default coefficient of 0.9 and a sound learning rate, since the two dials interact strongly.

Watch the loss curve for momentum's signature problems: oscillation suggests too much, slow crawl suggests too little.

Tune the coefficient together with the learning rate, because raising one usually calls for adjusting the other.

Use modern variants like Nesterov momentum when available, which looks ahead before committing to the direction.

Record the setting with the run: convergence behavior cannot be reproduced or diagnosed without it.

Three practical examples

A training run whose zigzag loss curve smooths visibly once momentum rises from zero to 0.9, converging in half the epochs.
A model that overshoots and oscillates after momentum is pushed to 0.99, calming when the learning rate is cut to match.
A fine-tuning job where momentum helps escape a shallow early dip that had trapped a momentum-free run at a worse result.

Think of it this way

Think of a heavy ball rolling down a bumpy hillside: it builds speed on consistent slopes, ploughs through small hollows, and ignores the jitter of the ground, reaching the bottom faster than a marble that reacts to every pebble.

Worked example

the zigzag that vanished

Suppose a team trains a demand model and the loss curve oscillates sharply, halving painfully slowly over a hundred epochs. The figures are illustrative.

The gradient is fighting a valley: steep across, shallow along, so raw descent zigzags and makes little forward progress each step.

Adding momentum at 0.9 cancels the sideways zigzag and compounds the forward drift, and the same model converges in forty epochs.

The diagnosis generalizes: oscillating loss curves are often a momentum problem, and the fix is a setting, not a bigger model or more data.

Illustrative case study

A fictional platform team raised momentum to 0.98 chasing faster training, without touching the learning rate, and the run's loss began rising after initially falling.

The optimizer was overshooting the minimum each step, building speed in directions that were already wrong, and the divergence looked like a data bug for a week.

The invented lesson: momentum amplifies whatever it is given, including a bad step size. Tune the pair together and read divergence as a settings symptom first.

Common misconceptions and mistakes

Momentum is just a speed boost. It is a direction stabilizer: it amplifies consistent progress and cancels oscillation, which is why it helps.
More momentum is always smoother. Past a point it overshoots, and the run oscillates or diverges instead of gliding.
It is obsolete now. Modern optimizers build on it: momentum remains the moving average at the heart of the most-used update rules.

Business use

Treat momentum and learning rate as one coupled pair in any tuning exercise.

Read oscillating loss curves as a possible momentum deficit and rising late curves as a possible excess.

Record the coefficient in every run log alongside the learning rate; convergence behavior is irreproducible without it.

Expect momentum defaults to transfer poorly across very different models or data, and re-tune rather than assume.

When a vendor demo shows suspiciously fast convergence, ask for the momentum and learning-rate settings before crediting the model.

Check your understanding

Why does momentum pair so strongly with the learning rate?

Answer: Because momentum multiplies the effective step size along consistent directions, so changing either dial changes the real distance traveled per update, and tuning one without the other mis-sets the pair.

Related terms

Learn next

Prompt Like a Pro explains how to read the training settings inside model claims: https://aiforbusinessandlife.com/academy/prompt-like-a-pro-clear-framework. [C3]

Sources

[S2] Google, Machine Learning Glossary; [S12] Aalto University, Dictionary of Machine Learning. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Moment generating function Next: Monitoring as a service