What happens if the learning rate is too high?

Updates overshoot the minimum, so training error oscillates or diverges instead of settling.

Training crawls: each update is nearly nothing, so convergence takes far longer and may stall short of a good minimum.

What is a learning-rate schedule?

A plan that varies the rate during training, typically larger early and smaller late, to combine fast approach with fine settling.

AI Glossary / AI and machine learning / Intermediate

Learning rate

Definition

Learning rate is the step-size dial of iterative training: how large an adjustment the model makes to its parameters at each update, trading speed of learning against stability.

What it means

Training adjusts the model in thousands of small steps, each nudging the parameters downhill against the error. The learning rate decides the size of those nudges, and it is the single most consequential dial in iterative training.

Too small and training crawls: thousands of updates make barely a difference, and the run wastes budget without reaching the valley. Too large and training overshoots: each step jumps past the bottom, and the error oscillates or diverges instead of settling.

The right value is found empirically, usually on a logarithmic ladder, and modern practice often varies it during training: larger early, when far from the goal, smaller late, when fine positioning matters.

The dial interacts with everything: batch size, model scale and the shape of the loss all move the good range, so a learning rate transplanted from another project is a guess until validated on yours.

For a business, the learning rate is the reason some training runs fail mysteriously: the model, the data and the code can all be right while the step size silently makes learning impossible, which is why it is the first dial checked.

How it works

Start from the toolkit's default or a published value for a similar setup, treated as a guess to be tested.

Run short trials across a log ladder of rates, watching training error for smooth descent, crawling or oscillation.

Pick the largest rate that descends smoothly, since that trains fastest without instability.

Consider a schedule that shrinks the rate across training, and tune its shape on validation.

Re-check the dial when batch size, architecture or data scale change materially, because the good range moves with them.

Three practical examples

A team whose training loss oscillates wildly until the rate drops a factor of ten, after which it descends smoothly to a better minimum.
A fine-tuning run for a large language model using a rate one hundredth of pre-training's, because small steps protect what was already learned.
A training dashboard where the loss curve's shape, crawling, smooth or jagged, is read as the learning rate's report card.

Think of it this way

Think of descending a mountain in fog with a compass. Small steps get you down slowly and safely; giant strides are faster until one carries you clean over the path and into the rocks. The learning rate is the stride length, and the mountain never warns you which it will punish.

The limit: the right stride changes with the slope. A rate that was perfect on the steep upper mountain is reckless on the flats near the bottom, which is why schedules shrink the stride as training approaches its goal.

Worked example

the oscillating run

Suppose a retailer trains a demand network and the training loss bounces between 0.4 and 0.9 for two days. The figures are illustrative.

The pattern is the classic overshoot signature: each update jumps across the valley, and a rate reduction from 0.01 to 0.001 turns the jagged line into a smooth descent within the hour.

The retrained model reaches a 12 per cent better minimum in a third of the compute, because the first run was never learning, only vibrating.

The team's runbook gains a first commandment: read the loss curve's shape before touching anything else.

Illustrative case study

A fictional bank's vendor delivered a model that could not be retrained successfully by the bank's own team, and months of suspicion about hidden magic ended when someone compared configuration files.

The vendor's learning rate was a fifth of the toolkit default, a value found by patient search, and every internal attempt had used the default and diverged.

The invented lesson: the learning rate is part of the model. Hand over the dial setting with the weights, or you have delivered half a product.

Common misconceptions and mistakes

Bigger rate means faster learning. Up to a point; past it, updates overshoot, training destabilises and effective progress falls to zero.
There is a best universal value. The good range depends on the model, batch size, data and loss shape, so it is found per project, not memorized.
If training converged, the rate was fine. Convergence is the floor, not the ceiling: a better-tuned rate often reaches a better minimum for less compute.

Business use

Treat learning rate as a first-class hyper-parameter with its own search in every training effort.

Read loss-curve shape as the diagnostic: crawl, smooth descent and oscillation each name a different dial problem.

Record the rate and schedule in the model file, because retraining without them is archaeology.

Check your understanding

Why is the learning rate usually the first dial checked when training misbehaves?

Answer: Because it controls every update, so its failure modes, crawling, oscillation, divergence, produce the most common training pathologies before any other cause is considered.

Related terms

Learn next

AI for Business Owners explains which training diagnostics deserve a question when a vendor or team reports results: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]

Sources

[S1] Russell and Norvig, AIMA glossary; [S2] Google, Machine Learning Glossary; [S8] Jurafsky and Martin, Speech and Language Processing (3rd ed. draft); [S12] Aalto University, Dictionary of Machine Learning. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Learning element Next: Learning rule