What does an optimizer actually do?

It repeatedly adjusts the model's parameters in the direction that reduces error, using the current errors to pick direction and step size.

What is the learning rate?

The size of each adjustment step: the single most influential training setting, balancing speed against stability.

Which optimizer should I use?

For most modern work, Adam or a close variant is the safe default; simpler optimizers appear when fine control or theory matters.

Why do two identical training runs differ?

Random starting points and data ordering mean the search takes different paths and can settle in different good valleys.

AI Glossary / AI and machine learning / Beginner

Optimizer

Definition

An optimizer is the procedure that adjusts a model's parameters during training to reduce its errors, step by step, until the model fits the data as well as it can.

What it means

Training a model is searching for the best settings of its internal dials, and the optimizer is the search strategy. It looks at how wrong the model currently is, decides which direction to turn each dial, and takes a step of a chosen size, repeating this thousands or millions of times.

The step size is the learning rate, and it is the optimizer's most consequential setting: too large and the search leaps over the best answers; too small and training crawls or stalls. Modern optimizers like Adam adjust effective step sizes automatically per parameter, which is why they are the default in most deep learning.

The optimizer explains why training is iterative rather than instant, why two runs of the same training can finish at different quality, and why a learning rate appears in every project plan.

For a business reader, the optimizer is rarely a personal choice, but it drives training time, compute cost and reproducibility, all of which are budget items.

How it works

Start from an initial guess for the parameters, often random, and measure the model's error on a batch of data.

Compute which direction to adjust each parameter to reduce the error, using the gradient of the loss.

Take a step in that direction, sized by the learning rate, then repeat until the error stops improving.

Schedule the learning rate to shrink as training progresses, so early steps explore and late steps settle.

Log the optimizer name, learning rate and schedule with every run, so results can be reproduced and compared.

Three practical examples

A vision model's training curve falls steeply for three days then flattens; the team switches to a schedule that shrinks the learning rate and squeezes out the final points of accuracy.
A prototype trained with a plain optimizer needs careful tuning, while the same model with Adam works on the first attempt, halving experimentation time.
A training run diverges, errors growing instead of shrinking, and the post-mortem traces it to a learning rate set ten times too high.

Think of it this way

Think of descending a hill in fog to find the valley floor. You feel the slope underfoot, step downhill, and repeat. The optimizer is how you choose each step's direction and length; the learning rate is your stride.

Worked example

the overshooting stride

Suppose a team trains a pricing model and the error bounces up and down instead of settling. The figures are illustrative.

The diagnosis is a learning rate too large: each step overshoots the best region, landing on the opposite slope, so the model oscillates rather than converging.

The fix halves the learning rate, and the error curve smooths and settles, at the cost of more steps to reach the same depth.

The general lesson: the optimizer's stride trades speed against stability, and a bouncing error curve is the classic signature of a stride too long.

Illustrative case study

A retail analytics team once budgeted two weeks to improve a demand model and spent nine days fighting training instability: runs diverged, results varied, and nothing reproduced. A consultant replaced their hand-tuned optimizer with Adam, set a schedule that shrank the learning rate as training progressed, and the model converged reliably on the first full run. The remaining days went into validating data instead. The team's takeaway entered their playbook: when training misbehaves, suspect the optimizer settings before suspecting the data, and always log the learning rate schedule alongside the model version.

Common misconceptions and mistakes

The optimizer finds the single best solution. It finds a good solution it can reach; models have many good valleys, and which one you reach depends on where the search started.
A fancier optimizer always trains a better model. It usually trains faster and more reliably, but final quality still depends on data, model shape and settings.
Optimizer choice is a research detail with no business meaning. It directly drives training time, compute cost and how reproducible results are.

Business use

Ask how long training takes and what drives it; optimizer and learning-rate choices are usually the honest answer.

Require the learning rate schedule and optimizer name in every experiment log, so results can be reproduced.

Treat diverging or oscillating training runs as a settings problem first, before assuming the data or model is broken.

Let the team use modern defaults like Adam unless they can articulate why a simpler optimizer suits the problem.

Expect training curves in project reviews: a smooth falling curve is health, a bouncing one is a settings question, and a flat one is a stall.

Check your understanding

What is the optimizer's job, and which setting dominates its behavior?

Answer: Its job is to search the space of parameter settings for ones that make the model's errors small, by repeatedly stepping in the direction that reduces error. The dominant setting is the learning rate, the step size: too large and the search overshoots and oscillates, too small and it stalls, so much of training practice is choosing and scheduling it well.

Related terms

Learn next

Be Productive Using AI builds the habit of asking how a model was trained before trusting its outputs: https://aiforbusinessandlife.com/academy/be-productive-using-ai [C1]

Sources

[S2] Google, Machine Learning Glossary; [S12] Aalto University, Dictionary of Machine Learning. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Optimization problem Next: Optimizer's curse