Is step size always called learning rate?

In gradient-based machine learning it often is, but optimization uses the term more broadly.

Can the setting change during training?

Yes. Schedules and adaptive procedures deliberately change the effective updates.

Does a lower training loss guarantee a better product?

No. Generalization and the business task must be assessed separately.

AI Glossary / AI and machine learning / Intermediate

Step size

Definition

In optimization, step size controls how far an update moves along a chosen search direction. In basic gradient descent, the learning rate is the multiplier that scales the gradient to produce the parameter update.

What it means

Training adjusts a model's parameters to reduce a loss, which measures how poorly the model is doing on the training objective. Knowing a promising direction is only part of the problem. The training process must also decide how large a move to make.

A large step can pass over a useful region or make the process unstable. A very small step can make progress unnecessarily slow. The appropriate setting depends on the objective, the scale of the inputs and the optimizer.

Step size and learning rate are often used interchangeably in introductory machine learning. More precisely, the actual distance moved also depends on the gradient or other search direction. A fixed learning rate does not guarantee equally sized movements.

For a manager, this is a reminder that training speed and quality depend on configuration, not only on buying more computing power or choosing a larger model.

How it works

In a basic gradient-descent update, the new parameter equals the old parameter minus the learning rate multiplied by the gradient. The gradient supplies a local direction of increasing loss, so subtracting it aims to move downhill.

The multiplier may stay fixed or follow a schedule. Some training plans start cautiously, increase the rate during a warm-up, then reduce it later. Other methods adapt updates using information accumulated from earlier gradients.

Adaptive optimizers and momentum make the relationship between one named learning-rate setting and the final movement more complex. Comparing raw learning-rate numbers across different optimizers can therefore be misleading.

Teams inspect learning curves and independent validation results while tuning. A decreasing training loss shows progress on the training objective; it does not by itself prove that the model is becoming more useful on unseen examples.

Three practical examples

Forecast training: an analyst reduces the learning rate after observing unstable loss values and checks whether convergence improves.
Fine-tuning: a team uses cautious updates when adapting an existing model, then evaluates whether important prior capabilities have been retained.
Optimizer comparison: engineers compare completed runs at matched quality rather than assume identical learning-rate numbers mean identical update behavior.

Think of it this way

Think of walking downhill in fog with a device that estimates the slope under your feet. The slope suggests a direction, but the length of your stride determines whether you move cautiously or overshoot.

The analogy has limits. A training landscape can have many dimensions, noisy estimates and complex geometry. There is no universal stride length that works equally well everywhere.

Worked example

Calculating one simple parameter update

Suppose an invented model has a parameter value of 5 and a gradient of 2 at the current point. With a learning rate of 0.1, the update subtracts 0.1 multiplied by 2, so the new value is 4.8.

With a learning rate of 0.01, the same gradient produces a new value of 4.98. The direction is the same, but the movement is ten times smaller.

Now keep the learning rate at 0.1 and change the gradient to 0.2. The update moves by 0.02 rather than 0.2. This shows why the learning-rate setting is not itself the literal distance moved.

These calculations do not tell us which setting is best. That depends on the loss surface and the behavior of later updates. They illustrate the mechanics of a basic update, without momentum or adaptive scaling.

Illustrative case study

A fictional engineering consultancy fine-tunes a classifier using a setting copied from a different project. The training loss oscillates and occasionally jumps sharply.

The team checks data scaling and the optimizer configuration, then compares a small set of controlled learning-rate choices. It records both training behavior and performance on held-out examples rather than selecting whichever run finishes fastest.

The invented lesson is that a familiar configuration is not automatically transferable. The size of an update depends on the surrounding problem, and troubleshooting should examine that context.

Common misconceptions and mistakes

Larger steps always train faster. They may cause instability or require recovery and retraining.
A fixed learning rate means fixed movement. The gradient and optimizer also affect the update.
Tiny steps guarantee the best solution. They can waste resources or make progress impractically slow without resolving other modeling problems.

Business use

Most business users do not need to tune step sizes directly. The concept is useful when reviewing training plans, unexplained instability or claims that longer training will automatically solve a quality problem.

Ask for recorded configurations, learning curves and a clear stopping criterion. Compare runs using quality, elapsed time and total computing cost, while keeping the evaluation data and task definition consistent.

Check your understanding

If the learning rate stays at 0.1 while the gradient falls from 2 to 0.2, what happens to the magnitude of a basic update?

Answer: It falls from 0.2 to 0.02. The multiplier is unchanged, but the update becomes ten times smaller because the gradient is smaller.

Related terms

Learn next

Read learning rate and loss curve. Compare training and validation curves from the same run before diagnosing whether the problem is slow optimization or poor generalization.

Sources

Google, Linear regression: Hyperparameters: https://developers.google.com/machine-learning/crash-course/linear-regression/hyperparameters. Checked 12 October 2026. The numerical updates and consultancy scenario are original teaching examples.

Previous: Step cost Next: Stepped-wedge trial