Can validation loss be lower than training loss?

Yes. Differences such as training-time augmentation or regularization can contribute, so check the measurement setup before assuming a bug.

Does a plateau prove the global optimum was reached?

No. It only shows little measured change over the displayed period.

Should every epoch improve the metric?

No. Stochastic training, finite validation sets and changing schedules can produce fluctuations even during useful learning.

AI Glossary / Graphs and symbolic AI / Beginner

Training curve

Definition

A training curve plots a model's measured performance against progress through training, such as optimization steps or epochs. It helps show how learning is proceeding on a fixed training dataset and is often viewed alongside a validation curve.

What it means

The horizontal axis tells you how far training has progressed; the vertical axis might show loss, error, accuracy or another task metric. The direction of improvement depends on the metric: lower loss is usually better, while higher accuracy is usually better.

An epoch is a pass through the training data under the chosen training procedure. A step commonly means one parameter update, so steps and epochs are not interchangeable when batch sizes or dataset sizes differ.

Terminology varies. Some people call an epoch-based plot a learning curve, while others reserve learning curve for performance against the amount of training data. Read the axis labels rather than assuming that every curve with the same name answers the same question.

How it works

During training, the system records metrics at specified checkpoints. A training metric reflects performance on examples used to update the model. A validation metric is measured on separate examples that are not used for those parameter updates, although validation can guide choices such as stopping time.

The curves can reveal a falling training loss, a plateau, unstable updates or a widening difference between training and validation behavior. These are clues rather than unique diagnoses. For example, a flat curve can have several causes, including an unsuitable learning rate, an implementation issue or limited model capacity.

Comparisons need consistent measurement. Training loss accumulated across changing mini-batches can differ from a validation loss evaluated once with a fixed model. Dropout, augmentation and loss normalization also affect values. A clean report explains what each point measures instead of assuming the curves are automatically comparable.

Three practical examples

A team monitors whether an image classifier's training loss is decreasing after a change to its optimization settings.
A researcher compares training and validation curves to investigate whether additional epochs are helping unseen-example performance.
An engineer notices a sudden loss jump after resuming a checkpoint and checks the learning-rate schedule and data pipeline.

Think of it this way

Think of recording practice scores after successive rehearsal sessions. The graph can show progress on the rehearsed material, while a separate assessment indicates how well the skill transfers to different material.

The analogy does not make every upward or downward fluctuation meaningful. Training measurements can be noisy, and the model's practice objective may differ from the final business outcome. A pleasing curve is evidence about the measured quantity, not a complete assessment of usefulness.

Worked example

Choosing a checkpoint from validation evidence

Consider an invented run measured at epochs 1, 2, 3, 4 and 5. Training losses are 0.90, 0.60, 0.40, 0.25 and 0.15. Validation losses are 0.95, 0.68, 0.52, 0.56 and 0.65, using the same loss definition and evaluation conditions.

Training loss improves at every checkpoint, but validation loss reaches its lowest observed value at epoch three. Under a predeclared rule of choosing the checkpoint with lowest validation loss among these checkpoints, epoch three is selected. Epoch five's lower training loss does not override that rule.

The pattern is consistent with overfitting after epoch three, but one small curve alone does not establish its cause. A separate test set is still needed for the final evaluation after selection. These values are an original illustration, not a reported training experiment.

Illustrative case study

Fictional case: Maple Vision presents a sharply falling training-loss graph as proof that its new model is better. During review, the team discovers that the new graph averages loss per image while the old graph sums loss over each batch.

It recomputes comparable metrics and adds a validation curve under a documented evaluation configuration. The team also marks a learning-rate change on the plot so a later change in slope has context. A held-out test set remains outside checkpoint selection.

The project now uses the plot to diagnose training and guide model selection within the agreed process. It does not claim that a smoother line, by itself, demonstrates better accuracy or business performance. The case is fictional.

Common misconceptions and mistakes

Falling training loss proves generalization. It shows improvement on the measured training objective; separate evaluation is needed.
A smooth curve means a stable underlying process. Smoothing can conceal spikes or variation.
The same epoch count makes two runs equivalent. Dataset size, batch size, update rules and compute per step can differ substantially.

Business use

Training curves make model-development decisions easier to audit when they include clear units, metric definitions and configuration changes. They help teams identify unproductive training, compare candidate settings and investigate unexpected behavior.

Keep raw measurements as well as any smoothed display. Label training, validation and test data distinctly. Compare quality against compute or elapsed time when cost matters, and keep the final test evaluation separate from repeated tuning. A curve supports a decision only when the underlying measurement answers the relevant question.

Check your understanding

Training loss falls for ten epochs, but validation loss begins rising after epoch six. Should the final epoch automatically be selected?

Answer: No. Selection should follow the chosen validation rule and task metric. The training curve alone does not establish that the last checkpoint generalizes best.

Related terms

Learn next

Read validation set and early stopping. Sketch the worked example with both curves, then label the selected checkpoint and explain why a separate test set is still required after making that selection.

Sources

Russell and Norvig, author-hosted statistical-learning chapter distinguishing training progress from training-set-size learning curves: https://aima.cs.berkeley.edu/newchap20.pdf

Lisa Zhang, University of Toronto training lecture, metric logging and training curves: https://www.cs.toronto.edu/~lczhang/360/lec/w02/training.html

Relevant source passages were read. The five-epoch values and fictional case are original teaching examples. They do not reproduce an actual model run or establish a universal diagnosis from a curve shape.

Previous: Training Next: Training error