Must every checkpoint be saved?

No. An arithmetic mean can be maintained recursively with a count and an additional parameter vector.

Does averaging always beat the last iterate?

No. The worked counterexample shows otherwise, and theoretical results have specific assumptions and limiting or expected-performance interpretations.

Is averaging model parameters identical to ensembling predictions?

Not generally. A nonlinear model evaluated at average parameters need not equal the average of its predictions at separate parameter settings.

AI Glossary / Probabilistic machine learning / Intermediate

Polyak-Ruppert averaging

Definition

Polyak-Ruppert averaging is a stochastic-approximation technique that averages successive parameter estimates rather than returning only the last estimate. Used with an appropriate learning-rate schedule and assumptions, this averaging can improve the statistical behavior of noisy iterative estimation. Its guarantees depend on the problem, noise, and update conditions.

What it means

A stochastic optimizer may keep moving around a useful solution because each update uses incomplete or noisy information. The final point can therefore reflect the last few random disturbances. Combining many iterates offers another estimate of the underlying solution.
The central object being averaged is the parameter vector. This is different from forming a larger minibatch of gradients before one update, and different from averaging predictions made by several models. Those operations can sometimes be related in particular settings, but they are not interchangeable definitions.

How it works

Start with a stochastic approximation sequence, for example theta_t = theta_(t-1) - eta_t g_t, where g_t estimates the objective's gradient. Maintain a separate arithmetic mean of the selected iterates. For t equally weighted included iterates, the mean can be updated as average_t = average_(t-1) + (theta_t - average_(t-1))/t.
The optimizer continues updating its ordinary iterate; the running average supplies a candidate final estimate. Only the current iterate, the average, and an inclusion count need be retained for this basic construction.
Classical efficiency results require conditions such as suitable local behavior of the estimation problem, controlled noise, and an appropriate decreasing step-size schedule. Averaging does not repair arbitrary biased gradients or unstable updates. Averaging only after an initial period is a related practical choice, but the inclusion rule must be stated because it changes the estimator.

Three practical examples

Streaming regression: retain an average of coefficient iterates produced while learning from successive observations, then evaluate that estimate separately from the final coefficients.
Noisy numerical estimation: smooth a sequence of stochastic root estimates when the conditions for the approximation procedure are appropriate.
Optimizer comparison: compare final-iterate and averaged-iterate performance under the same data order and computation budget, reporting repeated runs rather than one favorable sequence.

Think of it this way

Think of recording the position of a pointer that jitters around a target. Averaging positions can reduce the influence of the last jolt. The analogy has limits: optimizer positions are dependent, their bias can change over time, and the target may move if the data distribution changes. An average of positions far from the target need not be useful.

Worked example

Consider an original scalar sequence of four included estimates: 1.8, 2.4, 1.9, and 2.3. Their sum is 8.4, so the arithmetic average is 8.4/4 = 2.1. If a toy problem's known solution is 2, the average's absolute error is 0.1, compared with 0.3 for the final iterate 2.3.
Now include a fifth estimate of 1.6. The recursive update gives 2.1 + (1.6 - 2.1)/5 = 2.0. Direct averaging gives (8.4 + 1.6)/5 = 2.0 as well. This verifies the running-average calculation without storing all five values.
This sequence is illustrative, not a convergence experiment. For a counterexample to guaranteed finite-sample improvement, estimates 5 and 2 have average 3.5: when the target is 2, the last estimate is exact but the average is not. The quality of an average depends on which iterates it includes, their errors, and the underlying procedure.

Illustrative case study

Fictional case: Lakehouse Analytics fits a simple convex prediction model from a stream of simulated observations. A developer proposes returning the averaged coefficients because the final coefficients vary across runs. The team records both outputs, uses an identical learning-rate schedule, and checks them on untouched simulated evaluation data.
It also compares averaging from the beginning with a clearly recorded later start. An early transient can pull the full-history average away from the final region. The team treats these as experimental choices and reports the observed tradeoffs, without claiming that every neural network or changing data stream will behave similarly. This scenario is invented and contains no measured business result.

Common misconceptions and mistakes

Polyak-Ruppert averaging is not synonymous with momentum. Momentum accumulates a direction used to change subsequent iterates; basic iterate averaging can be maintained separately as an output estimate.
It is also not the same as a fixed-decay exponential moving average, often called a Polyak update in other contexts. Equal arithmetic weights and exponentially decaying weights emphasize history differently. Finally, dependent optimizer iterates cannot automatically be treated as independent observations when calculating the average's uncertainty.

Business use

A business team can use iterate averaging as a relatively inexpensive candidate improvement in an appropriate training workflow. The extra storage is roughly one parameter-sized average plus a counter, rather than an archive of every checkpoint.
The practical decision should include prediction quality, stability across runs, learning-rate selection, and the effect of stale history. In an evolving environment, a full-history average may respond too slowly to new conditions. Record the averaging start and weighting rule in model documentation so that the deployed parameters can be reproduced.

Check your understanding

After averaging four estimates to obtain 2.1, what weight does a fifth estimate receive in the updated arithmetic mean?

Answer: One fifth. The previous average receives weight four fifths, because it represents four equally weighted estimates.

Related terms

Stochastic gradient descent
Momentum
Stochastic approximation
learning-rate schedule
iterate averaging
tail averaging
exponential moving average
asymptotic efficiency

Learn next

Implement the running mean on a short scalar sequence and compare it with direct averaging. Then distinguish parameter averaging from gradient averaging and prediction ensembling. Study the assumptions of a stochastic-approximation result before transferring its guarantees to a new loss, noise process, or neural-network training method.

Neural network

Sources

Polyak and Juditsky, Acceleration of Stochastic Approximation by Averaging, original publication record: https://epubs.siam.org/doi/abs/10.1137/0330046 Bach and Moulines, Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Machine Learning, original paper: https://proceedings.neurips.cc/paper_files/paper/2011/file/40008b9a5380fcacce3976bf7c08af5b-Paper.pdf Bach and Moulines introduction and problem setup were independently consulted for iterate averaging and conditional noise and smoothness assumptions.

Previous: Pólya urn model Next: Polyglot persistence