Does preconditioning require a full Hessian?

No. Methods can use diagonal, low-rank, factored, or other structured information.

Does positive definiteness guarantee every SGD step improves the loss?

No. It ensures a descent direction for the exact gradient, while noisy gradients and overly large finite steps can still increase loss.

Is feature standardization the same thing?

It can change optimization geometry, but explicit gradient preconditioning is a broader parameter-update operation and need not be equivalent to standardizing input features.

AI Glossary / Probabilistic machine learning / Intermediate

Preconditioned stochastic gradient descent

Definition

Preconditioned stochastic gradient descent modifies a stochastic gradient with a matrix before using it to update model parameters. The preconditioner changes the relative scaling, and potentially the direction, of parameter adjustments. The aim is to make progress less uneven across directions while accounting for computational cost and noisy gradient information.

What it means

A training objective can be steep in one parameter direction and shallow in another. Ordinary stochastic gradient descent uses a common learning-rate multiplier, so a step small enough for the steep direction may move slowly along the shallow one.
A preconditioner can reshape the update to address this imbalance. The term describes a general approach, while particular algorithms differ in how they estimate, represent, and update the matrix. It should not be read as one universally defined optimizer with a single set of guarantees or software defaults.

How it works

A common update is theta_new = theta_old - eta P g_hat, where g_hat is a stochastic gradient, eta is a learning rate, and P is the preconditioning matrix. A diagonal P rescales coordinates separately; a full matrix can account for interactions between directions. Structured approximations trade flexibility against memory and computation.
With the exact nonzero gradient g and a symmetric positive-definite P, the direction -Pg has negative directional derivative because g^T(-Pg) < 0. This is a local directional statement, not a guarantee that an arbitrary finite step lowers the objective.
A stochastic gradient can point the wrong way because of noise, so positivity of P does not guarantee descent on every stochastic step. Curvature-inspired preconditioners also need safeguards: an inverse Hessian can be unsuitable in nonconvex regions or amplify noise in weak-curvature directions. The preconditioner, learning rate, and gradient estimator must be considered together.

Three practical examples

Unevenly scaled regression: compare a preconditioned update with ordinary SGD when parameter directions have markedly different sensitivities.
Large-model training research: use a structured matrix approximation to improve update geometry without storing a dense parameter-by-parameter matrix.
Noisy optimization: study whether a preconditioner improves progress while controlling the way stochastic-gradient noise is amplified.

Think of it this way

Think of adjusting the sensitivity of separate steering controls. If one control reacts dramatically and another barely moves the system, rescaling them can make navigation more balanced. The analogy is limited: a preconditioner can mix directions mathematically, and noisy measurements can still suggest a bad move. Balanced controls do not eliminate uncertain information.

Worked example

Consider an original deterministic illustration of the geometry: f(x, y) = (100x^2 + y^2)/2. At (1, 1), the gradient is (100, 1) and the objective is 50.5.
An ordinary step with eta = 0.01 gives (0, 0.99), whose objective is 0.99^2/2 = 0.49005. The first coordinate reaches zero quickly, while the second moves only slightly.
Now use P = diag(0.01, 1). The preconditioned gradient is (1, 1). With eta = 0.5, the next point is (0.5, 0.5), and the objective is (25 + 0.25)/2 = 12.625. Repeating gives (0.25, 0.25), with objective 3.15625. Both coordinates shrink by the same factor per step. This comparison illustrates geometry, not a universal one-step advantage or a fair tuned benchmark.
For a noise example, at the optimum the true gradient is zero, but a noisy estimate (0, 2) produces an update (0, -1) with the same P and eta. The objective rises from 0 to 0.5. A positive-definite preconditioner has not prevented a noisy uphill step.

Illustrative case study

Fictional case: Silverline Forecasting trains a model whose coefficients settle at very different speeds. The team proposes a preconditioner and compares runs using a common data split and a stated tuning budget.
It records elapsed time, memory use, objective values, and evaluation performance, because fewer iterations can be offset by more expensive updates. It also tests sensitivity to gradient noise rather than assuming curvature correction solves every training problem. The proposed method is retained only if the measured tradeoff supports the application. This is an invented review scenario, not a report of observed savings or a guaranteed training improvement.

Common misconceptions and mistakes

Preconditioning is not identical to reducing the learning rate. A scalar rate changes all update components together; a nontrivial matrix changes their relative geometry. It is also different from adding a regularization penalty to the objective, although both may influence training behavior.
Not every preconditioner is an exact inverse Hessian. Nor does every matrix called a preconditioner preserve a descent direction. Even suitable positive-definite matrices need appropriate step sizes and careful treatment of stochastic noise.

Business use

For business training workloads, judge preconditioning by time to a useful validated result, stability, and resource demand. Report the cost of estimating and applying the matrix, not just the number of optimizer steps.
A useful technical review asks what information defines the preconditioner, how singular or extreme scales are controlled, and whether comparisons include tuning effort. Ensure that the final model meets evaluation requirements; better optimization geometry alone does not guarantee better predictions on future data.

Check your understanding

A diagonal preconditioner has entries 0.1 and 2, and the stochastic gradient is (4, 3). What vector enters the update before multiplying by the learning rate?

Answer: P g_hat = (0.4, 6). The two components are rescaled differently.

Related terms

Stochastic gradient descent
Hessian
Learning rate
Condition number
preconditioner
curvature
positive-definite matrix
natural gradient
gradient noise

Learn next

Work through a two-dimensional quadratic with and without diagonal scaling. Separate a descent-direction calculation from a finite-step loss calculation. Then study a particular preconditioner estimation method, including its assumptions, matrix representation, and behavior under noisy gradients.

Sources

Xi-Lin Li, Preconditioned Stochastic Gradient Descent, original paper, update definition and preconditioner analysis: https://arxiv.org/pdf/1512.04202 Xi-Lin Li, Preconditioner on Matrix Lie Group for SGD, original research record: https://arxiv.org/abs/1809.10232 Li Section III Equation 11, curvature and noise-amplification discussion were independently consulted.

Previous: Precondition axioms Next: Preconditioning