Is weight decay identical to L2 regularization?

For standard gradient descent they can be equivalent after matching coefficients; for adaptive methods the equivalence generally fails.

Does it make weights exactly zero?

Ordinary multiplicative shrinkage need not produce exact zeros, so it is not equivalent to sparse pruning.

Can a weight increase while decay is enabled?

Yes. The data-driven update can outweigh the shrinkage contribution.

AI Glossary / Graphs and symbolic AI / Intermediate

Weight decay

Definition

Weight decay is a training technique that repeatedly shrinks selected model weights toward zero. It is used as a form of regularization, influencing the solutions a learning algorithm reaches. With ordinary gradient descent, a suitably scaled squared-weight penalty produces the same update. With adaptive optimizers, applying that penalty through the gradient is generally different from applying decay separately.

What it means

A model's weights control how inputs influence its outputs. Weight decay adds a systematic tendency toward smaller weight magnitudes while learning also moves the weights to fit data. These two effects act together; an individual weight can still grow when the data-driven update outweighs the shrinkage.
Terminology requires care. Some descriptions use weight decay as a synonym for L2 regularization because of their equivalence under standard gradient descent. Decoupled weight decay specifically separates shrinkage from the optimizer's processing of the loss gradient. Parameter names alone do not establish which update a software implementation performs.

How it works

For a simple update, write w_new = (1 - d)w - eta*g, where d is the per-step decay fraction, eta is the learning rate, and g is the data-loss gradient. Under ordinary gradient descent, adding lambda/2 times the squared weight norm to the loss gives the same update when d = eta*lambda.
Adaptive optimizers can scale gradient coordinates differently. Including the penalty gradient inside that processing changes its effect, whereas decoupled decay acts directly on the weights. AdamW implements the decoupled approach. The exact coefficient convention must be checked before comparing settings across implementations.

Three practical examples

A training experiment can compare several decay strengths while holding the data split and evaluation procedure fixed.
A model-development team can inspect optimizer equations before transferring a regularization setting from plain gradient descent to an adaptive optimizer.
A classroom exercise can separate the shrinkage contribution from the data-gradient contribution in one weight update.

Think of it this way

Think of a spring gently pulling a control knob toward its neutral position while another force adjusts it to improve performance. The final movement reflects both influences. The analogy is limited because model weights interact, different parameterizations change their interpretation, and smaller weights do not automatically mean a more useful or trustworthy model.

Worked example

Consider an original two-weight example with w = (2, -1), data gradient g = (0.5, -0.2), learning rate eta = 0.1, and per-step decay d = 0.01.
Shrinkage alone gives 0.99*w = (1.98, -0.99). Subtracting eta*g then gives (1.93, -0.97). Without decay, the same gradient step would produce (1.95, -0.98).
For ordinary gradient descent, choose lambda = d/eta = 0.1. The penalty gradient is lambda*w = (0.2, -0.1). Add this to the data gradient to obtain (0.7, -0.3), then subtract 0.1 times that vector from the original weights. The result is again (1.93, -0.97), demonstrating the stated equivalence.
Now introduce a deliberately simple coordinate scaling M = diag(1, 10). This is a toy preconditioner, not a full simulation of Adam. Processing the combined gradient through M gives (0.7, -3), so the update becomes (1.93, -0.7). Separately decaying the weights and processing only the data gradient instead gives (1.98, -0.99) - (0.05, -0.2) = (1.93, -0.79). The second coordinates differ.
Finally, with zero data gradient and the original one-percent decay, two steps turn a weight of 2 into 2*0.99*0.99 = 1.9602. Repeated decay is multiplicative; it does not subtract the same fixed amount each time.

Illustrative case study

Fictional case: an image-classification team changes optimizers and copies its old regularization coefficient into a setting labeled weight decay. Validation behavior changes unexpectedly.
The team checks whether the old setting added a penalty to the loss gradient and whether the new setting applies separate shrinkage. It records coefficient conventions, learning-rate schedules, and which parameters receive decay before running comparable experiments. The lesson is about interpreting an experiment correctly, not a claim that one optimizer must win on every dataset.

Common misconceptions and mistakes

Do not assume L2 penalties and decoupled decay are interchangeable for every optimizer. Do not interpret decay as deleting a fixed percentage of connections: shrinking weights is different from pruning them.
Also avoid claiming that every weight must decrease at every step. For example, a sufficiently negative data gradient can increase a positive weight despite decay. Excessive shrinkage can impair fitting, and favorable training behavior does not guarantee improvement on unseen data. Learning-rate and update-count differences can change the cumulative effect of a setting.

Business use

Weight decay is useful for designing and interpreting model-training experiments. Its practical value should be assessed through appropriate validation rather than through weight size alone.
Record the optimizer, update convention, decay coefficient, training duration, learning-rate schedule, and parameter groups. Compare experiments on meaningful task metrics. When transferring a configuration, inspect whether decay is multiplied by the learning rate and whether some parameters are excluded. These details make reported results easier to reproduce and help explain changes that would otherwise look mysterious.

Check your understanding

With eta = 0.1 and per-step decay d = 0.01, what L2 coefficient matches the plain gradient-descent update?

Answer: Lambda = d/eta = 0.1, when the penalty is lambda/2 times the squared weight norm.

Related terms

Learn next

Repeat the first update with d = 0.02 while keeping the original weights, gradient, and learning rate. The result should be (1.91, -0.96). Explain why this arithmetic alone cannot tell you which decay setting will perform better on new examples.

Learning rate

Sources

Primary source: Loshchilov and Hutter, Decoupled Weight Decay Regularization, equations and optimizer comparison: https://arxiv.org/pdf/1711.05101 . The numerical update, coordinate-scaling example, and case study are original.

Previous: Weight Next: Weight initialization