AI Glossary / Deep learning / Intermediate
Layer norm
Definition
Layer norm, short for layer normalization, is a neural-network operation that normalizes activation values across specified feature dimensions within an individual example or position. It subtracts their mean and divides by a stabilized standard deviation, usually followed by learned scaling and shifting. It helps control the numerical behavior of intermediate representations without estimating those statistics across different examples in a batch.
What it means
The key question is which values belong to the group being normalized. For a sequence represented as batch-by-token-by-feature, a common choice normalizes the feature vector of each token separately. It does not automatically combine all tokens or all examples. This distinction matters because changing the normalization axes changes the computation even when the input and output shapes look identical. Layer norm is applied inside a model and is different from fitting a preprocessing scaler to an entire training dataset.
How it works
For a group of d values, compute their mean and the average squared deviation from that mean. For each value x, calculate (x minus mean) divided by the square root of variance plus epsilon. Epsilon is a small positive stabilizing constant. A learned scale gamma and offset beta may then transform the normalized value. The statistics are recalculated from the current group during both training and inference. The learned affine parameters are shared according to the implementation's specified shape, rather than newly learned for every example.
Three practical examples
- Language models: normalize a token's hidden feature vector within a network block to support stable computation.
- Recurrent models: normalize selected activations at each step without relying on a large batch of other sequences.
- Model conversion: reproduce the same axes, epsilon, scale, and offset when transferring a trained architecture between software environments.
Think of it this way
Think of describing each person's set of measurements relative to that person's own average and spread. The analogy captures within-example comparison rather than comparison with other people in the room. Its limit is that model activations are abstract numerical features, and the learned scaling and shifting can deliberately change the final mean and spread.
Worked example
Take one feature vector [1, 3, 5]. Its mean is 3, and its variance using division by three is (4 + 0 + 4)/3 = 8/3. With epsilon = 0.00001, the denominator is approximately 1.6330. Before the learned affine transformation, the vector is approximately [-1.2247, 0, 1.2247]. If every feature has gamma = 2 and beta = 1 for this illustration, the output becomes approximately [-1.4495, 1, 3.4495]. Its mean is now 1, demonstrating why the final layer output need not have mean zero.
Now place a second vector [101, 103, 105] in the same batch. Normalizing each vector across its own three features produces the same normalized pattern because their deviations and variances match. Adding that second example does not change the first vector's layer-normalization statistics. If all features in a vector instead equal 7, its centered values are all zero; epsilon keeps the denominator nonzero, and the affine output equals beta. These calculations assume the stated axes and conventional formula, not every normalization variant.
Illustrative case study
A fictional team exports a text classifier to a different runtime and notices prediction differences. It checks the normalization layer and finds that the new configuration normalizes token and feature dimensions together, while the original normalizes only features within each token. The team restores the intended axes and checks epsilon and learned parameters as well. It compares intermediate outputs before concluding that the conversion matches, rather than relying only on the fact that both systems contain a component named LayerNorm.
Common misconceptions and mistakes
Layer norm does not make activations follow a Gaussian distribution. Centering and rescaling control simple statistics, not the full distribution's shape. The final output need not have zero mean or unit variance after learned affine parameters, and epsilon also prevents exact unit variance in the normalized intermediate values. Layer normalization is not batch normalization or RMS normalization. It does not automatically remove data bias, guarantee training convergence, or replace the need to choose a suitable architecture and learning rate.
Business use
For product teams, layer norm matters mainly through training stability, reproducibility, and compatibility with pretrained models. Preserve its configuration when loading or converting a model. Record which axes are normalized, which axes carry affine parameters, and what precision and epsilon are used. Compare outputs on ordinary and nearly constant feature vectors, where numerical details can matter. Changing normalization placement or replacing it with another variant is an architectural decision that needs evaluation, not a harmless formatting change.
People also ask
Does layer norm require a large batch?
Its normalization statistics are computed within each configured group, so it does not rely on batch statistics in the way standard batch normalization does.
Does it behave differently at prediction time?
The standard normalization formula uses current input statistics in both modes, though other model components may change behavior.
Are gamma and beta always present?
They are common learnable parameters, but implementations may allow scaling or shifting to be disabled.
Check your understanding
After normalization, a vector is multiplied by gamma = 2 and shifted by beta = 1. Must its final mean remain zero?
Answer: No. With those uniform affine parameters, an initially zero-mean normalized vector has final mean 1. The normalization step and the learned affine transformation must be considered separately.
Related terms
- Normalization
- Batch normalization
- Activation
- Variance
- Affine transformation
- RMS normalization
- epsilon
- hidden state
- tensor axis
- Transformer block
Learn next
Practice calculating mean and population-style variance for a short vector, then apply the affine transformation separately. Next compare the axes used by layer, batch, group, and RMS normalization. When inspecting a real model, trace the exact normalization shape and placement instead of assuming every normalization layer performs the same operation.
More in Deep learning
- Large memory storage and retrieval neural network - A large memory storage and retrieval neural network usually refers to LAMSTAR, a historical neural architecture that combines modules based on...
- Large width limits of neural networks - Large width limits of neural networks are mathematical descriptions of what happens when the number of units or channels in selected layers grows without...
- Layer - A layer is a computational component of a neural network that transforms one or more inputs into outputs.
- Learning rule - A learning rule is a procedure that specifies how a model changes its adjustable parameters or internal state in response to data or feedback.
- Least-squares Analysis - Least-squares analysis estimates model parameters by minimizing the sum of squared residuals: the differences between observed values and the model's fitted...
- Lernmatrix - The Lernmatrix, German for learning matrix, is an early associative-learning architecture developed by Karl Steinbuch around 1960.
Sources
Ba, Kiros, and Hinton, Layer Normalization: https://arxiv.org/abs/1607.06450 Keras, LayerNormalization formula and axis configuration: https://keras.io/api/layers/normalization_layers/layer_normalization/ PyTorch, LayerNorm documentation: https://docs.pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html
Previous: Layer Next: Layered coding