Does this problem occur only in recurrent networks?

No. Recurrent models provide a clear setting for long derivative products, but other deep computations can also produce excessive gradients.

Does norm clipping change direction?

Scaling the entire finite nonzero gradient by one positive factor preserves its direction; coordinatewise clipping can change it.

Is there one correct clipping threshold?

No. The threshold depends on the model, loss scale and training setup and should be assessed with diagnostics and validation.

AI Glossary / Mathematics and optimization / Intermediate

Exploding gradient problem

Definition

The exploding gradient problem occurs when gradients become excessively large during model training, making parameter updates unstable or numerically unreliable. It is especially associated with repeated derivative multiplication in deep or recurrent computations. Large gradients can produce abrupt loss increases, oversized updates or non-finite values. A large number alone is not sufficient diagnosis; its meaning depends on the loss scale, parameterization and optimizer.

What it means

A gradient tells a training algorithm how the objective changes with its parameters. Backpropagation carries this information through a chain of computations. If that chain repeatedly amplifies the signal, some gradient components can grow dramatically. In recurrent networks, the same transformation is applied over multiple time steps, creating long products of Jacobian matrices. Exploding gradients concern this sensitivity during differentiation. They are related to, but distinct from, large activations, an excessive learning rate or a numerical error elsewhere in the training pipeline.

How it works

By the chain rule, the derivative of a composed computation contains products of local derivatives. In a scalar example, repeatedly multiplying by a factor above one creates exponential growth. For vectors, products of Jacobian matrices can amplify particular directions, with behavior depending on the matrices and the path through the model. Training then combines these gradients with a learning rate and optimizer state. Gradient-norm clipping is one mitigation: if a gradient vector's norm exceeds a threshold, scale the whole vector down to that threshold before the optimizer uses it. This limits that gradient norm but does not prove that training will converge.

Three practical examples

A recurrent sequence model may exhibit sudden large gradient norms when long temporal dependencies create strongly amplified backward signals.
A deep-network training run can monitor layerwise gradient statistics to locate where large values first appear.
An optimization experiment can compare unclipped and norm-clipped updates while preserving the same data, loss definition and diagnostic measurements, rather than judging stability from the final loss alone.

Think of it this way

Think of a message passed through a sequence of amplifiers. A moderate amplification at each stage can produce a very large final signal. The analogy explains multiplicative growth, but neural-network derivatives are often matrices, can change with the input and can amplify some directions while shrinking others. It also does not imply that every long network inevitably has exploding gradients.

Worked example

Consider the simple recurrence h_t = a h_(t−1). Its sensitivity to the initial state is ∂h_T/∂h_0 = a^T. With a = 1.2 and T = 50, the sensitivity is approximately 9,100.44. With a = 0.8, it is approximately 0.00001427. These values illustrate amplification and attenuation of an initial-state derivative; they are not themselves a complete parameter-gradient calculation for a trained network. Now suppose a separate training step produces parameter gradient g = (30, 40), whose Euclidean norm is 50. Clipping the norm to threshold 5 multiplies the vector by 5/50 = 0.1, giving (3, 4). Under plain gradient descent with learning rate 0.01, the unclipped update has norm 0.5, while the clipped update has norm 0.05. The gradient direction is preserved because every component receives the same positive scale factor. Independently clipping each component to the interval [−5, 5] would instead produce (5, 5), a different direction and norm. For an adaptive optimizer or one with momentum, clipping the raw gradient does not automatically impose that same simple bound on the final parameter update.

Illustrative case study

In a fictional training investigation, Pine Sequence Lab sees intermittent loss spikes in a recurrent model. The team logs gradient norms before clipping, checks the input and loss scaling, and reproduces the issue on a controlled run. They test a clipping threshold while tracking how often it activates and whether validation behavior changes. Persistent clipping prompts further investigation of the model and learning settings. The case illustrates diagnosis and measurement; it does not claim that clipping universally fixes a model or that an invented experiment achieved a performance gain.

Common misconceptions and mistakes

A high learning rate can make updates too large without being the underlying cause of gradient explosion. Conversely, reducing the learning rate can reduce update size without correcting amplified derivatives. Clipping is not equivalent to normalizing every gradient to unit length; gradients below the threshold can remain unchanged. A finite clipping threshold cannot reliably repair a gradient that already contains NaN or infinity. Vanishing gradients are the opposite difficulty and are not solved merely by limiting large norms. Finally, one spike should be investigated in context rather than assigned a cause from the loss curve alone.

Business use

Instrument training before changing several settings at once. Record unscaled gradient norms, loss values and relevant activation statistics, and check for non-finite values before applying an update. Review initialization, sequence length, normalization, data scale and the loss reduction convention. When using clipping, document whether it is global, layerwise or coordinatewise and preserve the threshold. In mixed-precision training, follow the framework's procedure for handling loss-scaled gradients so diagnostics and clipping use the intended scale. Evaluate predictive performance as well as numerical stability, because a stable run can still learn an inadequate model.

Check your understanding

Why does the worked norm clip produce (3, 4) instead of (5, 5)?

Answer: It scales the whole vector by 0.1 so its total Euclidean norm becomes 5. Setting each coordinate to 5 would be a different clipping rule.

Related terms

Learn next

calculate a^T for several sequence lengths and distinguish that sensitivity from the derivative with respect to a. Then compare norm clipping with coordinatewise clipping on a vector with unequal components. Track both the gradient norm and the actual optimizer update when interpreting a training run.

Sources

Sources: Pascanu, Mikolov and Bengio, On the difficulty of training recurrent neural networks, https://proceedings.mlr.press/v28/pascanu13.pdf ; Google, Machine Learning Glossary, https://developers.google.com/machine-learning/glossary . The scalar sensitivity, clipping arithmetic and fictional investigation are original examples; the recurrence is intentionally distinguished from a full parameter-gradient derivation.

Previous: Explicit semantic analysis Next: Exploitation