Is the gradient the same as the learning rate?

No. The gradient describes sensitivity, while the learning rate controls how strongly a method uses it in an update.

Can a gradient have positive and negative components?

Yes. Different coordinates can increase or decrease the objective locally.

Does a zero gradient prove a minimum?

Not in general. Additional properties, such as convexity or an independent argument, are needed.

AI Glossary / Mathematics and optimization / Intermediate

Gradient

Definition

The gradient of a differentiable scalar-valued function is the vector of its partial derivatives. Each component describes the function's local rate of change with respect to one input coordinate. In machine learning, the function is often a training loss and the coordinates are model parameters. Under Euclidean geometry, the gradient points in the direction of greatest local increase, while its negative gives the direction of greatest local decrease for a unit-length move.

What it means

A derivative describes sensitivity to a small change in one variable. A gradient collects those sensitivities when there are several variables. It is evaluated at a particular point: changing model parameters generally changes the gradient. The gradient is information about a local approximation, not an instruction to take an arbitrarily large step. An optimizer combines it with a step-size rule and possibly additional state or geometry. The gradient itself is therefore distinct from both gradient descent and the actual parameter update.

How it works

For a differentiable function f and a small displacement d, the first-order approximation is f(x + d) ≈ f(x) + ∇f(x)ᵀd. The dot product predicts how the objective responds in a chosen direction. Among directions with Euclidean norm one, the gradient direction maximizes that first-order increase when the gradient is nonzero. Gradients can be calculated analytically, by automatic differentiation or approximately through finite differences. Backpropagation applies the chain rule efficiently through a computational graph. The calculation must identify which variables are being differentiated and which quantities are treated as fixed.

Three practical examples

A regression training routine can differentiate squared prediction error with respect to its coefficients to guide an optimization step.
A sensitivity study can differentiate a scalar model output with respect to input measurements, which is a different gradient from the loss gradient with respect to model parameters.
An implementation check can compare automatic derivatives with small finite-difference calculations on a controlled, differentiable example.

Think of it this way

Think of standing on a smooth landscape with a local slope indicator. It describes the steepest nearby uphill direction, and reversing it points locally downhill. The analogy assumes a geometry and sufficiently small movement. A long stride can cross a valley or reach a higher region, and a flat reading does not tell you whether you are at a bottom, top or saddle.

Worked example

Let f(u, v) = (u − 2)² + 2(v + 1)². Its gradient is (2(u − 2), 4(v + 1)). At (0, 0), the function value is 6 and the gradient is (−4, 4). Its Euclidean norm is √32, and the unit steepest-descent direction is (1/√2, −1/√2). For a plain step with learning rate 0.1, the displacement is −0.1∇f = (0.4, −0.4). The linear approximation predicts a change of (−4)(0.4) + 4(−0.4) = −3.2, suggesting value 2.8. The actual value is (−1.6)² + 2(0.6)² = 2.56 + 0.72 = 3.28. It decreases, but the approximation is not exact because curvature contributes additional terms. Taking learning rate 1 instead gives (4, −4), with value 4 + 18 = 22, exceeding the starting value. The same initial gradient therefore supports a useful small step and an excessive large one. Finally, at (2, −1) the gradient is zero and this particular function reaches its global minimum zero; that conclusion uses the sum-of-squares form, not merely the zero-gradient condition.

Illustrative case study

In a fictional calibration project, Silver Sensors differentiates a model loss with respect to two adjustable coefficients. A reviewer checks the derivative on a small synthetic problem and discovers that a preprocessing scale was omitted from one chain-rule calculation. After correcting the implementation, the team still evaluates the optimization settings and held-out behavior separately. The case illustrates derivative verification without claiming that a mathematically correct gradient guarantees a useful model.

Common misconceptions and mistakes

A gradient is a vector for a scalar-valued function; a vector-valued mapping generally has a Jacobian matrix of derivatives. A negative gradient is not a guarantee that every finite step decreases the objective. A zero gradient can indicate a saddle or maximum in a nonconvex problem. At a nondifferentiable point, an ordinary gradient may not exist, and a subgradient or other treatment needs to be specified. Gradient magnitude depends on coordinate units and loss scaling. Calling a direction steepest without naming the geometry can conceal an important assumption.

Business use

When reviewing a training process, distinguish the loss being differentiated from the metric used to judge business performance. Confirm the parameter and input scales, loss reduction convention and derivative implementation. Use small examples to verify signs and factors before investigating a large model. Monitor non-finite gradients and unexpected magnitudes, but interpret them in context rather than using one universal threshold. For sensitivity explanations, state whether derivatives concern inputs or parameters and remember that local sensitivity does not establish causal effects of real-world interventions.

Check your understanding

Why is the actual small-step loss 3.28 rather than the first-order prediction 2.8?

Answer: The gradient describes the linear part of the local change. The quadratic function also contributes curvature terms over the finite displacement.

Related terms

Learn next

compute the gradient at (1, −0.5) in the worked function and predict changes along each coordinate. Compare the prediction with exact values for progressively smaller displacements. Then rescale one coordinate and explain why the gradient components and Euclidean steepest direction change.

Sources

Sources: Jung and colleagues, The Aalto Dictionary of Machine Learning, gradient and gradient descent, https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; Google, Machine Learning Glossary, https://developers.google.com/machine-learning/glossary ; Alexander Jung, Federated Learning: From Theory to Practice, https://arxiv.org/pdf/2505.19183 . The quadratic gradient, finite-step comparison and fictional calibration review are original.

Previous: Gossip protocol Next: Gradient accumulation