Does gradient descent require a neural network?

No. It can minimize many differentiable objectives, including simple quadratics.

Is a larger learning rate always faster?

No. It can accelerate useful progress in some ranges but can also create cycles or divergence.

Does it always find the global minimum?

No. Guarantees depend on the objective and algorithmic assumptions; general nonconvex problems do not carry that promise.

AI Glossary / Mathematics and optimization / Intermediate

Gradient descent

Definition

Gradient descent is an iterative method for minimizing a differentiable function. At each step, it subtracts a scaled gradient from the current parameter vector: x_(k+1) = x_k − η_k∇f(x_k), where η_k is the learning rate or step size. The negative gradient is a local downhill direction in Euclidean geometry. Whether the sequence reaches a useful minimum depends on the objective, step sizes and other assumptions.

What it means

The method replaces a difficult direct solution with a sequence of corrections. In machine learning, f is often a training objective and x contains model parameters. The gradient measures local sensitivity, while the learning rate controls how far to move. These roles should not be confused: a correct gradient with an excessive step can increase the loss. Plain gradient descent uses the gradient of the specified full objective; stochastic and mini-batch variants estimate it using sampled data, introducing additional variability.

How it works

Choose initial parameters and evaluate the objective gradient there. Form a candidate update in the opposite direction and repeat according to the chosen step-size and stopping policies. A fixed learning rate is simple, while line-search methods adjust the step using information about objective behavior. For a differentiable objective with an L-Lipschitz gradient, a sufficiently small step gives a descent guarantee under the usual assumptions. Convexity helps connect stationary points with global minima; nonconvex objectives can contain saddles and different local minima. Gradient descent is an algorithm, not a claim that every training problem is convex.

Three practical examples

A linear-regression implementation can iteratively minimize squared error when an iterative solution is appropriate for its data and computational setting.
A differentiable calibration model can adjust several parameters together while monitoring the objective and gradient norm.
A numerical exercise can compare learning rates on the same function to observe convergence, oscillation and divergence without changing the model or data.

Think of it this way

Think of repeatedly taking a measured downhill step on a smooth surface. The local slope suggests a direction, but choosing the stride is equally important. The analogy becomes less reliable in many dimensions, with noisy gradient estimates or on a landscape containing broad flat regions and saddles. It explains the update idea, not a guarantee of finding the deepest valley.

Worked example

Let f(w) = 2(w − 3)², so f'(w) = 4(w − 3). A gradient-descent step is w_(k+1) = w_k − 4η(w_k − 3). Relative to the minimizer 3, the error e_k = w_k − 3 follows e_(k+1) = (1 − 4η)e_k. Starting from w_0 = 0 with η = 0.1 gives w_1 = 1.2, w_2 = 1.92 and w_3 = 2.352. Objective values are 18, 6.48, 2.3328 and 0.839808. Each error shrinks by factor 0.6, and each objective value by factor 0.36. If η = 0.4, the error factor is −0.6, so the sequence oscillates around 3 but still converges. At η = 0.5, the factor is −1, giving a two-point cycle from this start. At η = 0.6, the factor is −1.4 and the errors grow. Thus, for this quadratic, convergence from every starting point occurs when 0 < η < 0.5; the special start w_0 = 3 stays fixed for any step size. This calculation illustrates an exact stability range for one function, not a transferable learning-rate rule for all models.

Illustrative case study

In a fictional calibration project, Larch Instruments sees a loss curve alternating between high values. The team reproduces the behavior on a simplified objective and checks the gradient implementation before changing the learning rate. It logs objective values and actual parameter movements to distinguish a coding error from an unstable step choice. After choosing a numerically suitable procedure, the team evaluates the fitted model on separate measurements. The case illustrates diagnosis rather than a claimed production performance gain.

Common misconceptions and mistakes

Gradient descent does not always reduce the objective with an arbitrary learning rate. Oscillation does not always mean divergence: the worked η = 0.4 example oscillates while its error shrinks. A small gradient does not by itself establish a global minimum in a nonconvex problem. A very small learning rate can make progress impractically slow or produce tiny steps far from a solution. Momentum and adaptive optimizers use additional update rules and should not be described as identical to plain gradient descent. Lower training loss also does not guarantee improved prediction on new data.

Business use

Define and scale the objective carefully, including whether example losses are summed or averaged and how regularization is weighted. Verify gradients on a small test case. Record initialization, learning-rate settings and stopping criteria, then monitor loss, gradient norm and non-finite values. Use validation to assess predictive consequences separately from optimization progress. Compare iteration cost with alternative solvers when the problem has useful structure. Treat a maximum-iteration exit as a resource decision unless a stated convergence criterion has actually been met.

Check your understanding

Why does the worked update with η = 0.4 converge despite crossing the minimizer repeatedly?

Answer: Its error multiplier is −0.6. The negative sign changes sides, while the magnitude below one shrinks the distance at each step.

Related terms

Learn next

derive the error multiplier for f(w) = a(w − b)² with a > 0 and determine the fixed-step stability interval. Compare loss and parameter trajectories rather than judging convergence from crossings alone. Then explain how multiplying the loss by a constant changes a suitable learning-rate scale.

Learning rate

Sources

Sources: Jung and colleagues, The Aalto Dictionary of Machine Learning, gradient descent, https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; Alexander Jung, Federated Learning: From Theory to Practice, Chapter 4, https://arxiv.org/pdf/2505.19183 ; Google, Machine Learning Glossary, https://developers.google.com/machine-learning/glossary . The scalar stability derivation, numerical sequence and fictional instrument review are original.

Previous: Gradient clipping Next: Gradient domain