Does every gradient-based method use a fixed learning rate?

No. Methods can use line searches, trust regions, adaptive scales or other step controls.

Are finite-difference gradients allowed?

Yes, an algorithm can use an approximate gradient, though approximation error and evaluation cost affect its behavior.

Does using second derivatives make a method no longer gradient-based?

Not under the broad usage here; it makes the method richer than a purely first-order approach.

AI Glossary / Mathematics and optimization / Intermediate

Gradient-based method

Definition

A gradient-based method is an optimization procedure that uses derivatives of an objective to guide its search. Gradient descent is one example, but the category also includes methods that use momentum, curvature information, approximate inverse Hessians or adaptive coordinate scaling. The shared feature is that gradient information influences updates. The term identifies a broad family, not one update formula or a universal convergence guarantee.

What it means

The gradient describes how a differentiable objective changes locally with its variables. Different methods use that information differently: one may follow the negative gradient directly, another may combine it with earlier directions, and another may solve a local curvature-based problem. The methods can therefore take very different steps from the same starting point. Choosing among them involves derivative availability, smoothness, constraints, memory, evaluation cost and numerical conditioning. Calling a method gradient-based does not mean it is limited to neural-network training.

How it works

A typical iteration evaluates or estimates a gradient, chooses a direction using the method's rule, selects or accepts a step, and updates the variables and any internal state. Some methods use a full objective gradient; others use sampled estimates. A line search can decide how far to move, while a trust-region method limits a local model's range of use. Methods using second derivatives can still be gradient-based in this broad sense, but they are not first-order methods. A derivative-free procedure instead makes search decisions without requiring explicit gradient information.

Three practical examples

A smooth parameter-estimation problem can be approached with a quasi-Newton method that uses gradients to build a curvature approximation.
A large model-training problem can use stochastic gradient information when evaluating the full data objective at every update is too costly.
An engineering optimization study can compare direct negative-gradient steps with curvature-scaled steps on an ill-conditioned objective, measuring both evaluation cost and progress.

Think of it this way

Think of navigators who all consult a local slope measurement but use different strategies. One walks directly downhill, another remembers recent movement, and another uses a map of curvature to choose a better-scaled route. The analogy highlights shared information with different decisions. It does not imply that more elaborate equipment is always better or that any navigator can infer the entire landscape from local measurements.

Worked example

Consider f(u, v) = u²/2 + 50v². Its gradient is (u, 100v), and its Hessian is the diagonal matrix with entries 1 and 100. At (1, 1), the objective is 50.5 and the gradient is (1, 100). A plain gradient-descent step with η = 0.01 gives (0.99, 0), reducing the objective to 0.99²/2 = 0.49005. The steep v direction is corrected, but the u coordinate changes little. A Newton direction solves H p = −∇f, giving p = (−1, −1). A full Newton step reaches (0, 0), the exact minimizer for this quadratic. This example shows how curvature scaling can change the direction and step effectiveness. It does not imply that Newton's method always solves an objective in one step: the result relies on this exact positive-definite quadratic and an exact solve. With plain gradient descent at η = 0.03, the candidate is (0.97, −2), whose objective is 0.47045 + 200 = 200.47045, worse than the starting value. Both methods use gradients, but their use of geometry and their step controls differ.

Illustrative case study

In a fictional calibration study, Poplar Engineering compares two gradient-based solvers on a smooth model with parameters in different units. The team measures derivative-evaluation time and checks the effect of rescaling variables. A method with fewer iterations is not automatically chosen if each iteration is much more expensive. Engineers also verify constraints and held-out calibration behavior. The case illustrates method comparison without claiming that a particular solver universally outperforms another.

Common misconceptions and mistakes

Gradient-based is broader than gradient descent. Newton and quasi-Newton methods are not identical to first-order updates. An automatically differentiated gradient can be accurate for the implemented computation while that computation represents the wrong objective. Nondifferentiable points require appropriate treatment rather than blindly applying smooth formulas. Stochastic gradients add sampling variability, so deterministic convergence intuitions may not transfer unchanged. A method can optimize training loss effectively while producing poor generalization. Finally, local derivative information does not ordinarily establish that a nonconvex solution is globally best.

Business use

Start with the objective's properties and computational budget rather than a favored optimizer name. Determine whether gradients and curvature information are available and reliable. Assess variable scaling, constraints and expected noise in evaluations. Compare methods under consistent stopping criteria and report function, gradient and other expensive evaluations. Inspect solver status, feasibility and the original objective at the returned point. For predictive models, evaluate generalization separately. Preserve configuration details and initialization so a method change can be distinguished from a change in the problem itself.

Check your understanding

Why does the Newton step correct both coordinates in one move in the worked example?

Answer: The Hessian exactly describes this quadratic's curvature. Solving H p = −∇f scales the two gradient components appropriately and gives the displacement to the minimizer.

Related terms

Gradient descent
Line search
Derivative-free optimization
Newton's method
quasi-Newton method
stochastic gradient

Learn next

rescale v in the worked objective so the two directions have comparable curvature and examine plain gradient descent again. Then compare iteration counts with the cost of computing or approximating a Hessian. Explain why a method's efficiency cannot be inferred from its number of iterations alone.

Gradient descent

Sources

Sources: Jung and colleagues, The Aalto Dictionary of Machine Learning, gradient-based method, https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; SciPy, Optimization tutorial, https://docs.scipy.org/doc/scipy/tutorial/optimize.html ; David Bindel, Cornell numerical-analysis notes, https://www.cs.cornell.edu/courses/cs4220/2026sp/lec/2026-03-27.html . The anisotropic quadratic and exact update comparisons are original; the one-step Newton result is restricted to the stated example.

Previous: Gradient bandit algorithm Next: Gradient boosted (decision) trees