Does the method require a full Hessian?

Common variants such as Fletcher - Reeves do not; they use gradients and stored vectors.

Will it solve every n-variable problem in n steps?

No. The ideal finite-termination statement belongs to linear conjugate gradients on suitable quadratic problems with exact arithmetic, not general nonlinear objectives.

Returning to the negative gradient can discard an unhelpful direction history and restore a suitable search direction under the chosen algorithm's rules.

AI Glossary / Mathematics and optimization / Advanced

Nonlinear conjugate gradient method

Definition

The nonlinear conjugate gradient method is a family of algorithms for minimizing a smooth function by combining the current negative gradient with information from the previous search direction. It extends ideas from linear conjugate gradients to general nonlinear objectives. Its modest memory requirements make it useful for some large optimization problems where storing a full Hessian matrix would be costly.

What it means

The gradient points toward local steepest increase, so its negative provides a natural direction for reducing an objective. Repeated steepest-descent steps can make slow, zigzagging progress on some landscapes. Nonlinear conjugate gradient methods retain a scaled part of the preceding direction to improve the search. The scaling formula distinguishes variants such as Fletcher - Reeves and Polak - Ribière - Polyak. Despite the name, directions are not generally exactly conjugate under one fixed matrix for a nonlinear function, and finite-step guarantees from ideal linear conjugate gradients do not carry over unchanged.

How it works

Start at an initial point and set the first direction to the negative gradient. Use a line search to choose a positive step length, then evaluate the gradient at the new point. Form the next direction as minus the new gradient plus beta times the previous direction. Fletcher - Reeves uses beta equal to the new squared gradient norm divided by the old squared norm. Other variants use different combinations of gradient changes and directions. Practical methods use appropriate line-search conditions and may restart with steepest descent when retained information becomes unhelpful. Convergence properties depend on the variant, smoothness assumptions and step-selection rules.

Three practical examples

A researcher minimizes a large smooth fitting objective while avoiding storage of a dense second-derivative matrix.
An engineering model supplies objective values and gradients for an unconstrained parameter-adjustment problem, possibly obtaining those gradients through an adjoint calculation.
A machine-learning experiment compares deterministic optimization methods for a full-data smooth loss, measuring gradient evaluations and runtime rather than assuming an optimizer is best because it takes fewer iterations.

Think of it this way

Think of walking downhill while remembering a useful part of your previous heading. Instead of resetting your route completely at each step, you combine current slope information with that history. The analogy captures directional memory, but beta is computed mathematically and a line search checks the step. Blindly continuing a previous heading would not provide the method's safeguards or convergence behavior.

Worked example

Use the nonlinear objective f(x, y) = x⁴/4 + y²/2. At (1, 1), f = 0.75 and the gradient is (1, 1), so the first direction is (−1, −1). Suppose a line search accepts step length 0.5. The new point is (0.5, 0.5), with objective 0.5⁴/4 + 0.5²/2 = 0.140625. Its gradient is (0.125, 0.5). Fletcher - Reeves gives beta = (0.125² + 0.5²)/(1² + 1²) = 0.1328125. The next direction is therefore (−0.125, −0.5) + 0.1328125 × (−1, −1), or (−0.2578125, −0.6328125). Its dot product with the new gradient is approximately −0.348633, so it is a descent direction at that point. A new line search must still choose how far to move. This example demonstrates one accepted step and direction update, not an exact line minimization or a general guarantee for arbitrary step choices.

Illustrative case study

In a fictional project, Summit Models fits a smooth simulation model with many adjustable parameters. A team tests nonlinear conjugate gradients because memory is limited. It discovers that one iteration can require several objective and gradient evaluations during line search, so it compares total computational work with alternative optimizers. It also checks gradients numerically and uses documented restart rules. The scenario describes an evaluation process; it does not claim a particular speedup or real engineering result.

Common misconceptions and mistakes

Do not confuse nonlinear conjugate gradients with the linear method for symmetric positive-definite systems. Do not assume beta always behaves like a fixed momentum coefficient: it is recomputed according to the variant. A descent direction does not guarantee that an arbitrarily large step reduces the objective. The basic method is for unconstrained smooth optimization, so simply clipping variables to bounds creates a different algorithm. Noisy stochastic gradients can interfere with the usual line-search logic, and a convergence theorem for one formula should not be attributed to every variant.

Business use

Consider this method when gradients are available, the objective is reasonably smooth and memory matters. Benchmark using objective quality, gradient norm, wall-clock time and evaluation counts. Record the exact beta formula, line-search settings, stopping criteria and restart strategy so results can be reproduced. Inspect variable scaling because badly scaled parameters can impair optimization. For nonconvex objectives, compare initializations where appropriate and distinguish a stationary point from a proven global minimum. A successful numerical termination does not validate the business meaning of the fitted objective.

Check your understanding

The next search direction has a negative dot product with the gradient. Can any positive step length be accepted?

Answer: No. That sign indicates local descent, but a sufficiently large step may overshoot and increase the objective. Step selection remains necessary.

Related terms

Gradient descent
Line search
Hessian
Fletcher–Reeves method
Polak–Ribière method
unconstrained optimization

Learn next

reproduce the worked update, then compare several candidate step lengths along its next direction. Study sufficient-decrease and curvature conditions in line search, and observe how changing beta or restarting affects a small smooth test problem.

Line search

Sources

Sources: William W. Hager and Hongchao Zhang, A Survey of Nonlinear Conjugate Gradient Methods, equations 1.2–1.3 and Table 1.1, https://www.math.lsu.edu/~hozhang/papers/cgsurvey.pdf ; authors' methods and software bibliography, https://people.clas.ufl.edu/hager/gradient-methods-and-software/ . The quartic example above is an original checked calculation.

Previous: Nonhomogeneous Gaussian regression Next: Nonlinear constraints