Does every objective have a Hessian everywhere?

No. Nonsmooth functions and insufficient differentiability can prevent its existence or the usual properties.

Does a positive-definite Hessian at a stationary point prove a global minimum?

It proves a strict local minimum under the usual smoothness conditions; global claims need additional structure, as in the worked convex quadratic.

Must a solver store the full Hessian?

No. Some methods use products with vectors or structured approximations.

AI Glossary / Mathematics and optimization / Advanced

Hessian

Definition

The Hessian of a scalar-valued function is the matrix of its second-order partial derivatives. It describes how the gradient changes with the input coordinates and therefore captures local curvature. In optimization, a Hessian can help distinguish different stationary points and guide curvature-aware updates. For a function with continuous second partial derivatives near a point, the mixed derivatives agree, so the Hessian there is symmetric.

What it means

The gradient gives first-order sensitivity, while the Hessian describes changes in that sensitivity. Its diagonal entries measure curvature along coordinate directions, and off-diagonal entries describe interactions between directions. A Hessian is evaluated at a point and can change across the domain. For a quadratic objective it is constant, which makes quadratic examples useful for learning. Curvature information can improve an optimization method, but storing and processing the entire matrix can be expensive when a model has many parameters.

How it works

For a sufficiently smooth function, the second-order approximation is f(x + d) ≈ f(x) + ∇f(x)ᵀd + (1/2)dᵀH(x)d. The quadratic term predicts curvature along displacement d. At a stationary point, a positive-definite Hessian establishes a strict local minimum, a negative-definite Hessian a strict local maximum, and an indefinite Hessian a saddle. A semidefinite Hessian can leave the test inconclusive. Newton's method uses curvature by solving H(x)p = −∇f(x) when appropriate. Practical methods may modify this approach or use Hessian-vector products instead of storing the full matrix.

Three practical examples

A numerical optimizer can use second derivatives to scale a search direction differently along steep and shallow directions.
A mathematical model review can inspect the Hessian at a stationary point to distinguish a minimum from a saddle when the second-order test applies.
A large-scale solver can compute how the Hessian acts on a vector without constructing every entry, reducing storage requirements for suitable algorithms.

Think of it this way

Think of inspecting how a landscape bends around your position, rather than only which way it slopes. A bowl, a hilltop and a saddle can all have a flat first-order reading at their centers, but their curvature differs. The analogy is limited because a high-dimensional Hessian can mix coordinates, and flat second-order curvature may hide behavior determined by higher-order terms.

Worked example

Let f(u, v) = u² + uv + 2v² − 3u. Its gradient is (2u + v − 3, u + 4v), and its Hessian is H = [[2, 1], [1, 4]]. The matrix is positive definite because its first leading principal minor is 2 and its determinant is 8 − 1 = 7, both positive. Solving the stationary equations gives u + 4v = 0 and 2u + v = 3, so v = −3/7 and u = 12/7. This quadratic has a unique global minimum there. From (0, 0), the gradient is (−3, 0). The Newton equation H p = (3, 0) gives p = (12/7, −3/7), reaching the minimizer in one exact step. For a curvature check along d = (1, 1), dᵀHd = 8, so the quadratic contribution is 4. Indeed, f(1, 1) = 1, while f(0, 0) + ∇f(0, 0)ᵀd + 4 = 0 − 3 + 4 = 1. Contrast f(t) = t⁴ and g(t) = −t⁴: both have zero first and second derivatives at zero, but one has a minimum and the other a maximum. A zero Hessian does not settle the classification.

Illustrative case study

In a fictional solver project, Birch Design investigates slow progress in a smooth two-parameter calibration model. Engineers calculate the Hessian on a small reference case and compare a curvature-aware step with a basic gradient step. They verify mixed derivatives and check whether a candidate direction is suitable before accepting it. For the larger model, they assess whether Hessian-vector products offer a practical alternative to full storage. The case illustrates numerical design rather than a claim that second-order methods always train faster.

Common misconceptions and mistakes

A Hessian is not simply a vector of second derivatives along the coordinates; mixed derivatives matter. Symmetry requires appropriate smoothness assumptions and should not be asserted for arbitrary functions with some partial derivatives. Positive curvature along coordinate axes alone does not prove positive definiteness. At a nonstationary point, the Hessian sign test does not by itself establish an optimum. A positive semidefinite Hessian at one point is weaker than positive definiteness there or convexity throughout a domain. Explicit matrix inversion is also unnecessary for computing a Newton direction; a linear solve is generally the relevant operation.

Business use

Use Hessian information when its computational cost and mathematical assumptions suit the problem. Verify derivative implementations on small examples and assess conditioning before trusting a curvature solve. Record whether the matrix is exact, approximated or modified. For nonconvex objectives, include suitable safeguards rather than assuming a Newton direction is always downhill. Monitor the actual objective and constraints at accepted steps. When explaining a result to business users, distinguish efficient numerical fitting from evidence that the fitted model predicts well or represents the intended decision process.

Check your understanding

Why do t⁴ and −t⁴ show the limitation of a zero-Hessian test at zero?

Answer: Their first and second derivatives agree there, yet higher-order behavior makes one a minimum and the other a maximum.

Related terms

Gradient
curvature
Newton's method
Hessian-vector product
positive definiteness
saddle point

Learn next

derive the mixed derivatives in the worked objective and verify the Newton equation by matrix multiplication. Then examine u² − v² at the origin and identify its Hessian signs. Compare what first-order and second-order information can establish in each example.

Sources

Sources: Jung and colleagues, The Aalto Dictionary of Machine Learning, Hessian, https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; SciPy, Optimization tutorial, Newton and trust-region methods, https://docs.scipy.org/doc/scipy/tutorial/optimize.html . The quadratic Hessian, Newton solve, second-order identity and fourth-power counterexamples are original calculations.

Previous: Hermitian Next: Heterogeneity of variance