Does BHHH require individual gradients?

It needs observation-level score information, whether supplied analytically or obtained numerically from individual contributions.

Is its matrix always the exact Hessian?

No. It is an outer-product approximation motivated by an expected information identity.

Can it replace Newton's method for any objective?

Its statistical justification is specific to likelihood settings with suitable assumptions. Arbitrary objectives do not inherit that justification.

AI Glossary / Mathematics and optimization / Advanced

Berndt - Hall - Hall - Hausman algorithm

Definition

The Berndt - Hall - Hall - Hausman algorithm, usually called BHHH, is a numerical method for maximum-likelihood estimation. It uses the individual observations' log-likelihood gradients to approximate curvature, avoiding explicit second derivatives. Its key matrix is a sum of outer products of individual score vectors. This must not be confused with the outer product of the total gradient, which generally produces a different and unsuitable matrix.

What it means

Maximum-likelihood estimation chooses parameters that make the observed data more probable under a statistical model. A score vector is the gradient of a log-likelihood contribution: it describes how that observation pushes the parameter estimate.

BHHH uses the variation among these individual scores to scale a parameter update. Its justification comes from the information-matrix equality under suitable likelihood and regularity assumptions. It is therefore a specialized likelihood optimization method, rather than an arbitrary replacement for the Hessian of every function one might wish to optimize.

How it works

Write the log-likelihood as a sum of contributions l_i(theta), and calculate each score s_i = gradient l_i(theta). Sum the scores to obtain the total gradient g. Form B = sum of s_i times s_i transposed.

When B is nonsingular, solve B d = g for an ascent direction d. Update theta along d with a suitable step length, checking the likelihood and parameter domain. Repeat until the selected convergence criteria are met.

B is positive semidefinite by construction, but it can be singular or poorly conditioned. The empirical matrix is not generally identical to the observed negative Hessian. The information equality concerns expectations under appropriate assumptions, not an exact equality for every dataset and trial parameter.

Three practical examples

1. A custom likelihood estimator uses observation-level analytic scores when second derivatives are difficult to implement reliably.

2. A statistical modeling team compares BHHH and Newton updates on a small model with a known solution before extending its estimation routine.

3. An econometric software interface accepts a matrix with one score row per observation so that it can construct the BHHH curvature approximation correctly.

Think of it this way

Think of deciding a step from many individual recommendations, using both their combined direction and how strongly they vary. BHHH records the separate score contributions before combining them into a scale matrix. The analogy does not mean that observations vote equally in a social sense or that disagreement necessarily signals a bad statistical model.

Worked example

Suppose observations 0, 1, and 3 are modeled as independent normal values with unknown mean mu and known variance one. Ignoring constants, the log-likelihood is minus one-half the sum of squared residuals. Each individual score is y_i − mu.

At mu = 0, the scores are 0, 1, and 3. Their sum is g = 4. The scalar BHHH matrix is B = 0 squared + 1 squared + 3 squared = 10. Solving B d = g gives d = 0.4. With a full step, the proposed new mean is 0.4.

The squared-residual sum falls from 10 to 0.16 + 0.36 + 6.76 = 7.28. Thus the log-likelihood rises by half of 10 − 7.28, or 1.36. This proposed step is an improvement, but it does not yet reach the maximum-likelihood mean, 4/3.

For comparison, the observed negative Hessian is 3 throughout this example. A Newton direction at zero would be 4/3, showing that BHHH's empirical value 10 is not the exact curvature here.

At the optimum mu = 4/3, individual scores are −4/3, −1/3, and 5/3. They sum to zero, yet their squared sum is 14/3. If one incorrectly squared the total score instead, the result would be zero and would discard the information carried by individual contributions. These calculations are hand-worked and do not establish convergence behavior for other likelihoods.

Illustrative case study

Fictional case: Elm Research writes a custom estimator and supplies only the summed gradient to a BHHH routine. The optimizer produces an unstable matrix, which the team first attributes to poor starting values.

The developers inspect the interface and provide one score vector per independent observation instead. They verify derivatives against a small analytic example, record likelihood values, and investigate rank and scaling before adjusting tolerances. They also review whether repeated measurements can really be treated as independent contributions. The case illustrates implementation and modeling checks; it claims no real study result.

Common misconceptions and mistakes

Do not confuse sum(s_i s_i-transpose) with sum(s_i) times sum(s_i)-transpose. In multiple dimensions, the latter has rank at most one and generally loses essential information.

Do not assume positive semidefinite means invertible. Redundant parameters or insufficiently varied scores can make the matrix singular.

Also distinguish an optimizer's curvature approximation from a final uncertainty estimator. Standard-error choices require statistical assumptions and may need different treatment under misspecification or dependence. A successful numerical stopping code does not validate the likelihood model or prove that a global maximum has been reached.

Business use

BHHH can be useful for custom maximum-likelihood models when individual gradients are available and Hessians are costly. Compare achieved likelihood, derivative accuracy, conditioning, and total computation with other suitable solvers.

Define the unit of likelihood contribution carefully. Artificially splitting or aggregating terms can change the outer-product approximation, so the statistical decomposition matters. Keep parameter constraints explicit, especially for probabilities and positive scales, and retain enough diagnostics to distinguish genuine convergence from a failed step or exhausted iteration budget.

Check your understanding

Why is B still 14/3 at the optimum in the worked example even though the total gradient is zero?

Answer: The individual scores are nonzero and their squares add. Their signed sum cancels, which is a different operation from summing squared contributions.

Related terms

Hessian
Line search
Maximum likelihood
score function
Fisher information
outer product of gradients
Newton–Raphson method
parameter identification

Learn next

Study score vectors and information matrices next. Recalculate the example after shifting every observation by the same constant. Then distinguish the total score, observed Hessian, and outer-product matrix at a proposed parameter value.

Hessian

Sources

Primary software-author reference opened: Ott Toomet, “Maximum Likelihood Estimation with maxLik,” https://cran.r-project.org/web/packages/maxLik/vignettes/using-maxlik.pdf, section 2.3. It explains information equality, observation-level scores and BHHH input requirements. The three-observation example and fictional case are original illustrations.

Previous: Berlin procedure Next: Bernoulli