Is a formation matrix always an exact covariance matrix?

No. Exact equality holds in some models; many uses depend on large-sample approximation.

Can negative off-diagonal entries be valid?

Yes. Covariances can be negative even when the complete matrix is positive definite.

What if information is singular?

The ordinary inverse does not exist. Investigate parameter identifiability and the model before reporting conventional standard errors.

AI Glossary / Mathematics and optimization / Advanced

Formation matrix

Definition

In likelihood-based statistics, a formation matrix is an inverse information matrix: the expected formation matrix is the inverse of Fisher information, while the observed formation matrix is the inverse of observed information. These inverses exist only when the corresponding matrices are nonsingular.

What it means

Information describes how strongly the likelihood distinguishes nearby parameter values. Its inverse expresses the associated scale and direction of parameter uncertainty. In a regular large-sample maximum-likelihood analysis, inverse information provides an approximation to the estimator's covariance matrix.

The distinction between information and its inverse matters. Strong curvature can indicate precise estimation, whereas large inverse-information values can indicate substantial uncertainty. Off-diagonal entries describe relationships between estimated parameters, not just separate error bars. Here, formation refers to statistical likelihood analysis, not the spatial arrangement of robots or vehicles.

How it works

Let l(theta) be the log likelihood for the complete dataset. Observed information is J(theta) = −Hessian(l(theta)). Expected information is I(theta) = E[J(theta)], under regularity conditions permitting this representation. Their inverses are J(theta) inverse and I(theta) inverse.

For uncertainty estimation, analysts commonly evaluate the observed inverse at an interior maximum-likelihood estimate, or substitute that estimate into expected information. The resulting diagonal entries estimate parameter variances; their square roots give approximate standard errors.

Check the likelihood's scaling, parameter units, invertibility and local curvature before interpreting the calculation. Information for n independent, identically distributed observations is n times per-observation information. Inverting the total therefore supplies the sample-size adjustment; dividing by n again would apply it twice.

Three practical examples

1. Model coefficients: an analyst estimates two overlapping effects. Their individual standard errors can be large even when a particular combination of them is precisely estimated. The full inverse matrix reveals that distinction.

2. Experimental design: under a specified statistical model, alternative measurement plans yield different expected information matrices. Their inverses help compare how well each plan could estimate the quantities the project actually needs.

3. Fit diagnostics: a nearly singular information matrix warns that some parameter combination is weakly determined. An apparently successful optimizer does not, by itself, resolve that uncertainty.

Think of it this way

Picture a bowl-shaped landscape around a fitted model. A narrow direction indicates that moving the parameters there quickly worsens the fit. A broad direction permits much more movement. Inverse information translates that local shape into an uncertainty scale, with the bowl's orientation determining which parameter combinations are uncertain.

Worked example

Consider an original teaching model with two unknown parameters. Around its estimate, suppose the log likelihood is exactly quadratic with observed information J = [[5, 3], [3, 5]]. Its determinant is 25 − 9 = 16, so the observed formation matrix is V = [[5, −3], [−3, 5]] / 16.

Multiplication confirms J times V equals the identity matrix: the upper-left entry is (25 − 9)/16 = 1, and the upper-right is (−15 + 15)/16 = 0. The other row gives zero and one.

Each diagonal entry of V is 5/16 = 0.3125. Interpreting V as an estimated covariance matrix gives each parameter a standard error of about 0.559. Their covariance is −3/16 = −0.1875, corresponding to correlation −0.6.

Now consider the sum of the parameters. Its estimated variance is 0.3125 + 0.3125 + 2(−0.1875) = 0.25, giving standard error 0.5. The difference has estimated variance 0.3125 + 0.3125 − 2(−0.1875) = 1, giving standard error 1.

Ignoring the off-diagonal entries would incorrectly assign both combinations variance 0.625. The full matrix therefore changes the practical conclusion: this model estimates the sum more precisely than the difference.

For an exact realization, let one bivariate normal observation have unknown mean theta and known covariance V. Its mean estimate is that observation, and the covariance is exactly V. In a general nonlinear likelihood problem, the inverse-information covariance interpretation is usually approximate rather than exact.

Illustrative case study

In a fictional forecasting project, two fitted coefficients represent highly overlapping demand signals. A dashboard initially displays their estimates without uncertainty. Managers interpret the coefficients separately and propose different operational responses.

The analyst examines the information matrix and discovers a weakly identified direction. The inverse shows that a combined effect is much better estimated than the individual contrast. The team reports the relevant combination and designs additional observations intended to separate the signals.

This does not prove either signal causes demand. It establishes a narrower finding about estimation precision within the selected model, while exposing a limitation that point estimates had concealed.

Common misconceptions and mistakes

Taking reciprocals of individual matrix entries is not matrix inversion. Off-diagonal dependence must be included.

Expected and observed information need not be identical. They can coincide in particular models, but expectation over possible datasets differs from curvature calculated from the observed data.

An inverse Hessian is not automatically a trustworthy covariance estimate. Boundaries, singularity, model misspecification or unsuitable asymptotic approximations can invalidate that interpretation. A pseudoinverse does not make unidentifiable parameters identifiable.

More reported decimal places do not resolve weak information or model error.

Business use

The practical value is explaining how much confidence a model's parameter estimates support and which combinations are informative. Teams can use that evidence when planning measurements, comparing model specifications or communicating uncertainty in estimated effects.

Reports should identify whether the calculation uses expected or observed information, total or per-observation scaling, and which parameterization is being used. A precise estimate under an unsuitable model can still support a poor business conclusion.

Check your understanding

Answer: What is the worked example's covariance? −0.1875.

Which combination is more precisely estimated?

Answer: The sum: its variance is 0.25, compared with 1 for the difference.

Should total-sample inverse information be divided by sample size again?

Answer: No; its scaling already includes the total information.

Related terms

Hessian
Covariance matrix
Standard error
Fisher information
observed information
maximum likelihood estimation
identifiability

Learn next

Verify the matrix product and calculate uncertainty for another linear combination. Then compare observed and expected information in a likelihood model where they differ, paying attention to scaling and regularity assumptions.

Sources

Peter McCullagh, Tensor Methods in Statistics, author-hosted edition, Sections 1.4 and 7.3, observed information, matrix inverses and large-sample estimator covariance: https://www.stat.uchicago.edu/~pmcc/tensorbook/edition2.pdf

Meyer and Hill, Approximating Sampling Errors of (Co)variance Component Estimates, 1992, p.423, equations1–2, explicitly names formation matrix: https://www.aaabg.org/livestocklibrary/1992/ab92099.pdf

Numerical example and fictional case are original. Relevant primary source passages and equations were checked.

Previous: Formant synthesis Next: Forney factor graph