AI Glossary / Probabilistic machine learning / Advanced
Statistical linear regression
Definition
Statistical linear regression, in nonlinear Bayesian estimation, approximates a relationship between uncertain variables by an affine transformation plus an error term. The affine coefficients minimize expected squared approximation error under a specified input distribution, and the error covariance accounts for variation the affine transformation cannot explain.
What it means
The word statistical matters because the approximation considers a distribution of plausible inputs. It does not simply draw a tangent at one estimated input value. Regions with more probability receive more weight when choosing the line or affine map.
This entry uses the filtering and smoothing meaning of the term, rather than treating it as another name for ordinary regression analysis. A nonlinear observation or transition model is already available; the task is to approximate how uncertainty passes through it. A different input distribution can produce a different approximation to the same nonlinear function.
How it works
For a scalar input X with positive variance, write the approximation as Y approximately aX + b + e. The least-squares slope is Cov(X,Y) divided by Var(X), and the intercept is E(Y) minus aE(X). The residual variance is Var(Y) minus a squared times Var(X).
In vectors, the same construction uses covariance matrices and a linear system. It requires finite relevant moments and care with singular input covariance. In Gaussian filtering, the residual is represented by zero-mean Gaussian noise independent of the approximating input. The true nonlinear residual need not be independent or Gaussian.
Exact moments yield matching means, covariances, and cross-covariances. Usually those moments require numerical integration, sigma points, or sampling. Consequently, a practical algorithm inherits both the distributional approximation and numerical integration error. A moment-matched surrogate is not an exact replacement for the original nonlinear probability model.
Three practical examples
1. Sensor tracking: approximate a nonlinear sensor response over the current uncertainty in an object's state. Include residual covariance so that the approximation does not pretend a straight line explains all response variation.
2. Process forecasting: approximate a nonlinear state transition before applying Gaussian prediction calculations. Recompute the approximation as the estimated operating region and uncertainty change.
3. Retrospective reconstruction: use statistically linearized relationships inside a smoother that combines observations across time. Test whether moment-based approximations adequately represent ambiguous trajectories rather than assuming a single Gaussian captures every possibility.
Think of it this way
Think of fitting a straight walkway through a curved area that people use unevenly. You choose the walkway to serve the distribution of traffic, not just to match the curve at its center. The leftover distance still matters. Statistical linear regression records that unexplained variation instead of pretending the walkway is the whole area.
Worked example
Consider an original exact-moment example. Let X be Gaussian with mean 1 and variance 2, and let Y = X squared, without additional measurement noise. Gaussian moment identities give E(X squared) = 3, E(X cubed) = 7, and E(X to the fourth power) = 25.
Therefore E(Y) = 3, Var(Y) = 25 minus 9 = 16, and Cov(X,Y) = 7 minus 1 times 3 = 4. The optimal affine slope is 4 divided by 2 = 2. Its intercept is 3 minus 2 times 1 = 1. The residual variance is 16 minus 4 times 2 = 8.
The statistical linear regression surrogate is thus Y approximately 2X + 1 + e, with independent Gaussian e of mean zero and variance 8. Its mean is 3 and its variance is 4 times 2 plus 8 = 16. Its cross-covariance with X is 2 times 2 = 4, matching all the moments just calculated.
The tangent approximation at X = 1 is instead Y approximately 2X minus 1. It predicts mean 1 and, without an error term, variance 8. The statistical surrogate still is not exact: the real squared variable cannot be negative, whereas its Gaussian marginal surrogate assigns some probability below zero.
Illustrative case study
Fictional case: Maple Robotics estimates a machine state through a curved sensor response. The team finds that a tangent approximation is sensitive to the point around which it linearizes when initial uncertainty is broad. It implements a statistical linear regression comparison using the same nonlinear sensor model and data.
The evaluation records predicted observation means, variances, and held-out tracking errors. It also checks whether the Gaussian surrogate hides two plausible state regions that produce similar readings. If ambiguity dominates, the team considers a richer state representation instead of attributing every error to tuning. This is an illustrative fictional workflow, not a reported performance result.
Common misconceptions and mistakes
Statistical linear regression does not require the original physical relationship to be linear. Matching first and second moments does not match the whole distribution or every conditional probability. The fitted residual is uncorrelated with the input under exact least squares, but this alone does not establish independence. Do not omit residual covariance or count independent measurement noise twice when constructing the output moments.
Business use
The business benefit is a tractable bridge between nonlinear models and efficient Gaussian estimation methods. Its suitability depends on the operating region, uncertainty shape, computational budget, and consequence of tail errors. A review should identify the linearization distribution and integration method and distinguish exact moment calculations from approximations. Monitoring prediction uncertainty is as important as monitoring the estimated state itself.
People also ask
How does this differ from Taylor linearization?
It minimizes an average squared error over uncertain inputs rather than matching a derivative at one point.
Must the residual be truly Gaussian?
No. Gaussian residual noise belongs to the surrogate model.
Does the approximation stay fixed?
Not generally; changing input uncertainty can change its coefficients and residual covariance.
Check your understanding
Answer: 1.
In the example, why is residual variance 8 included?
Answer: The affine part explains only 8 of the output's total variance of 16.
Answer: 2.
Does a correct output variance prove an exact output distribution?
Answer: No; the squared variable and Gaussian surrogate have different support.
Answer: 3.
What information determines the approximation beyond the nonlinear function?
Answer: The input distribution and any conditional output noise.
Related terms
- Unscented transform
- Kalman filter
- Covariance
- Statistical linearization
- Gaussian filtering
- moment matching
- posterior linearization
Learn next
Derive the scalar least-squares coefficients from moments. Then compare tangent and statistical approximations on a nonlinear transformation, and study how sigma-point integration supplies the moments needed by nonlinear filters.
More in Probabilistic machine learning
- Spectral mixture kernel - A spectral mixture kernel is a covariance function built by modeling a stationary process's frequency spectrum with a mixture of Gaussian components.
- Spike-and-slab distribution - A spike-and-slab distribution is a mixture with one component concentrated at or near zero and another spread over a broader range.
- Stationary kernel - A stationary kernel is a covariance function whose value depends on the displacement between two inputs, rather than their absolute locations.
- Stein discrepancy - A Stein discrepancy measures how strongly a candidate distribution violates identities that should hold under a target probability distribution.
- Stochastic gradient Hamiltonian Monte Carlo - Stochastic gradient Hamiltonian Monte Carlo, or SGHMC, is a posterior sampling method that combines momentum-based dynamics with gradients estimated from...
- Stochastic neighbor embedding - Stochastic neighbor embedding, or SNE, is a nonlinear dimensionality-reduction method that represents objects in a lower-dimensional space by trying to...
Sources
Särkkä and Svensson, Bayesian Filtering and Smoothing, second edition, Section 9.4, Definition 9.11 and Theorem 9.12, explains statistical linear regression and residual covariance: https://users.aalto.fi/~ssarkka/pub/bfs_book_2023_online.pdf
The Gaussian-square calculation and Maple Robotics case are original teaching examples; the case is fictional.
Independent review consulted Definition 9.11, Theorem 9.12 and Corollary 9.13 in the author-hosted PDF, read directly after the browser size limit.
Previous: Statistical learning theory Next: Statistical machine translation