Why do residuals sum to zero?

In ordinary unweighted least squares with an intercept column, orthogonality to that all-ones column implies the sum is zero.

What changes for weighted least squares?

With a suitable weight matrix W, the equations become XᵀWX beta = XᵀWy.

Is the normal equation matrix itself a normal matrix?

XᵀX is real symmetric and therefore normal in the linear-algebra sense, but the terms describe different concepts.

AI Glossary / Statistics and probability / Intermediate

Normal equations

Definition

The normal equations are the linear equations that characterize a least-squares solution. For a real design matrix X, outcome vector y, and coefficient vector beta, ordinary least squares satisfies XᵀX beta = Xᵀy. They express that the residual vector is orthogonal to every column of the design matrix. The word normal refers to this perpendicularity, not to an assumption that the observations follow a normal distribution.

What it means

Least squares seeks predictions X beta that are as close as possible to y in squared Euclidean distance. When an exact fit is unavailable, the best fitted vector is the projection of y onto the space spanned by X's columns.

The normal equations describe that projection algebraically. They can be useful for understanding regression, deriving formulas, and checking solutions. However, forming and directly solving the equations is not always the best numerical strategy. Modern software can solve the underlying least-squares problem through factorizations that handle numerical difficulties more reliably.

How it works

Write the squared-error objective as the squared norm of y − X beta. Differentiating with respect to beta gives the gradient 2Xᵀ(X beta − y). Setting it to zero produces the normal equations. Since the objective is convex, solutions to these equations minimize it.

If X has linearly independent columns, XᵀX is invertible and the coefficient solution is unique. If the columns are dependent, several coefficient vectors can produce the same optimal fitted values. The normal equations remain meaningful, but an inverse formula is unavailable.

For computation, choose a solver appropriate to rank, dimensions, sparsity, and conditioning. QR or singular-value decompositions can avoid explicitly constructing an inverse. In the full-column-rank case, forming XᵀX squares the two-norm condition number, which can amplify numerical sensitivity. A mathematically valid identity therefore should not automatically be used as a literal implementation recipe.

Three practical examples

An analyst derives the intercept and slope of a simple least-squares regression from two normal equations.
An AI team checks that a fitted linear model's residual is approximately orthogonal to its training features, allowing for numerical tolerance.
A data engineer detects duplicate predictor columns and explains why the fitted values can be unique even when coefficient estimates are not.

Think of it this way

Think of finding the closest point on a flat surface to a point above it. The shortest connecting line is perpendicular to that surface. Least squares performs the same geometric operation in a space with as many coordinates as there are observations.

Worked example

Consider an original dataset with predictor values 0, 1, 2 and outcomes 1, 2, 4. Fit y approximately equal to a + bx. The design matrix has rows [1,0], [1,1], and [1,2], where the first column represents the intercept.

Here XᵀX = [[3,3],[3,5]] and Xᵀy = [7,10]. The normal equations are 3a + 3b = 7 and 3a + 5b = 10. Subtracting gives 2b = 3, so b = 1.5 and a = 5/6, approximately 0.8333.

The fitted values are 5/6, 7/3, and 23/6. Observed minus fitted residuals are 1/6, −1/3, and 1/6. Their sum is zero, and their predictor-weighted sum is 0 × 1/6 + 1 × (−1/3) + 2 × 1/6 = 0. Thus the residual is orthogonal to both design columns. Its squared norm is 1/36 + 1/9 + 1/36 = 1/6. The example demonstrates least-squares geometry; it makes no claim that these three invented observations support a dependable population regression.

Illustrative case study

Fictional case study: Maple Analytics adds a second copy of a predictor to a reporting model. An implementation based on directly inverting XᵀX fails, and the team initially interprets the failure as evidence that no regression fit exists.

An analyst identifies the redundant column and uses a rank-aware least-squares solver. The fitted predictions remain well-defined, while the report explains that separate coefficients for identical columns cannot be uniquely identified without an additional convention or constraint. Maple removes the unnecessary duplication from its reporting model. The fictional case distinguishes failure of an inverse formula from failure of the underlying least-squares problem.

Common misconceptions and mistakes

“Normal equations require normal data.” Their algebra follows from squared-error minimization, independently of a normal-distribution assumption.

“Always calculate an explicit inverse.” Solving a system or using an appropriate factorization is generally preferable numerically.

“Singularity means there is no least-squares solution.” A finite linear least-squares problem has a solution; singularity can make its coefficients nonunique.

“Orthogonal residuals prove a good model.” They are an in-sample fitting property and do not establish causal validity or future predictive quality.

Business use

Understanding the normal equations helps businesses interpret why redundant features and poorly scaled variables can destabilize linear-model coefficients. It also provides a useful distinction between exact mathematical formulas and dependable numerical implementations.

When regression results change dramatically after small data edits, inspect conditioning, rank, and feature definitions before treating every coefficient change as a substantive discovery. Record the solver and any regularization. A regularized problem modifies the objective and corresponding equations, so its coefficients should not be described as an ordinary least-squares solution without qualification.

Check your understanding

If two columns of X are identical, can X have full column rank?

Answer: No. One column is redundant, so the coefficients are not uniquely determined by ordinary least squares. Different coefficient allocations can produce the same fitted vector.

Related terms

Least squares
Singular value decomposition
Condition number
design matrix
projection
orthogonality
matrix rank
QR decomposition

Learn next

Derive the two equations for a straight-line fit, then express them in matrix notation. Compare a full-rank design with one containing duplicate columns. Next, study QR and singular-value methods and why stable numerical computation differs from symbolically writing an inverse.

Sources

The Aalto Dictionary of Machine Learning, normal equations and their orthogonality interpretation: https://aaltodictionaryofml.github.io/ADictML_Main.pdf

Boyd and Vandenberghe, Introduction to Applied Linear Algebra, least-squares chapters: https://web.stanford.edu/~boyd/vmls/vmls.pdf

The small regression calculation and Maple Analytics case are original teaching illustrations.

Previous: Normal distributions transform Next: Normal form