Must the residual be zero?

No. It is zero only when the target already belongs to the subspace.

Can different coefficients describe the same projection?

Yes, when the spanning columns are linearly dependent.

Does orthogonality mean statistical independence?

No. Here it is a geometric relation defined by an inner product.

AI Glossary / Mathematics and optimization / Advanced

Orthogonality condition

Definition

The orthogonality condition characterizes the nearest-point projection onto a linear subspace: the projected point lies in the subspace, and the remaining error is perpendicular to every direction in that subspace. In ordinary Euclidean coordinates, perpendicular means that the relevant dot product is zero. This condition is central to least-squares fitting and geometric approximation.

What it means

Suppose a vector contains the information we want to approximate, but our model can produce vectors only inside a particular linear subspace. Projection finds the allowed vector closest to the target in Euclidean distance.

At the projection, the leftover error has no component along any allowed subspace direction. Otherwise, moving slightly in a suitable allowed direction would shorten the error. The condition therefore explains why the best approximation leaves precisely the part that the chosen representation cannot express.

Here the term refers to subspace projection, as in the supplied glossary. Other mathematical uses of orthogonality should be interpreted in their own context.

How it works

Let z be the target vector and W the subspace. A candidate p is its projection if p belongs to W and vᵀ(z − p) = 0 for every v in W. Checking only that p itself is perpendicular to the residual is generally insufficient; all subspace directions must be covered.

If the columns of a matrix A span W, the condition becomes Aᵀ(z − p) = 0. Writing p = Aw gives the normal equations AᵀAw = Aᵀz. Redundant columns can make the coefficient vector w nonunique even though the projected vector p is unique.

The inner product matters. With a different positive-definite weighting, the relevant perpendicularity changes. Also, projecting onto a general convex set has an inequality condition involving feasible directions, rather than the equality for every direction that holds for a linear subspace.

Three practical examples

1. Regression checking: verify that ordinary least-squares residuals are perpendicular to the design-matrix columns, up to numerical error.

2. Signal approximation: represent a signal using a selected linear span of basis signals and identify the residual excluded by that representation.

3. Dimensionality reduction: reconstruct data from an orthogonal principal-component subspace and separate the retained component from the reconstruction error.

Think of it this way

Think of lowering a point directly onto a flat floor. The shortest connecting segment meets the floor at a right angle. This captures the geometry of projection onto an affine plane. Curved surfaces and restricted patches can behave differently, especially at boundaries, and a different distance measure changes what “straight down” means mathematically.

Worked example

Let W consist of all vectors (t, t, s), with real t and s. It is spanned by u = (1, 1, 0) and v = (0, 0, 1). We want to project z = (4, 0, 3) onto W.

Choose p = (2, 2, 3). It lies in W. The residual is r = z − p = (2, −2, 0). Its dot products with the spanning vectors are rᵀu = 2 − 2 = 0 and rᵀv = 0. Therefore it is perpendicular to every linear combination of u and v, establishing the orthogonality condition.

We can verify minimality directly. The squared distance from z to a general point (t, t, s) is (4 − t)² + t² + (3 − s)². Completing the square gives 2(t − 2)² + (s − 3)² + 8.

The minimum is 8, attained at t = 2 and s = 3. Thus the distance is the square root of 8. The alternative point (1, 1, 3) has squared distance 10, exceeding the optimum by 2.

Finally, the zero vector is in W and has zero dot product with every residual, but it is not the projection of z. This shows why checking only pᵀ(z − p) = 0 can be misleading. The full spanning set supplies the necessary directions. All calculations are hand-checked teaching examples.

Illustrative case study

Fictional case: Meadow Instruments compresses three-channel readings using a two-dimensional linear representation. A developer checks that each reconstruction belongs to the intended span but does not check the residual.

The team adds dot-product checks against the basis vectors and verifies reconstruction distances on a small known example. It also discovers that one implementation used scaled channels while another used raw channels. They agree on the inner product and units before comparing results. Passing the geometric check establishes correct projection for that representation; it does not establish that the retained information is adequate for every downstream task.

Common misconceptions and mistakes

Orthogonal vectors need not be unit vectors. Orthogonality and normalization are separate properties. Likewise, a basis need not be orthonormal for the residual condition to hold, although an orthonormal basis simplifies projection formulas.

Do not apply the subspace equality blindly to inequality-constrained projection. For example, projecting the number 2 onto [0, 1] gives 1, leaving residual 1. The feasible direction from 1 toward 0 is −1, whose product with the residual is −1, not zero. The endpoint is still the closest allowed point.

Business use

The condition gives teams an interpretable diagnostic for least-squares and reconstruction code. Check membership in the intended subspace, residual dot products, and the units defining distance. Use tolerances appropriate to scale and numerical precision instead of requiring exact computer zeros.

Keep the check separate from business validation. A mathematically exact projection can discard a small but important signal if that signal lies outside the chosen subspace.

Check your understanding

A residual has zero dot product with every vector in a basis of W. Is it perpendicular to all of W?

Answer: Yes. Every vector in W is a linear combination of the basis vectors, and the dot product distributes over that combination.

Related terms

Inner product
Least squares
Normal equations
Residual
projection
linear subspace
orthonormal basis

Learn next

derive the normal equations from the projection condition, then compare subspace projection with projection onto a constrained convex set.

Sources

Primary source: Aalto Dictionary of Machine Learning, orthogonality condition and projection entries: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . The exact subspace characterization was read. This entry explicitly retains that geometric scope; the worked construction, boundary counterexample, and fictional case are original.

Previous: Orthogonal Matching Pursuit Next: Orthographic projection