Is the column space always the whole output space?

No. It is the whole m-dimensional space only when the matrix has rank m.

Can a column be removed without changing it?

Yes, if that column is a linear combination of the remaining columns.

What if the target is outside the column space?

Exact equality Ax=b is impossible. Unconstrained Euclidean least squares finds its closest attainable projection instead.

AI Glossary / Mathematics and optimization / Intermediate

Column space

Definition

The column space of a matrix is the set of all linear combinations of its columns. For a real m-by-n matrix A, it is the set {Ax:x is an n-dimensional real vector}. It is a subspace of the m-dimensional output space and describes every output the matrix transformation can produce.

What it means

Multiplying a matrix by a vector gives each column a coefficient and adds the resulting scaled columns. The column space contains every vector obtainable in this way, allowing positive, negative, or zero coefficients.

This is a capability statement about a linear transformation. A target inside the column space can be represented exactly as Ax. A target outside it cannot, whatever coefficient vector is chosen. In a linear prediction model, this gives a geometric way to understand which fitted-value patterns the available features can express across the observed cases.

How it works

Start with the columns of A and consider their span. Redundant columns do not add new directions: if a column is already a linear combination of others, removing it leaves the column space unchanged. The number of independent column directions is the matrix rank.

The equation Ax=b has at least one solution exactly when b belongs to the column space. Whether that solution is unique is a different question, governed by dependencies among the columns, or equivalently the null space.

When b lies outside the column space, ordinary unweighted least squares seeks the attainable vector closest to b in Euclidean distance. The fitted vector is the orthogonal projection of b onto the column space, and the residual is perpendicular to every column. This statement concerns unconstrained least squares; additional coefficient restrictions can change the set of permitted fitted vectors.

Three practical examples

1. Linear regression: understand why a feature matrix may be unable to reproduce every possible target pattern, even when its coefficients are optimized.

2. Feature design: determine whether a newly added column introduces a new direction or merely duplicates information already present linearly.

3. Equation solving: test whether a requested output can be achieved by a linear system and distinguish representability from uniqueness of the coefficients.

Think of it this way

Think of a set of building directions. You may travel any positive or negative amount along each direction and combine those movements. The column space is the collection of destinations reachable by those combinations.

The analogy assumes unrestricted real coefficients. If a practical problem allows only positive amounts, bounded amounts, or discrete choices, its feasible destinations may be smaller than the column space. The linear span alone does not encode those extra rules.

Worked example

Let A have columns c_1=(1,0,1) and c_2=(0,1,1). For coefficients x_1 and x_2, Ax=(x_1,x_2,x_1+x_2). Its column space is therefore the plane of vectors whose third coordinate equals the sum of the first two.

The target (2,3,5) is attainable with coefficients (2,3). The target b=(2,3,6) is not attainable, because 6 differs from 2+3.

For the second target, the least-squares equations are 2x_1+x_2=8 and x_1+2x_2=9. Solving gives x_1=7/3 and x_2=10/3. The fitted vector is p=(7/3,10/3,17/3), and the residual b−p is (−1/3,−1/3,1/3).

Its dot product with each column is zero: (−1/3)+1/3=0. The squared residual length is 1/9+1/9+1/9=1/3. This explicitly checks that the error lies perpendicular to the attainable plane, rather than merely asserting that the fitted vector looks close.

Illustrative case study

Fictional case: a team adds a feature equal to the sum of two existing features and expects its ordinary linear model to gain a new ability to fit outcomes.

An analyst examines the design matrix and shows that the added column lies in the existing column space. With unrestricted coefficients, the set of attainable fitted vectors has not expanded. The team then distinguishes representational capacity from other effects: regularization or numerical implementation can behave differently under a redundant parameterization. The conclusion is specific to the linear span, not a claim that all feature engineering is ineffective.

Common misconceptions and mistakes

Do not confuse column space with row space. For an m-by-n matrix, column vectors live in an m-dimensional space, while rows live in an n-dimensional space.

More columns do not necessarily mean a larger column space. Nor does an exactly representable target guarantee unique coefficients. A rank-deficient matrix can have several coefficient vectors producing the same fitted vector. Finally, the column space of an observed design matrix describes patterns across those observed rows; it does not by itself establish predictive performance on new cases.

Business use

The concept helps explain limitations and redundancies in linear models without relying only on a final error score. It can guide discussions about whether additional measurements add a new linear direction and whether unstable coefficients reflect redundant features.

For decision makers, ask whether the goal is accurate predictions, interpretable coefficients, or both. Unique fitted values can coexist with nonunique coefficients. Representation, numerical stability, and generalization are separate issues, so a column-space analysis complements rather than replaces model evaluation.

Check your understanding

A matrix has three columns, but the third equals twice the first. Does the third necessarily increase its rank?

Answer: No. It adds no new direction to the span. The rank is determined by independent columns, not the raw column count.

Related terms

Least squares
Linear span
matrix rank
linear independence
row space
null space
orthogonal projection
design matrix

Learn next

connect the column space to the solvability of Ax=b, then contrast it with the null space. Work through a small least-squares projection and verify that its residual is orthogonal to every column.

Least squares

Sources

Aalto Dictionary of Machine Learning, column space entry: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . The span definition and output-space dimensions were read. The plane representation, least-squares solution, orthogonality checks, and fictional feature example are original teaching derivations.

Previous: Column Next: Combinations