Can a polynomial fit use linear least squares?

Yes, when its basis functions are fixed and the unknown coefficients enter linearly.

Are the coefficients always unique?

No. Full column rank ensures uniqueness in the ordinary unconstrained problem; dependent columns can yield multiple coefficient vectors.

Must the residual be zero at the solution?

No. The best achievable fit can retain nonzero residuals when y is outside the design matrix's column space.

AI Glossary / Mathematics and optimization / Intermediate

Linear least squares

Definition

Linear least squares chooses coefficients for a model that is linear in those coefficients by minimizing a sum of squared residuals. In matrix form, it solves minθ ||Aθ−y||², where A is the design matrix and y contains the observed targets. It has a global solution, although the coefficient vector need not be unique.

What it means

Linear refers to dependence on the unknown coefficients. A model a+bx is linear in a and b. So is a+bx+cx² in a, b, and c, even though its graph is curved as a function of x. A model whose unknown parameter appears inside an exponential generally defines a different, nonlinear fitting problem.
The design matrix packages each example's chosen features into a row. An intercept is represented by a column of ones. The coefficients combine those columns to produce the fitted target vector.
The mathematical fit is separate from a statistical interpretation. Least-squares coefficients can be computed without assuming Gaussian errors. Claims about uncertainty, unbiased estimation, or causal effects require additional assumptions and appropriate data.

How it works

Construct A and y with matching row order. The objective's gradient is 2Aᵀ(Aθ−y), so every minimizer satisfies the normal equations AᵀAθ=Aᵀy. These equations express that the fitted residual is orthogonal to the column space of A.
Geometrically, the fitted values Aθ are the orthogonal projection of y onto that column space. Full column rank gives a unique coefficient vector. If columns are dependent, different coefficients can produce the same fitted values.
The normal equations describe the solution but are not always the best computational route. Forming AᵀA can worsen numerical conditioning. QR factorization or singular value decomposition can solve the problem while handling numerical concerns more carefully. Constraints or regularization change the optimization problem and should be reported explicitly.

Three practical examples

An analyst can fit an intercept and slope as a baseline numerical predictor.
A calibration model can fit coefficients of fixed basis functions, including polynomial terms, while remaining linear in its parameters.
A numerical application can find the closest achievable output to a desired vector when an overdetermined system cannot be satisfied exactly.

Think of it this way

Think of mixing a fixed collection of building blocks to approximate a target shape. The unknowns are how much of each block to use. The analogy explains linear combinations, but coefficients can be negative, the blocks are numerical vectors, and the closest fit is defined by Euclidean squared error rather than visual resemblance.

Worked example

Consider an original line-fitting problem with observations (x,y)=(0,1), (1,2), and (2,4). Use the model ŷ=a+bx. The design matrix has rows (1,0), (1,1), and (1,2), and the target vector is (1,2,4)ᵀ.
The normal equations are 3a+3b=7 and 3a+5b=10. Subtracting gives 2b=3, so b=1.5. Substituting back gives a=5/6. The fitted values are 5/6, 7/3, and 23/6.
Using residuals as fitted minus observed, the residual vector is (−1/6,1/3,−1/6). Its components sum to zero, confirming orthogonality to the intercept column. Their x-weighted sum is 0×(−1/6)+1×(1/3)+2×(−1/6)=0, confirming orthogonality to the second column. The residual sum of squares is 1/36+1/9+1/36=1/6.
At x=3, the fitted line predicts 5/6+4.5=16/3, approximately 5.333. This is extrapolation beyond the observed x values, so the algebraic prediction is not evidence that the relationship remains valid there.
Finally, replace the intercept a with two coefficients α and β multiplying identical columns of ones. Any pair satisfying α+β=5/6 gives the same fitted line when b=1.5. The fitted values remain unique, but the separate intercept coefficients are not identified. A numerical routine's choice of one pair does not create information that the data lack.

Illustrative case study

In a fictional repair-planning team, an analyst fits duration using a constant term, item count, and a duplicated version of item count accidentally introduced during preprocessing. The fit returns plausible predictions, but individual coefficients vary greatly between implementations.
The team checks the design matrix, identifies the redundant column, and revises the representation before interpreting coefficients. It evaluates predictions on separate observations and records the range of inputs represented in training. The scenario illustrates a modeling and numerical review, not a claimed improvement in a deployed business system.

Common misconceptions and mistakes

Do not assume that linear least squares always means fitting a straight line in the original input. Do not infer unique coefficients from unique fitted values. More observations than parameters also does not guarantee full column rank if the columns are redundant.
Avoid treating an explicit matrix-inverse formula as the preferred implementation. Do not assume residuals always sum to zero: that property follows when the unweighted unconstrained design includes an intercept column. Changing weights, constraints, or penalties can change the relevant optimality conditions. A good fit also does not establish a causal relationship between inputs and targets.

Business use

Linear least squares can provide a useful, understandable baseline for numerical prediction and approximation. Document the feature construction, units, intercept convention, and data range. Review extreme residuals and the stability of coefficients when features are strongly related.
If the application requires nonnegative coefficients, a fixed budget, or a penalty on complexity, formulate those requirements explicitly instead of assuming ordinary least squares enforces them. Compare performance on appropriate evaluation data, and separate uncertainty about the model's usefulness from numerical precision in solving its training objective.

Check your understanding

A design matrix contains two identical columns. Can ordinary least squares separately identify their coefficients from predictions alone?

Answer: Generally no. Shifting an amount from one coefficient to the other leaves their combined contribution unchanged.

Related terms

Learn next

Next, verify the normal equations for a small fitted line and inspect its residual vector. Then study QR factorization and rank deficiency to understand why a correct formula and a reliable numerical implementation are different parts of the solution.

Sources

Primary reference: Lieven Vandenberghe, ECE133A, Least Squares, original UCLA slides on projection, normal equations, and QR solution: https://www.seas.ucla.edu/~vandenbe/133A/lectures/ls.pdf . The three-point line fit and duplicate-column example are original calculations.

Previous: Linear inverse problem Next: Linear light