Does OLS mean all residuals are small?

No. It minimizes their total squared size within the chosen model.

Are its coefficients always unique?

No. Full column rank is needed for uniqueness in the usual unconstrained formulation.

Does a slope show causation?

No. Causal interpretation needs evidence and assumptions beyond fitting a line.

AI Glossary / Mathematics and optimization / Intermediate

Ordinary Least Squares

Definition

Ordinary Least Squares, usually abbreviated OLS, fits a linear model by choosing coefficients that minimize the sum of squared residuals. A residual is the observed target minus the model's fitted value. In its ordinary form, each observation has equal weight and no coefficient penalty is added to the objective.

What it means

For one input, the fitted relationship may be written as predicted y = a + bx. The intercept a sets the baseline and the slope b describes how the prediction changes with x. With several inputs, the model combines multiple coefficients and features.

“Linear” refers to the coefficients. A model can include a prepared feature such as x² and still be linear in its fitted coefficients. OLS selects the best fit within the specified feature representation; it does not discover whether that representation is a scientifically correct explanation of the outcome.

How it works

Collect feature values in a design matrix X and targets in a vector y. If an intercept is included in the matrix formulation, one column contains ones. OLS minimizes the squared Euclidean length of y − Xw.

At an unconstrained least-squares solution, Xᵀ(y − Xw) = 0. These normal equations state that residuals are orthogonal to the columns of X. If X has full column rank, the coefficients are unique. If columns are linearly dependent, different coefficients can produce the same fitted vector, which remains the unique projection onto the column space.

Numerical implementations commonly use factorizations such as QR or singular value decomposition rather than explicitly forming an inverse. Statistical claims require additional assumptions. The minimization itself does not require Gaussian errors, while standard uncertainty calculations and particular optimality results depend on assumptions about the data and errors.

Three practical examples

1. Demand baseline: fit a transparent relationship between a numeric demand target and selected predictors before evaluating more complex models.

2. Sensor calibration: estimate a linear correction between instrument readings and reference values, checking whether the error pattern supports that form.

3. Process analysis: summarize how an outcome varies with several measured inputs, while keeping predictive association distinct from a causal effect.

Think of it this way

Think of drawing a line through a scattered set of points while charging a growing penalty for vertical misses. Squaring makes large misses especially expensive. The analogy stops at geometry: OLS minimizes vertical target residuals, not generally the shortest perpendicular distances from each point to the line, and it does not guarantee that unusual points are errors.

Worked example

Take three observations: x = 0, 1, 2 and y = 1, 2, 4. Fit an intercept and slope. The input mean is 1 and the target mean is 7/3.

The slope is the sum of centered cross-products divided by the sum of squared centered inputs. The numerator is (−1)(−4/3) + 0(−1/3) + 1(5/3) = 3. The denominator is 1 + 0 + 1 = 2. Therefore b = 3/2.

The intercept is a = 7/3 − (3/2)1 = 5/6. Predictions are 5/6, 7/3, and 23/6. Residuals are 1/6, −1/3, and 1/6, whose squared sum is 1/36 + 1/9 + 1/36 = 1/6.

The residuals sum to zero because this unconstrained fit includes an intercept. Their input-weighted sum is also zero: 0(1/6) + 1(−1/3) + 2(1/6) = 0. These checks verify the two normal equations.

For comparison, the simple line y = 1 + x gives residuals 0, 0, and 1, with squared sum 1. The fitted OLS line has smaller training error. Three invented observations are far too little evidence to claim dependable forecasting or infer a causal relationship; this is a hand-checked arithmetic example.

Illustrative case study

Fictional case: Pebble Repairs builds an OLS baseline for job duration. It includes both total work units and two component counts whose sum is exactly the total. The fitted coefficients vary across implementations because the columns are redundant, even when fitted durations agree.

The analysts remove or reparameterize the redundant representation and document what each remaining coefficient means. They inspect residuals by job type and assess predictions on later jobs. An unstable coefficient is not turned into a staffing policy merely because the training fit is good. This scenario illustrates a diagnostic workflow and reports no measured improvement.

Common misconceptions and mistakes

OLS does not require features to be statistically independent. The relevant uniqueness issue is linear dependence among design-matrix columns; near dependence can make coefficients sensitive to perturbations. Correlation is not the same as exact redundancy.

Outliers can have substantial influence because residuals are squared, and unusual feature values can also create leverage. A high training fit does not rule out omitted variables, changing relationships, or leakage. Zero-sum residuals depend on including an unpenalized intercept in the ordinary unconstrained formulation; they are not a universal property of every regression procedure.

Business use

OLS is useful as a readable baseline and a diagnostic starting point. Record feature construction, units, intercept handling, and the population represented by the data. Evaluate on appropriate held-out observations and inspect the size and pattern of errors, not just one fit score.

If observations deserve different weights, errors have problematic structure, or coefficients need stabilization, consider the corresponding weighted, robust, or regularized formulation. Those choices change the method's assumptions or objective and should be named explicitly.

Check your understanding

Adding an exact duplicate feature column leaves the possible fitted vectors unchanged. Must it leave a unique coefficient vector?

Answer: No. The same combined effect can be split between the duplicate columns in multiple ways.

Related terms

Learn next

examine residual plots and rank deficiency, then compare OLS with ridge regression on a dataset containing strongly related predictors.

Sources

Primary sources: scikit-learn, Ordinary Least Squares and related linear-model documentation: https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares ; Aalto Dictionary, orthogonality condition: https://aaltodictionaryofml.github.io/ADictML_Main.pdf . Relevant passages were read. Editorial clarification: linear independence of columns is distinguished from statistical independence of features. The worked fit is original.

Previous: Ordinal variable Next: Ordination (statistics)