Why are residuals squared?

Squaring prevents sign cancellation and increases the relative penalty for larger errors.

Does least squares require normally distributed errors?

No. The optimization objective is defined without that assumption; particular statistical interpretations and inference procedures may add distributional assumptions.

Is mean squared error the same as the sum of squared errors?

It is the sum divided by the number of evaluated observations. Their units match, but their magnitudes differ.

AI Glossary / Mathematics and optimization / Intermediate

Least squares

Definition

Least squares is a fitting principle that chooses model parameters to minimize the sum of squared residuals. A residual is the difference between a predicted value and its observed or desired value. Squaring makes positive and negative residuals contribute nonnegative penalties and gives larger discrepancies disproportionately greater influence.

What it means

The phrase describes an objective, not a particular shape of model. A straight line can be fitted by least squares, but so can a nonlinear curve or a numerical system with several outputs. The resulting optimization problem depends on how predictions depend on the parameters.
For a fixed dataset, minimizing the sum of squared errors and minimizing their average gives the same unregularized minimizers. Their reported numerical values differ by the number of observations. Once other penalties are added, changing that scaling without adjusting the penalty coefficient can change the fitted solution.
Least squares asks which available model gives the smallest squared discrepancy. It does not establish that the model explains causation, that the measurements are unbiased, or that the same relationship will continue outside the observed setting.

How it works

Specify the prediction rule and form one residual for each observed target. Add the squared residuals to produce the objective, then choose parameters that minimize it. In linear least squares, predictions have the form Aθ, and the objective is ||Aθ−y||².
The linear problem is convex. Its minimizers satisfy Aᵀ(Aθ−y)=0, meaning that the residual is orthogonal to every column of A. Full column rank gives a unique coefficient vector. Numerical implementations often use matrix factorizations rather than explicitly forming an inverse.
Nonlinear least squares uses the same squared-residual principle with nonlinear parameter dependence and can require iterative optimization. Weighted least squares assigns different nonnegative importance to observations. In either case, the choice of residuals, weights, and parameterization is part of the problem specification.

Three practical examples

An analyst can fit a simple baseline relationship between an observed input and a numerical outcome.
A calibration exercise can choose parameters that reduce discrepancies between a model's outputs and reference measurements.
A planning model can approximate several desired numerical targets when they cannot all be matched exactly, provided squared discrepancies reflect a useful objective.

Think of it this way

Think of choosing one compromise setting for several measurements, with large misses carrying an especially heavy penalty. The analogy captures the trade-off, but least squares is not automatically the fairest compromise. Its definition of a costly miss is mathematical and must be appropriate for the application.

Worked example

Consider an original problem in which one constant prediction c must represent three measured values: 2, 4, and 9. The least-squares objective is F(c)=(c−2)²+(c−4)²+(c−9)². Expanding gives 3c²−30c+101, or equivalently 3(c−5)²+26.
The squared term is smallest at c=5, the arithmetic mean of the observations. The residuals are 3, 1, and −4. Their sum is zero, but their squared sum is 9+1+16=26. The mean squared error is 26/3, approximately 8.667, and the root mean squared error is approximately 2.944 in the original measurement units.
Compare the candidate c=4. Its squared residuals are 4, 0, and 25, giving a larger total of 29. However, its absolute-error total is 2+0+5=7, smaller than the absolute-error total 3+1+4=8 at c=5. Different loss functions can therefore favor different compromises.
Now change the third measurement from 9 to 90. The least-squares constant becomes (2+4+90)/3=32. This illustrates sensitivity to an extreme value; it does not prove the extreme value is wrong. The analyst must investigate whether the value reflects an error, a valid rare case, or a mismatch between the chosen loss and the intended use.

Illustrative case study

In a fictional packaging workshop, a team fits a constant baseline to observed preparation times before introducing a more complex predictor. It inspects the largest residuals and finds that some unusually long times correspond to legitimate custom orders.
Rather than automatically deleting those records, the team considers whether custom and standard orders require separate inputs or evaluation summaries. It also compares squared-error and absolute-error views of performance. This hypothetical workflow uses least squares as one modeling choice and diagnostic starting point, without claiming that it is universally best or that a particular improvement was measured.

Common misconceptions and mistakes

Do not assume least squares always means a straight line or that every least-squares problem has a unique parameter solution. Do not confuse minimizing residuals with making every residual zero. A nonzero optimum can be the correct mathematical answer when the model cannot match all observations.
Squaring residuals also changes units: errors measured in hours produce squared errors in hours squared. Combining residuals from different physical quantities without suitable scaling can make one quantity dominate for arbitrary unit reasons. A small training objective alone does not establish reliable prediction on new data.

Business use

Least squares offers an interpretable baseline when numerical deviations are meaningful and large misses deserve substantial weight. Teams should specify the output units, observation weights, treatment of extreme values, and evaluation dataset. If the business costs of overprediction and underprediction differ, a symmetric squared penalty may not reflect those costs.
When comparing models, separate the quality of the fit from the appropriateness of the model and objective. A more flexible model can reduce training error while making future predictions worse. Evaluation on suitable held-out data is therefore a different step from solving the least-squares optimization problem.

Check your understanding

For observations 1, 1, and 7, which constant minimizes the sum of squared residuals?

Answer: Their mean, 3. The squared-error total is 4+4+16=24.

Related terms

Learn next

Next, compare mean and median fits on a small dataset with one extreme observation. Then study linear least squares to see how the same squared-error principle extends from one constant to several model coefficients.

Linear least squares

Sources

Primary reference: Lieven Vandenberghe, ECE133A, Least Squares, original UCLA lecture slides on the objective, normal equations, and numerical solution: https://www.seas.ucla.edu/~vandenbe/133A/lectures/ls.pdf . The constant-fit example and loss comparison are original calculations.

Previous: Least privilege Next: Least-squares Analysis