Does weighted least squares require a straight-line model?

No. The weighted squared-error criterion can also fit nonlinear models.

What does a zero weight do?

It removes that observation's residual from the objective, although other processing may still use the observation.

Do weights have to add to one?

No. Their relative sizes determine the unregularized fit; normalization affects the reported objective scale.

AI Glossary / Mathematics and optimization / Intermediate

Weighted least squares

Definition

Weighted least squares fits a model by minimizing a sum of squared prediction errors in which each observation has a nonnegative weight. Larger weights give a mismatch greater influence on the fitting objective. The method can reflect differing measurement precision or other explicitly chosen priorities. It is a fitting criterion that can be used with linear or nonlinear models, not a guarantee that unusual observations will be handled robustly.

What it means

Ordinary least squares gives each squared residual the same multiplier. Weighted least squares changes that balance. If two observations have the same residual but weights 4 and 1, the first contributes four times as much to the objective.
One statistical motivation is unequal error variance. Under an appropriate model, more precise observations can receive larger inverse-variance weights. Other applications use importance or confidence weights, but these have different interpretations and need their own justification.
The observation weights are distinct from the model's fitted coefficients. The weights normally enter the chosen fitting problem as specified values; the optimization then determines the coefficients that best satisfy that weighted objective.

How it works

For targets y[i], predictions f(x[i];beta), and weights w[i], minimize the sum of w[i](y[i]−f(x[i];beta))^2. Dividing the entire unregularized objective by a positive constant does not change its minimizers.
For a linear model X beta, this is equivalent to ordinary least squares after multiplying each row of X and the corresponding target by sqrt(w[i]). The square root matters: multiplying residuals by w[i] instead would square the intended weights.
With positive weights and a full-column-rank design matrix, the linear weighted least-squares fit has a unique coefficient solution. Nonlinear models can introduce local minima. If weights are estimated rather than known, their uncertainty and sensitivity should be considered when interpreting the fitted model and statistical summaries.

Three practical examples

A calibration study can give more precise measurements greater influence when their error variances are supported by a suitable measurement model.
A prediction exercise can emphasize selected observations through documented importance weights and evaluate whether that objective matches the intended use.
An alternating matrix-factorization algorithm can solve weighted least-squares subproblems while holding the other factor matrix fixed.

Think of it this way

Think of balancing a fitted line against observations that carry different amounts of influence. A heavier observation pulls harder when the model misses it by the same amount. The analogy captures unequal influence, but a heavy weight does not make an observation true, nor does a light weight justify ignoring an inconvenient result without evidence.

Worked example

Fit a single constant c to three observed values [10,14,20] with weights [4,1,1]. The objective is J(c)=4(10−c)^2+(14−c)^2+(20−c)^2.
Differentiating and setting the derivative to zero gives 4(c−10)+(c−14)+(c−20)=0, or 6c=74. The fitted constant is therefore 37/3, approximately 12.333. This is the weighted mean.
At this value, the squared residuals are 49/9, 25/9, and 529/9. After weighting, the objective is (196+25+529)/9=250/3, approximately 83.333.
The unweighted mean is 44/3, approximately 14.667. Evaluated under the same weighted objective, it gives 116, so it is worse for this particular fitting criterion. That does not establish that the weighted mean is better for every application; the weights define which trade-off is being optimized.
Multiplying all three weights by 10 leaves the fitted constant unchanged while multiplying the objective by 10. If a separate penalty is added without rescaling, however, that change can alter the balance between fit and regularization.

Illustrative case study

In a fictional sensor-calibration project, one instrument is believed to be more precise, so its readings receive greater weight. A diagnostic plot reveals that this instrument also has a systematic offset.
The team investigates and corrects the measurement process rather than assuming larger weights will solve the inconsistency. It then revisits the variance estimates and checks the fitted relationship against independent calibration cases.
The lesson is that precision and bias are different. A consistently wrong reading can be highly repeatable, and assigning it more influence can make a fit worse. The scenario does not report a measured calibration improvement.

Common misconceptions and mistakes

Do not automatically assign low weight to every large residual. A large residual can signal a poor model, a meaningful rare case, or a data issue; choosing weights based on residuals becomes an additional procedure that needs analysis.
Do not assume fixed-weight least squares is inherently resistant to outliers. A large-weight outlier can have substantial influence.
Inverse-variance weighting uses variance, not standard deviation, in the denominator. Also distinguish nonnegative fitting weights from negative model coefficients; negative observation weights generally destroy the usual least-squares interpretation.
Finally, a lower weighted objective cannot be fairly compared with a differently weighted objective without accounting for the changed definition.

Business use

For business use, document why observations receive different weights and how those weights were estimated. Evaluate sensitivity to plausible alternatives, especially when a few observations dominate the fit.
Keep fitting weights separate from evaluation choices. A model optimized for one weighted population may not be best under an unweighted metric or another group's needs. Report the actual target of the analysis and relevant subgroup performance rather than presenting the weighted loss as an all-purpose measure of quality.

Check your understanding

Two residuals both equal 3, but their weights are 2 and 0.5. What are their contributions to the weighted squared-error objective?

Answer: They contribute 2×9=18 and 0.5×9=4.5. The first contribution is four times the second.

Related terms

Least squares
Residual
Linear regression
Regularization
Robust regression
observation weight
heteroscedasticity
inverse variance
weighted mean

Learn next

Derive the weighted-mean formula for a constant model and verify the worked example. Then compare multiplying residuals by sqrt(w) with multiplying them by w. Study how weights are estimated before relying on inverse-variance interpretations or uncertainty intervals.

Sources

Aalto Dictionary of Machine Learning, weighted least squares entry. https://aaltodictionaryofml.github.io/ADictML_Main.pdf NIST Engineering Statistics Handbook, Weighted Least Squares criterion. https://www.itl.nist.gov/div898/handbook/pmd/section4/pmd432.htm NIST, Weighted Least Squares Regression: precision weighting, estimated-weight limitations, and outlier sensitivity. https://www.itl.nist.gov/div898/handbook/pmd/section1/pmd143.htm

Previous: Weighted graph Next: Weighted linear function