Can a least-squares model include curves?

Yes. Polynomial or other known basis terms can remain linear in their coefficients, and nonlinear least squares extends the fitting criterion further.

Does R-squared measure future accuracy?

Not by itself. In-sample fit and held-out predictive performance are different assessments.

When should weights be used?

When justified by the measurement or modeling objective, such as known relative precision; arbitrary weights can change the question being answered.

AI Glossary / Statistics and probability / Intermediate

Least squares regression

Definition

Least squares regression fits a relationship between predictors and a response by minimizing the sum of squared response residuals. In ordinary linear least squares, the fitted prediction is a linear combination of coefficients and known predictor terms, and each observation receives equal weight. The method includes straight-line and multiple regression, while least squares can also fit nonlinear models. It supplies a fitted relationship and predictions; statistical inference and causal interpretation require assumptions beyond the minimization itself.

What it means

Regression uses observed associations to describe how an outcome varies with inputs. Least squares specifies how the model is fitted: large vertical discrepancies between observed and predicted responses receive greater emphasis because they are squared. Linear refers to the coefficients, so a model containing x and x squared can still be a linear regression. Under an appropriate conditional-mean model, the fitted relationship estimates an average response. It is not necessarily the best prediction for every business loss, and the fitted average should not be confused with an individual outcome.

How it works

Define the response, predictors and model form before fitting. Create the design matrix, including an intercept where appropriate, and solve the squared-error minimization with a stable numerical method. Check residual patterns, unusual observations, collinearity and whether the available data support the prediction domain. For inference, assess assumptions such as a correctly specified conditional mean and an appropriate covariance structure. For prediction, evaluate the entire modeling procedure on held-out data. Transformations, feature selection and missing-data treatment belong inside that validation process rather than being informed by the final test set.

Three practical examples

Operational planning: model average processing time as a function of workload and other measured conditions.
Calibration analysis: fit a relationship between reference quantities and observed instrument responses.
AI benchmarking: establish a transparent regression baseline before evaluating more complex prediction methods. In each case, the data collection process determines which interpretations and uncertainty calculations are defensible.

Think of it this way

Think of fitting a straight board through a cloud of points while penalizing long vertical gaps especially strongly. The board represents the chosen relationship; the fitting rule finds its best placement under squared error. A curved pattern, a few influential points or a changed operating environment can make that board a poor guide even when the calculation is exact.

Worked example

Fit a line to observations (0,1), (1,2), (2,2) and (3,5). The predictor mean is 1.5 and response mean is 2.5. The centered cross-product sum is 6 and the predictor squared-deviation sum is 5, giving slope 1.2 and intercept 0.7. Predictions are 0.7, 1.9, 3.1 and 4.3, with residuals 0.3, 0.1, -1.1 and 0.7. The squared residual sum is 1.8. The response squared-deviation sum around its mean is 9, so the ordinary in-sample R-squared is 1 - 1.8/9 = 0.8. At x = 4, the line predicts 5.5, but this lies beyond the observed predictor range. Neither the 80% in-sample figure nor the fitted line alone establishes that the extrapolated prediction is accurate. This original example demonstrates fitting and interpretation, not a validated forecasting result.

Illustrative case study

In a fictional case, Delta Fulfillment builds a linear baseline for daily packing time. Its initial fit performs well on random held-out days, but errors increase during a later seasonal surge. Analysts discover that workload conditions have shifted beyond much of the training range. They add time-aware validation, inspect residual patterns and compare a revised model with the original baseline. The team keeps the simple fit as a useful benchmark while documenting where its average predictions are unreliable.

Common misconceptions and mistakes

Least squares does not require normal errors merely to calculate coefficients. Constant error variance is relevant to some efficiency claims and conventional standard errors, but its violation does not make every fitted coefficient meaningless. High R-squared does not prove causality, correct specification or good future predictions. Residuals are vertical response discrepancies, not generally perpendicular distances to the line. Outliers and high-leverage points can strongly affect a fit. Extrapolation and omitted-variable bias remain concerns even when the optimization is solved perfectly.

Business use

Businesses benefit from the method's transparency, speed and well-developed diagnostics. Report the target quantity, predictor units, fitted range and validation results alongside coefficients. Use uncertainty estimates suited to the data structure, especially with repeated, clustered or time-dependent observations. Assess whether the business cost of error matches squared loss; a median or asymmetric-loss model may answer a different operational question. Maintain a clear separation between a predictive association and a proposed intervention effect, and monitor performance after deployment.

Check your understanding

What does R-squared of 0.8 mean in the example?

Answer: The fitted line reduces the observed squared residual total by 80% relative to predicting the sample mean. It does not mean that 80% of future predictions will be correct, that the relationship is causal or that extrapolation to x = 4 is validated.

Related terms

Learn next

Learn residual diagnostics and the difference between confidence intervals for a mean response and prediction intervals for individual outcomes. Then examine weighted and generalized least squares, regularization and validation under temporal or grouped data.

Sources

NIST/SEMATECH e-Handbook, Linear Least Squares Regression: https://www.itl.nist.gov/div898/handbook/pmd/section1/pmd141.htm . NIST, Least Squares estimation: https://www.itl.nist.gov/div898/handbook/pmd/section4/pmd431.htm . The fitted-line calculation and fictional operational case above are original illustrations.

Previous: Least-squares minimization Next: Least-squares support vector machine