AI Glossary / Statistics and probability / Beginner

Regression line

Definition

A regression line is a straight-line representation of a modeled relationship between a numerical outcome and a predictor. In ordinary simple linear regression, the fitted line estimates the outcome's conditional mean as an intercept plus a slope times the predictor. Ordinary least squares chooses the intercept and slope to minimize the sum of squared vertical residuals. Other regression criteria can produce different lines, so the fitting method should be stated rather than assumed from the appearance of a graph.

What it means

The line summarizes how the model's expected outcome changes across predictor values. Observations generally lie above and below it because of unexplained variation, measurement and model limitations. A regression line is not the same as a line joining observed points, and it does not describe every individual outcome exactly. Its slope has outcome units per predictor unit. The intercept describes the fitted value at predictor zero, which may be outside the observed range.

How it works

Plot the data and decide whether a straight-line relationship is a useful approximation for the intended purpose. Estimate the slope and intercept using the selected criterion. For ordinary least squares with an intercept, the slope is the centered cross-product sum divided by the predictor's centered squared sum, provided the predictor varies. Calculate fitted values and residuals, then inspect patterns and influential observations. Add appropriate uncertainty summaries when inference or prediction is required, and evaluate predictions separately from training fit.

Three practical examples

Example 1: An analyst uses a line to summarize how completion time changes with task size over a limited range. Example 2: A laboratory fits a calibration relationship between an instrument reading and a reference quantity, while checking measurement assumptions. Example 3: A planner compares a straight-line baseline with a more flexible predictive model. A line can be useful as an approximation without implying that the relationship remains linear beyond the observed conditions.

Think of it this way

Think of balancing a straight guide through a cloud of points according to a specified rule. Ordinary least squares balances squared vertical misses, so large residuals receive substantial weight. It does not minimize every possible notion of distance to the line. The chosen rule reflects which variable is treated as the outcome and what kind of prediction error the analysis is designed to reduce.

Worked example

Take x = [0, 1, 2, 3] and y = [1, 2, 2, 5]. Their means are 1.5 and 2.5. The centered predictor squared sum is 5 and the centered cross-product sum is 6, so the least-squares slope is 6/5 = 1.2. The intercept is 2.5 - 1.2 × 1.5 = 0.7. The fitted line is y-hat = 0.7 + 1.2x, producing predictions [0.7, 1.9, 3.1, 4.3]. Residuals are [0.3, 0.1, -1.1, 0.7], summing to zero, and their squared sum is 1.8. The line passes through (1.5, 2.5), the point of sample means. These last properties hold for ordinary least squares with an intercept, not for every possible fitted line.

Illustrative case study

Fictional case study: Hazel Planning fits a line between task size and completion time. The scatterplot suggests curvature for the largest tasks, and residuals increase systematically in that region. Analysts retain the line as a limited baseline while testing alternatives and clearly marking its supported range. This fictional case illustrates diagnostic reasoning, not evidence that a particular nonlinear model is superior or that the observed relationship is causal.

Common misconceptions and mistakes

Common mistakes include treating the line as a guarantee for individual observations, assuming correlation establishes causation, and extrapolating far beyond the observed predictor values. Reversing outcome and predictor generally gives a different least-squares line, not the algebraic inverse of the original fit. Extreme or influential observations can change the slope substantially. A line fitted without an intercept need not pass through the sample means or have residuals summing to zero. Visual fit alone does not validate uncertainty calculations.

Business use

Business use: Regression lines provide simple baselines and concise summaries when a linear approximation is defensible. Show the data alongside the line so readers can see spread, gaps and unusual observations. Label units and distinguish a confidence band for the mean from a prediction interval for an individual outcome. Compare the line's predictive error with alternatives and limit its use to conditions supported by data and relevant process knowledge.

Check your understanding

Why does the worked line pass through the sample means?

Answer: That is a property of ordinary least squares when an intercept is included.

Does the residual -1.1 mean the prediction was too low?

Answer: No. Residuals are observed minus fitted, so the negative value means the fitted prediction exceeded the observed outcome by 1.1.

Related terms

Slope describes the line's change per predictor unit
while prediction intervals concern future observations

Learn next

Calculate the least-squares line for a tiny dataset and verify its residual sum and passage through the means. Add an unusual observation and inspect how the fit changes. Then reverse the regression direction and compare the resulting line. Study residual plots before using a straight-line model outside the narrow range directly supported by the observations.

Least squares

Sources

Sources: University of Vermont, Statistical Methods glossary, Regression line: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html ; Penn State STAT 501, Simple Linear Regression: https://online.stat.psu.edu/stat501/Lesson01 . These educational sources explain fitted lines and least-squares interpretation. The four-point derivation, residual checks and fictional task-planning example are original illustrations.

Previous: Regression fallacy Next: Regression model