AI Glossary / Statistics and probability / Foundation
Simple linear regression
Definition
Simple linear regression models the mean of one response variable as a straight-line function of one predictor. Its usual form is Y = intercept + slope times X + error, with the error having conditional mean zero under the model. The slope describes the change in the modeled mean response for a one-unit increase in the predictor. Simple refers to the single predictor; linear describes the functional form. The model does not automatically establish a causal relationship.
What it means
A fitted line summarizes an association and can support predictions within a relevant range. Observations usually fall above and below the line because the predictor does not explain everything. The intercept gives the fitted mean at predictor value zero, which may be outside the meaningful range. The slope has response-units per predictor-unit. Its interpretation depends on the data-generating context, possible confounders and whether a straight-line relationship is an adequate description of the conditional mean.
How it works
Inspect the data and define the response and predictor. Ordinary least squares chooses the intercept and slope that minimize the sum of squared vertical residuals. With an intercept and nonconstant predictor, the slope is the sum of centered predictor-response products divided by the sum of squared centered predictor values; the intercept makes the line pass through the sample means. Examine residual patterns, unusual influence and the study design. Inference requires additional assumptions or suitable adjustments for unequal variance, dependence and other departures.
Three practical examples
Example 1: A workshop models average processing time against the number of items in a small batch. Example 2: An AI team uses a one-predictor regression as a transparent baseline for predicting resource consumption. Example 3: A household examines the association between heating use and outdoor temperature over comparable periods. Each application needs a plausible range and measurement definition. An observed slope can reflect omitted variables or time patterns rather than a direct effect of changing the predictor.
Think of it this way
Think of drawing a straight guide through a cloud of points to describe their average direction. The guide is a summary, not a rail that every observation must follow. Least squares selects the guide by penalizing squared vertical misses, so large residuals receive more weight. Looking only at the line can hide a curved pattern, changing spread or one influential point that determines much of its apparent direction.
Worked example
Take predictor values 1,2,3,4 and responses 2,3,5,6. The means are 2.5 and four. The centered predictor sum of squares is five, and the centered cross-product sum is seven. The least-squares slope is 7/5 = 1.4, and the intercept is 4 - 1.4 times 2.5 = 0.5. Fitted responses are 1.9,3.3,4.7 and 6.1; residuals are 0.1,-0.3,0.3 and -0.1. Their squared sum is 0.20. The response's total squared deviation from its mean is ten, giving R-squared = 1 - 0.20/10 = 0.98. At X = 5, the line predicts 7.5, but this is outside the observed range and needs an extrapolation warning. Four points and a high in-sample R-squared do not establish reliable future performance or causality.
Illustrative case study
In this fictional case study, Fieldstone Computing predicts processing time from document length. A simple linear regression provides an interpretable starting point for comparison with an AI model. Residual plots reveal that very long documents behave differently, so the team limits the baseline's interpretation and tests alternatives on held-out documents. It reports prediction errors across the relevant length range rather than relying on a strong training-set fit. The slope is described as an observed model association, not proof that length alone determines processing time.
Common misconceptions and mistakes
Common mistakes include equating a high R-squared with accurate future predictions, interpreting the slope causally without a suitable design, and extrapolating far beyond the data. Another error is assuming normality of the predictor is required. Normal errors are used for some exact small-sample inference, but fitting the least-squares line itself does not require normal observations. Residuals and unobserved errors are also different: residuals are calculated after fitting and are constrained by that fit.
Business use
Businesses use simple linear regression for understandable baselines, rough planning relationships and initial diagnostics. It can expose whether more complex AI methods add useful predictive value. Evaluate on representative data and compare against a sensible benchmark. For individual predictions, use an appropriate prediction interval rather than an interval for the mean alone. If data are collected over time or within groups, account for that dependence. A compact model is valuable when its assumptions and operational range are made explicit.
People also ask
Does simple linear regression require a perfect straight-line pattern?
No; the line models a conditional mean with variation around it.
Is correlation the same as the regression slope?
No; correlation is unitless and symmetric, while the slope depends on which variable is the response and on their units.
Can the intercept be meaningless in practice?
Yes, if zero is outside the relevant predictor range.
Does least squares resist extreme points automatically?
No; influential observations can strongly affect the fit.
Check your understanding
What fitted equation describes the worked example?
Answer: Predicted Y = 0.5 + 1.4X.
What is the residual when X = 2 and Y = 3?
Answer: 3 - 3.3 = -0.3.
What does R-squared of 0.98 describe here?
Answer: The fraction of observed response variation explained by this fitted line in this dataset.
Why treat the prediction at X = 5 cautiously?
Answer: It extrapolates beyond the observed range of one to four.
Related terms
- Intercept
- Residual
- Conditional means
- Correlation
- Related terms include ordinary least squares
- slope
- conditional mean
- coefficient of determination and prediction interval
Learn next
Learn next how residual plots reveal curvature, unequal variance and unusual observations. Compare confidence intervals for the mean response with prediction intervals for individual outcomes. Then study held-out evaluation and the assumptions needed for causal interpretation, since a mathematically correct fitted line can still answer the wrong practical question.
More in Statistics and probability
- Simalto - SIMALTO is a market-research approach that uses a grid of product or service attributes and levels to explore customer priorities under trade-offs.
- Simple effect - A simple effect is the effect of one factor evaluated at a specified level of another factor.
- Simple interaction - A simple interaction is the interaction between two factors evaluated at one specified level of a third factor.
- Simple regression - Simple regression is regression with a single explanatory variable.
- Simpson's paradox - Simpson's paradox is a statistical phenomenon in which an association observed within separate groups reverses when the groups are combined.
- Single-arm study design - A single-arm study design follows one group receiving an intervention without an internally assigned concurrent comparison arm.
Sources
Sources: Penn State STAT 501, Simple Linear Regression, https://online.stat.psu.edu/stat501/Lesson01; Glossary of Forecasting Terms, https://robjhyndman.com/mwh3/FG4.pdf; Hyndman and Athanasopoulos, Forecasting: Principles and Practice, regression chapter, https://otexts.com/fpp3/regression.html. The four-point fit, residuals and R-squared were calculated independently for this entry. The fictional processing-time project illustrates model evaluation rather than a reported performance claim.
Previous: Simple interaction Next: Simple Paths