Does leverage depend on the outcome?

In ordinary linear least squares, no; it depends on the design matrix.

Is high leverage always bad?

No. Designed experiments intentionally include informative predictor extremes, but their validity and uncertainty still matter.

Can a point have low residual and high influence?

Yes. A high-leverage observation can pull the fit close to itself, so its residual alone may understate its role. Additional influence diagnostics are needed.

AI Glossary / Statistics and probability / Intermediate

Leverage

Definition

Leverage is a regression diagnostic that measures how unusual an observation's predictor configuration is relative to the fitted design and how strongly its own response can affect its fitted value. In ordinary linear least squares, leverage is the corresponding diagonal element of the hat matrix, which maps observed responses to fitted responses. Leverage depends on the predictor matrix, not on the observed response values. A high-leverage point has the potential to affect the fit strongly, but actual influence also depends on its response and the surrounding data.

What it means

An observation far from the main predictor cloud can help determine a regression line's slope. This can be valuable when the observation is valid and represents an important part of the domain. It can also make the fit sensitive to errors or unusual behavior at that location. In multiple regression, leverage reflects the whole predictor combination, so a point can be unusual jointly even when no single coordinate looks extreme. The diagnostic concerns the model's design geometry and changes when predictors or transformations are changed.

How it works

For a full-column-rank ordinary linear model with design matrix X, form the hat matrix H = X(X-transpose-X) inverse X-transpose, conceptually or through a stable numerical equivalent. The diagonal h_ii is observation i's leverage. Values lie between zero and one and sum to the number p of fitted coefficients, including the intercept when present. Average leverage is p/n. Examine unusually large values alongside residuals, influence measures and data provenance. Rules such as multiples of average leverage are screening heuristics, not universal hypothesis tests or instructions to remove records.

Three practical examples

Calibration review: identify reference measurements at predictor extremes that strongly anchor a fitted line.
Predictive-model diagnostics: investigate unusual combinations of customer or equipment characteristics in a linear baseline.
Data-quality assessment: prioritize verification of records whose predictor values give them considerable potential influence. A verified extreme observation may be scientifically important and should not be discarded solely for being unusual.

Think of it this way

Think of a long lever attached to a fitted line. A point far from the main predictor cluster can move the line substantially if its response changes. The length of the lever describes potential sensitivity; whether it actually distorts the fit depends on where the point sits vertically. A long lever supporting the correct pattern is different from a long lever pulling toward an erroneous value.

Worked example

Consider a straight-line regression with an intercept and predictor values 0,1,2,10. Their mean is 13/4 and their centered squared sum is 251/4. In simple regression, leverage is 1/n + (x_i - mean x) squared divided by that centered squared sum. At x = 10, this gives 1/4 + (27/4) squared/(251/4) = 245/251, approximately 0.9761. The four leverages sum to 2 because the model fits two coefficients. If responses lie exactly on y = 2 + 3x, this point has zero residual despite its high leverage. Holding predictors fixed, increasing only its observed response by 10 changes its own fitted value by 10 times 245/251, approximately 9.7610. This original calculation shows why a high-leverage point can pull the fitted line toward itself and leave a deceptively small residual.

Illustrative case study

In a fictional case, Fir Systems fits a linear model to equipment tests concentrated in a narrow operating range, with one test at a much higher setting. The high-setting record has substantial leverage. Engineers verify its input and response, then perform sensitivity analysis with and without the record and collect additional tests near that region. They report the dependence of the slope on sparse coverage rather than deleting the observation automatically. The resulting discussion distinguishes a valid but influential design point from a measurement error.

Common misconceptions and mistakes

High leverage is not synonymous with a large residual or an influential observation. Leverage can be computed before responses are observed, whereas influence measures assess the fitted result's sensitivity to particular observations. A high-leverage point may fit closely precisely because it pulls the model toward itself. Predictor-wise range checks can miss unusual multivariate combinations. Standard ordinary-least-squares formulas do not transfer unchanged to every weighted, penalized or nonlinear model. A cutoff above one is uninformative, illustrating why screening rules require judgment in small datasets.

Business use

Businesses should use leverage as a prompt for investigation of data quality and coverage. Review whether unusual predictor combinations are genuine, supported by repeated evidence and relevant to deployment. Combine leverage with residual diagnostics, deletion or perturbation analysis and knowledge of the measurement process. Document any exclusion based on verified reasons. When an important operational region is represented by one high-leverage point, collecting more information there can be more useful than changing a diagnostic threshold.

Check your understanding

Why does the point at x = 10 have high leverage even when all responses lie exactly on a line?

Answer: Its leverage comes from its unusual predictor location, not its response error. Perfect agreement with the line gives zero residual, while the geometry still makes the fitted result sensitive to a change in that observation's response.

Related terms

Influence
Studentized residual
Outlier
Multicollinearity
Hat matrix
Cook's distance
design matrix
regression diagnostics
experimental design

Learn next

Learn how leverage enters residual variance and studentization. Then compare Cook's distance, coefficient deletion diagnostics and leave-one-out prediction errors. Extend the analysis carefully to weighted or regularized models, where the relevant smoother matrix may have different properties.

Residual variance

Sources

Penn State STAT 462, Using Leverages to Help Identify Extreme X Values: https://online.stat.psu.edu/stat462/node/171/ . Penn State, Matrix Formulation of Multiple Regression: https://online.stat.psu.edu/stat462/node/132/ . Relevant indexed course text was reviewed. The predictor-geometry example and fictional case above are original.

Previous: Levenshtein Distance Next: Lewandowski-Kurowicka-Joe distribution