Should every influential observation be deleted?

No. Investigate its accuracy, relevance, and what it reveals about the model.

Is influence the same as feature importance?

No. Here it concerns sensitivity to observations; feature importance concerns the role of predictor variables under a particular definition.

AI Glossary / Statistics and probability / Intermediate

Influence

Definition

In regression diagnostics, influence describes how much an observation affects a fitted model or a particular estimated quantity. A common assessment compares results with and without that observation. Influence is relative to the model and the quantity being examined: a case can strongly affect one coefficient or prediction while having less effect on another.

What it means

A fitted regression summarizes all the available observations, but some observations can have much greater impact than others. An influential case is one whose removal or perturbation materially changes the result of interest.
This is a statistical use of the word influence. It does not mean that the observation has a causal effect on the real-world outcome. It also does not establish that the observation is wrong or should be removed.

How it works

Start by identifying what matters: a regression slope, a fitted value, a prediction at a particular input, or the fitted model more broadly. Different diagnostics summarize different kinds of sensitivity, so no single score answers every influence question.
Distinguish influence from leverage and residual size. Leverage reflects how unusual a case's predictor values are relative to the design. A residual measures discrepancy between its observed response and fitted response. Influence depends on how the case interacts with the fitted model, often involving both.
A high-leverage case can pull a regression line toward itself and therefore have a modest ordinary residual despite substantial influence. Conversely, an unusual response near the center of the predictor distribution may affect some quantities without strongly changing the slope.
For ordinary least squares, deleting one case and refitting gives a direct sensitivity comparison. Equivalent algebraic formulas can often compute deletion diagnostics without fitting the entire model repeatedly. These shortcuts rely on the model structure and regularity conditions.
Cook's distance summarizes an observation's overall effect on fitted values, scaled by the residual variance and the number of fitted coefficients. DFBETA-style measures examine changes in individual coefficients. DFFITS focuses on a change in a fitted value. Interpret the specific definition used by the software.
Use diagnostic thresholds as prompts for investigation rather than universal rules for rejection. The practical importance of a change depends on the intended decision, the scale of the quantity, and how much independent information supports the fitted relationship.
Inspect influential records for data-entry problems, unusual measurement conditions, or membership in a different population. A genuine rare case can reveal a missing interaction, a nonlinear relationship, or inadequate coverage of part of the predictor range.
Report sensitivity transparently. If a consequential conclusion changes after one case is omitted, explain why that case was examined and show the relevant comparisons. Removing observations solely because they weaken a preferred conclusion undermines the analysis.
Single-case diagnostics can miss groups of observations that support each other or mask their joint effect. Some nonlinear-model diagnostics are approximations and can be unreliable for highly influential cases. When the conclusion matters, verify important flags through appropriate refitting and broader model checks.

Three practical examples

An analyst checks whether a sales regression's slope depends heavily on one unusually large account.
A quality team investigates a measurement whose removal substantially changes a calibration model.
An AI evaluation team tests whether a fitted performance trend is dominated by one benchmark setting at an extreme input size.

Think of it this way

Think of balancing a ruler on several supports. A support far from the others may strongly affect the ruler's position, even if it appears to fit neatly beneath it. Removing that support reveals its contribution. The analogy illustrates sensitivity of the fitted summary, not the validity of discarding evidence.

Worked example

An original example fits an ordinary least-squares line with an intercept to five pairs: (0,0), (1,1), (2,2), (3,3), and (10,20).
Using all five points gives slope 327/157, approximately 2.0828, and intercept -230/157, approximately -1.4650. Removing the final point leaves the exact line y = x, with slope 1 and intercept 0. The change directly demonstrates that the final observation influences the fitted relationship.
Yet its full-model fitted value is 3040/157, approximately 19.3631, giving a residual of only 100/157, approximately 0.6369. The fitted line has moved toward this distant point, so its ordinary residual alone understates the sensitivity issue.
For an additional check, its leverage is 147/157, approximately 0.9363. The full-model residual mean square is 1000/471. With two fitted coefficients, Cook's distance is [residual squared/(2*MSE)]*[leverage/(1-leverage)^2] = 441/20 = 22.05.
The deletion comparison is the central conclusion. The example does not establish that the final point is erroneous; it may instead expose that one straight line is unsuitable or that the predictor range needs more observations.

Illustrative case study

In a fictional case, Westbridge Planning models handling time against request size. A single very large request substantially changes the estimated slope.
The team verifies that the record is genuine and discovers that large requests follow a different processing path. Rather than silently dropping it, analysts investigate a model that represents that distinction and seek more observations in the large-request range. They report how sensitive the original result was to the case. This illustrates diagnostic reasoning, not a documented improvement in prediction accuracy.

Common misconceptions and mistakes

Do not equate high leverage with guaranteed influence. Do not assume a small residual makes a case harmless. Do not treat a large Cook's distance as a probability, a causal effect, or proof of a bad record.
Influence depends on the fitted model. A case influential under one specification may be less influential under another, and deleting a case can change what population the analysis effectively represents.

Business use

Influence checks help teams identify conclusions supported by very little data and prioritize validation of consequential records. They are useful in forecasting, calibration, and explanatory analysis. Combine them with subject knowledge and predictive assessment. A model that appears stable after inconvenient observations are removed may be less useful for the difficult cases the business actually encounters.

Check your understanding

Why is the last point influential despite its residual of about 0.637?

Answer: It has pulled the fitted line toward itself. Removing it changes the slope from about 2.083 to 1, revealing substantial sensitivity.

Related terms

Learn next

Next, compare residual, leverage, and influence plots. Then study case-deletion formulas, grouped influence, and robust methods appropriate to the types of unusual observations in the application.

Sources

Howell, Statistical Methods for Psychology glossary, Influence: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html NIST Dataplot, Regression Diagnostics: https://itl.nist.gov/div898/software/dataplot/refman1/auxillar/regrdiag.htm R documentation, Regression Deletion Diagnostics: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/influence.measures.html

Previous: Inflectional morphemes Next: Infobox