Must the removal threshold be 0.05?

No. The criterion and threshold should be specified for the purpose; some versions do not use p-values at all.

Does it search every subset?

No. It follows a sequence of deletions.

Can excluded predictors still matter?

Yes. Exclusion is conditional on the data, competing terms, and selection rule.

AI Glossary / Statistics and probability / Intermediate

Backward elimination

Definition

Backward elimination is a model-selection procedure that starts with a specified full set of candidate predictors and removes terms one at a time according to a chosen criterion. After each removal, the model is refitted and the remaining candidates are reassessed. It stops when no permitted removal meets the rule for further simplification. The selected model is a result of that search procedure, not necessarily the best possible model.

What it means

The method asks whether a model can be simplified without losing enough fit or predictive usefulness to justify its extra terms. It begins with a fuller model and works toward a smaller one, unlike forward selection, which starts smaller and adds terms.

A common version removes the term with the largest p-value when it exceeds a specified threshold. Other versions compare information criteria, such as AIC, or use another explicit objective. These are different procedures and can produce different selected models.

The full candidate model must first be estimable. Ordinary least-squares backward elimination is therefore problematic when there are too many predictors relative to observations or exact collinearity makes the starting model unidentified.

How it works

Define the response, candidate terms, terms that must remain, and the selection criterion. Fit the full model using a consistent analysis dataset. Consider each allowed single-term deletion and identify the one preferred by the rule.

If it improves the criterion or satisfies the chosen removal threshold, delete that term and refit. Recalculate the relevant statistics because other coefficients, standard errors, and tests can change after a deletion. Continue until the stopping rule applies.

Model hierarchy may constrain deletion. For example, a retained interaction often requires retaining its component main effects for coherent interpretation. A multi-level categorical predictor may need to be assessed as a group of coefficients rather than removing individual indicator columns arbitrarily.

Pure backward elimination does not ordinarily reintroduce removed terms; bidirectional stepwise procedures can. Neither approach examines all possible subsets. Selection using the same data also affects subsequent uncertainty statements: ordinary final-model p-values and confidence intervals do not automatically account for the search. Predictive evaluation should repeat the entire selection process inside each training split to avoid leakage into validation results.

Three practical examples

A researcher simplifies an initially specified regression while retaining variables required by the study design.
A forecasting exercise compares backward AIC selection with a prespecified model and a regularized alternative.
A teaching example illustrates why removing one predictor can change the apparent contribution of the others.

Think of it this way

Think of packing a toolkit and then removing items one at a time under a clear usefulness rule. Removing one tool may make another more important, so you reassess after each change. The process can produce a practical kit, but it does not prove you examined every possible combination.

Worked example

Consider an original Gaussian linear-regression example with 100 observations, an intercept, and three centered, mutually orthogonal candidate predictors A, B, and C. Suppose the full-model residual sum of squares is 100, and the fitted contributions of A, B, and C account for 60, 1, and 0.5 additional units of sum of squares respectively.

Compare AIC values after omitting the same model-independent constant: AIC = 100 log(RSS/100) + 2k, where k counts regression coefficients including the intercept, plus the estimated error-variance parameter.

The full model has k = 5 and AIC 10. Removing C gives RSS 100.5 and k = 4, yielding approximately 8.499. Removing B instead gives RSS 101 and AIC approximately 8.995. Removing A gives RSS 160 and a much larger AIC, approximately 55.000. The first deletion is therefore C.

From the remaining A-and-B model, removing B gives RSS 101.5 and k = 3, with AIC approximately 7.489. Removing A would be substantially worse. Finally, deleting A from the one-predictor model gives intercept-only RSS 161.5 and k = 2, with AIC approximately 51.933, so the procedure stops with A. Training RSS increased during simplification; the criterion improved because the complexity penalty fell enough to compensate.

Illustrative case study

Fictional case: Oakfield Analytics begins a workload regression with several candidate measurements. The team specifies an AIC-based backward procedure and protects the variables needed to represent its planned interaction structure.

After selection, analysts inspect residuals and evaluate the whole modeling process on held-out periods. They also repeat the selection on resampled training data and notice that two related measurements frequently substitute for each other.

The fictional report therefore presents the selected model as one supported candidate rather than declaring the excluded measurements irrelevant. It records selection instability and compares operational prediction performance before deciding whether the simpler measurement process is worthwhile.

Common misconceptions and mistakes

Common mistakes include deleting all nonsignificant predictors simultaneously, failing to refit after each removal, and assuming a variable with a large conditional p-value has no substantive relevance.

Another mistake is comparing candidate models fitted on different rows because missing values change the sample. Correlated predictors can make selection unstable, and a greedy deletion path can miss useful alternatives. Data-driven selection does not justify causal conclusions or automatically preserve valid hypothesis tests. Finally, a low selection criterion is not a substitute for checking model assumptions and performance on new data.

Business use

Backward elimination can reduce model complexity and measurement burden when the starting model is manageable. Its transparency makes the sequence of decisions easy to record, but that simplicity should not hide the uncertainty introduced by selection.

State whether the goal is prediction, explanation, or a prespecified scientific test. Keep essential design variables for defensible reasons, compare relevant alternatives, and validate the complete procedure. A small final model may be easier to maintain, but unstable selection or biased interpretation can outweigh those operational benefits.

Check your understanding

Why can AIC improve even though RSS rises after a deletion?

Answer: AIC balances fitted likelihood against parameter count; a small loss of fit can be outweighed by the reduction in its complexity penalty.

Related terms

Stepwise regression
Collinearity
Cross-validation
variable selection
forward selection
AIC
model hierarchy
post-selection inference

Learn next

compare subset selection with regularization, study selection stability, and examine how validation and inference must account for data-driven modeling choices.

Sources

Sources: Howell, Statistical Methods for Psychology glossary, https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html; NIST Dataplot documentation, BEST CP, https://itl.nist.gov/div898/software/dataplot/refman1/auxillar/bestcp.htm, for selection strategies and limitations. The orthogonal-predictor AIC example is an original calculation with its parameter-count convention stated explicitly.

Previous: Backward-chaining Next: Backward shift operator