Does stepwise regression test every subset?

Usually not; it follows a sequence of permitted local changes.

Can a removed variable be reintroduced?

That depends on the direction and implementation.

Are final p-values automatically valid?

No. Standard outputs generally ignore the selection process.

AI Glossary / Statistics and probability / Intermediate

Stepwise regression

Definition

Stepwise regression is a family of procedures that builds a regression model by adding or removing candidate predictors according to a specified rule. Forward selection begins with a smaller model, backward elimination begins with a larger model, and bidirectional procedures can make both kinds of move. The selected model depends on the candidate set, starting point, criterion, and data. Automatic selection does not establish that the retained variables are causes or that the model is best for future use.

What it means

The procedure turns model choice into a sequence of local decisions. At each stage it asks whether an eligible addition or deletion improves a criterion, such as an information score, or satisfies a testing rule. Different rules need not produce the same model.

The word stepwise is sometimes used narrowly for a procedure that rechecks earlier inclusions after adding a predictor, and sometimes broadly for sequential selection methods. State the actual direction and criterion. The important distinction is that the data participate in choosing the model, so ordinary summaries that pretend the final model was fixed in advance can overstate evidence or precision.

How it works

Specify the response, candidate terms, mandatory variables, allowed interactions, and starting model. Establish a comparison criterion and stopping rule. Use comparable data across the candidate fits; differences caused by changing missing-data subsets can undermine the model comparison.

Fit the eligible neighboring models at each stage. Select an allowed improvement and repeat until no qualifying move remains or another stopping rule is reached. Preserve the selection path so reviewers can see what was considered and why a term was retained or removed.

Assess stability by examining how selection changes under appropriate resampling or plausible analytical choices. Evaluate predictive performance with a validation process that includes the selection procedure inside each training stage. Selecting variables on the entire dataset before cross-validation leaks information into the evaluation.

For interpretation, distinguish exploratory model selection from confirmatory inference. Ordinary coefficient p-values and confidence intervals generally do not account for the search. Consider substantive knowledge, prespecified models, shrinkage, and suitable post-selection methods rather than assuming automation resolves model uncertainty.

Three practical examples

1. An analyst explores a small set of demand predictors, records the selection path, and checks performance on a separate future period.

2. A research team examines whether selected predictors change substantially across resamples, revealing instability hidden by one final regression table.

3. An AI team compares a stepwise linear baseline with a regularized model using the same carefully separated evaluation data.

Think of it this way

Think of choosing a route by repeatedly taking the locally most promising turn. The rule can be systematic and reproducible, yet still miss a better route requiring an initially unattractive move. Repeating the journey on a slightly different map may also produce a different destination.

Worked example

Consider an original teaching example with 20 common observations and two optional predictors, A and B. Suppose fitted Gaussian linear models have residual sums of squares of 200 for an intercept-only model, 100 with A, 150 with B, and 95 with both.

Use the AIC-equivalent comparison score S = 20 ln(RSS / 20) + 2p, where p counts regression coefficients including the intercept. Constants common to all models, including the common variance-parameter penalty, are omitted. Lower scores are preferred.

The intercept-only score is about 48.052. Adding A gives 36.189, while adding B gives 44.298. Forward selection therefore chooses A. The model with both predictors has score 37.163, which is higher than 36.189, so the search stops with A alone.

Adding B reduced RSS from 100 to 95, but the improvement did not outweigh the score's complexity penalty. This illustrates a specified decision rule, not evidence that B has no relationship with the response or no business relevance. The RSS values are stipulated for the example; predictive validation and selection-aware inference remain separate tasks. With larger candidate sets, a local search need not find the globally best score.

Illustrative case study

In a fictional company, Larch Logistics uses stepwise regression to explore delivery-delay predictors. The first selected model includes one of two strongly related workload measures. When analysts repeat the process on resampled training records, the two measures frequently replace each other. They stop describing the first measure as the uniquely important driver and instead report the instability. The business compares predictive performance with a regularized alternative and retains operationally necessary variables in its decision process, even where an automated rule would omit them.

Common misconceptions and mistakes

Selection is not causal discovery. A retained predictor may be a proxy, and an omitted variable can still matter scientifically or operationally.

A smaller final model does not mean the analysis used only that many choices. The search across candidates affects optimism and uncertainty.

Using AIC rather than p-values does not eliminate every selection problem. Nor does a high training R-squared demonstrate reliable prediction. Correlated predictors, weak signals, and limited data can produce unstable selections. Reporting only the winning model conceals important information about how the result was obtained.

Business use

Stepwise regression can provide an exploratory baseline or a transparent record of a constrained model search. Its convenience should be weighed against instability, inferential limitations, and the cost of collecting selected predictors.

Business teams should document the full candidate set and selection rule, compare credible alternatives, and evaluate the whole modeling process. If coefficients will guide interventions, a predictive selection routine alone is inadequate: the causal question and study design require separate treatment. A model that forecasts reasonably well may still give misleading advice about what to change.

Check your understanding

Answer: A candidate predictor lowers training error but raises the chosen AIC score.

Should an AIC-based stepwise procedure add it?

Answer: No, under that rule it is not an improvement. This does not prove the predictor is useless for every purpose.

Related terms

Backward elimination
Overfitting
Multicollinearity
Regularization
Variable selection
forward selection
AIC
post-selection inference

Learn next

Study validation of complete modeling pipelines and shrinkage methods. Then examine why inference after variable selection differs from inference for a prespecified model.

Sources

Howell's author glossary, stepwise procedures: https://www.uvm.edu/~statdhtx/methods8/Glossary/Glossary.html . R documentation, step: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/step.html . StataCorp's author-commentary FAQ on selection problems: https://www.stata.com/support/faqs/statistics/stepwise-regression-problems/ . The numerical comparison and fictional case are original; the entry distinguishes broad terminology from specific implementation rules.

Previous: Stepped-wedge trial Next: Stereo cameras