Is stacking the same as boosting?

No. Boosting builds a sequence of models through a different training process; stacking learns to combine model outputs.

Can the final estimator be simple?

Yes. A simple combiner may be easier to evaluate and less prone to overfitting.

Does cross-validation replace the final test?

No. An independent final evaluation remains valuable, especially after extensive model selection.

AI Glossary / AI and machine learning / Intermediate

Stacked generalization

Definition

Stacked generalization, often called stacking, combines predictions from several models using another model that learns how to combine them. The combining model is trained on predictions made for examples the contributing models did not train on.

What it means

Different models can make different mistakes. One may handle broad trends well while another captures interactions or unusual patterns. Stacking tries to learn how their outputs work together.

The first layer contains the base models. Their predictions become inputs to a second layer, often called a meta-model or final estimator. That second layer can learn a more useful combination than a fixed average, although improvement is not guaranteed.

The crucial detail is how those second-layer training inputs are obtained. Predictions on the same records used to fit a base model may be unrealistically good. A meta-model trained on those flattering outputs can learn a combination that fails on new data.

For a business reader, stacking is a complexity trade-off. Any gain must justify additional training, maintenance, explanation and prediction cost.

How it works

The development data is divided into folds. A base model trains on all but one fold and predicts the held-out fold. Repeating this produces an out-of-fold prediction for each development record.

Each base model contributes a column or set of columns of these predictions. The meta-model learns from that prediction table and the known outcomes. A separate final test set is retained for assessing the overall system.

For deployment, base models are commonly refitted on the available development data. They produce predictions for a new record, and the fitted meta-model combines those outputs into the final prediction.

The fold design must reflect the task. Time-dependent data, repeated customers and related records may need chronological or grouped splits. Preprocessing must also be fitted within the appropriate training folds to avoid leaking information.

Three practical examples

Demand forecasting: a team tests whether combining models with different strengths improves forecasts on later periods.
Customer-support classification: text and structured-data models contribute predictions to a final classifier, with performance compared against simpler alternatives.
Maintenance analytics: models using different measurements are combined only after an evaluation shows that their errors are usefully different.

Think of it this way

Think of a manager learning how to combine estimates from several experienced planners. The manager studies how each planner performs on genuinely unseen assignments, then decides how to weigh their advice.

The analogy fails if the planners have already seen the answers. A combination learned from rehearsed successes can look excellent while teaching little about future performance. That is the reason for held-out predictions.

Worked example

Building an honest second-layer training table

Suppose an invented project has 1,000 development records and three base models. The team divides the records into five folds of 200.

For each fold, each base model trains on the other 800 records and predicts the 200 held out. After five rounds, every record has three predictions made without training that base model on that record.

Those three prediction columns, together with the actual outcomes, train the meta-model. The procedure requires 15 base-model fits for the out-of-fold stage, before any final refitting or tuning.

An untouched test set then compares the stack with the strongest individual model and a simple averaging baseline. The numbers illustrate the workflow and cost. They do not imply that stacking will improve any particular dataset.

Illustrative case study

A fictional retailer celebrates a large improvement from a stacked forecasting model. A review finds that the meta-model was trained on base-model predictions for their own training records.

The team rebuilds the experiment using time-respecting held-out predictions and a later test period. The apparent advantage becomes much smaller. Managers then compare that remaining improvement with additional operating cost.

The invented lesson is that an honest evaluation can change the purchasing decision. The relevant result is how the complete process behaves on future work, not how well a complicated combination explains familiar examples.

Common misconceptions and mistakes

Stacking is just averaging. A learned combiner is its defining feature; averaging uses a fixed combination rule.
More base models always help. Redundant or weak models can add cost without useful information.
Only the meta-model needs a test. Data preparation, base models, fold construction and final combination all belong inside the evaluation design.

Business use

Consider stacking after establishing strong single-model baselines and a trustworthy evaluation process. It can be worthwhile where a modest reliable improvement has sufficient operational value.

Ask how predictions were generated for training the combiner. Also require an explanation of model dependencies, prediction latency and what happens if one component becomes unavailable or changes.

Check your understanding

Why is it risky to train the combiner using base-model predictions on the records those models were fitted to?

Answer: Those predictions may be overly optimistic. The combiner can learn from a level of performance the base models will not reproduce on new records.

Related terms

Learn next

Read ensemble and cross-fitting. Draw the data path for training each base model, generating held-out predictions and evaluating the finished stack.

Sources

scikit-learn, Ensembles, stacked generalization section: https://scikit-learn.org/stable/modules/ensemble.html#stacked-generalization. Checked 12 October 2026. The fold calculation and retail scenario are original teaching examples.

scikit-learn, StackingClassifier: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html. Checked 12 October 2026.

Previous: Stack search Next: Stacked RNNs