What does bootstrap mean?

Sampling from the data with replacement to create new datasets of the same size.

How is it different from boosting?

Bagging builds models independently and averages; boosting builds them in sequence to fix errors.

Is bagging only for trees?

No, but trees are the most common choice.

Rarely on its own, because averaging limits noise-fitting, but base models that are too flexible can still memorize patterns.

AI Glossary / AI and machine learning / Intermediate

Bagging

Definition

Bagging, short for bootstrap aggregating, is an ensemble method that trains many models on different random resamples of the training data and combines their predictions, usually by averaging or voting.

What it means

A single model trained on one dataset can be unstable: change a few training examples and its predictions may shift a lot. Bagging reduces that instability by using many models.

Each model is trained on its own bootstrap sample, made by drawing examples from the training set at random with replacement. Some examples appear several times and others not at all in a given sample.

The models are then combined. For numbers, their predictions are averaged. For categories, they vote and the majority wins.

Because each model sees slightly different data, their individual errors differ, and averaging cancels much of the random part. This mainly reduces variance, the tendency to overfit quirks of the sample.

Random forests are the best-known example: bagged decision trees with an additional random choice of features at each split. For a business, bagging is part of why tree-based models are strong, reliable defaults for tabular data.

Bagging also gives a built-in error estimate. Each model is trained on a sample that leaves out about a third of the data, and those left-out examples act as a test set for that model. Averaging these out-of-bag scores gives an honest estimate of performance without needing a separate validation set.

How it works

Draw many bootstrap samples, each the same size as the original data but sampled with replacement. A typical number might be a hundred or more.

Train one model on each sample, commonly a decision tree, since trees are flexible but unstable, which is where bagging helps most.

To predict, run the new example through every model and combine their outputs by averaging or voting.

Examples left out of a given sample, around a third of the data, can be used to test that model, giving an out-of-bag error estimate without needing a separate test set.

Two practical settings matter: how many models to train, and how flexible each one may be. Adding models helps until gains level off, and bagging works best with flexible, unstable base models such as deep trees rather than very simple, stable ones.

Three practical examples

Random forests for customer churn or credit risk prediction.
Averaging many models to stabilize forecasts of demand.
Reducing the effect of noisy data points in medical or sensor datasets.

Think of it this way

Think of asking a hundred people to guess the number of sweets in a jar. Individual guesses are wild, but the average is often close. Bagging asks many slightly different models and averages them.

The limit: if every person shares the same bias, such as everyone seeing the jar from the same side, averaging does not remove it.

Worked example

stabilizing a churn model

Suppose a team trains a single decision tree to predict churn on 5,000 customers. Retrained on a slightly different sample, it picks different rules and gives different scores to the same customers.

They switch to bagging: 200 trees, each trained on a bootstrap sample, with predictions averaged into a churn probability.

Scores for the same customer now barely change between retrains, and accuracy on held-out data improves slightly. The numbers are illustrative; the point is stability.

The cost is that the model is no longer one readable tree, so the team uses feature-importance summaries to explain it.

Illustrative case study

A fictional lender finds its single-tree model flips decisions for borderline applicants whenever it is retrained. Complaints rise.

It adopts a bagged ensemble. Decisions become consistent across retrains, and the team documents a clear review process for borderline cases.

The invented lesson: stability can matter as much as raw accuracy where decisions affect people.

Common misconceptions and mistakes

Bagging makes models more accurate in all cases. It mainly cuts variance; it does not fix a model that is wrong in a consistent way.
It is the same as boosting. Bagging trains models independently in parallel; boosting trains them in sequence, each correcting the last.
More models always means better. Gains level off after enough models.

Business use

Use bagged or random-forest models as strong defaults for tabular business data when you want accuracy and stability without heavy tuning.

Accept reduced readability and compensate with importance summaries and example-level checks.

Expect training to cost more compute than a single model, and scoring to be slightly slower. For most business datasets the trade-off is worth it for the stability gained.

Check your understanding

Why does averaging many bagged models usually help?

Answer: Each model makes partly different random errors, and averaging cancels much of that variation, giving a more stable prediction.

Related terms

Learn next

AI for Business Owners helps you choose reliable, explainable approaches for business prediction: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]

Sources

[S1] Russell and Norvig, AIMA glossary; [S2] Google, Machine Learning Glossary; [S12] Aalto University, Dictionary of Machine Learning. Full URLs are in the source ledger. Verified 11 October 2026.

Previous: Bag of words Next: Balance