Is out-of-bag evaluation the same as cross-validation?

They share the idea of predicting excluded examples, but their sampling and fitted-model structure differ.

Does every tree evaluate every example?

No. For an example's out-of-bag prediction, only trees that omitted that example are eligible.

Can it replace every external test?

No. It cannot by itself establish performance under new operating conditions or correct an unsuitable sampling design.

AI Glossary / Causal inference and experimentation / Intermediate

Out-of-bag evaluation

Definition

Out-of-bag evaluation estimates a bagged model's predictive performance by using, for each training example, only the component models that were trained without that example. It is commonly associated with random forests built using bootstrap samples. The examples omitted from a particular component's training sample are out of that component's bag. Combining their eligible predictions provides an internal evaluation without reserving one fixed validation subset for every component.

What it means

Bagging trains multiple models on different sampled versions of the training dataset. With bootstrap sampling, examples are drawn with replacement, so some appear repeatedly and others are omitted from a given sample.

An omitted example can serve as a check on the model trained from that sample. In a forest, the same example may be absent from some trees' training samples but present in others. Its out-of-bag prediction must use only the trees that omitted it.

This differs from ordinary training accuracy, which can include predictions from trees that learned from the example. The evaluation uses the original training collection, but enforces an exclusion rule separately for each example and tree. Its usefulness still depends on appropriate sampling and data assumptions.

How it works

Train each component on its sampled bag and retain which examples were included. For an individual example, identify the components whose bags did not include it. Aggregate predictions from only those components, using the appropriate voting or averaging rule.

Compare these out-of-bag predictions with the reference outcomes across examples to calculate a chosen performance measure. A regression forest might report squared error; a classifier might report accuracy or another suitable classification metric.

Examples may have different numbers of eligible components. With very few trees, some may lack an out-of-bag prediction, so implementation details and coverage must be checked. The final forest generally predicts new examples using all its trees, whereas each out-of-bag prediction uses a subset. Out-of-bag estimates are therefore an assessment method, not identical predictions from the final full ensemble.

Three practical examples

1. Tabular classification: a team uses a bootstrap random forest's out-of-bag accuracy as a convenient development check while inspecting important types of errors separately.

2. Demand regression: analysts evaluate out-of-bag forecast errors for independent observations, checking whether the forest improves on a simple baseline.

3. Feature inspection: developers calculate permutation importance with out-of-bag predictions to investigate which variables the fitted forest relies on under that evaluation method.

Think of it this way

Think of several tutors preparing with different practice folders. To assess one exercise fairly, ask only tutors whose folders did not contain that exercise. The analogy is limited because nearby or related exercises might still reveal its answer. Likewise, excluding a row does not prevent leakage from duplicate records, related customers, future information, or preprocessing learned from inappropriate data.

Worked example

Suppose five trees, T1 through T5, predict a numerical outcome for example X. Trees T1, T3, and T5 included X in their training bags, while T2 and T4 did not. Their predictions for X are 20, 16, 20, 18, and 20 respectively. X's actual outcome is 20.

Its out-of-bag prediction averages only T2 and T4: (16 + 18)/2 = 17. The absolute error is 3 and squared error is 9. Averaging all five predictions gives 18.8, but that is not X's out-of-bag prediction because it includes trees trained with X.

Now suppose two other examples have valid out-of-bag squared errors of 1 and 4. Across these three examples, mean squared error is (9 + 1 + 4)/3, approximately 4.67. This miniature calculation illustrates eligibility and aggregation, not a stable estimate of real-world performance. Every example's eligible tree set must be determined separately.

Illustrative case study

Fictional case: Aspen Components builds a forest to classify inspection records. Its out-of-bag score initially looks reassuring. The team then notices that several rows are repeated measurements of the same physical component.

Although a row may be excluded from a tree's bag, another measurement of that component can remain in it. The team therefore adds an evaluation that keeps all measurements of a component together across the training and assessment boundary. It retains the out-of-bag score as a clearly labeled internal diagnostic and uses the grouped assessment to examine performance on new components. No real company performance claim is intended.

Common misconceptions and mistakes

Out-of-bag does not mean an example was excluded from the entire forest. It means exclusion from the particular components used to predict that example for evaluation. Using all trees defeats that distinction.

Do not assume every random forest configuration provides meaningful out-of-bag results. The procedure requires suitable sampling, and software options differ. Similarly, out-of-bag evaluation does not automatically protect against leakage introduced before tree training.

Repeatedly choosing configurations because they perform best on the same out-of-bag results can make that score part of model selection. A separate final assessment may still be valuable. Time-dependent forecasting also needs an evaluation that respects the intended prediction timeline; random omission alone does not reproduce forecasting the future.

Business use

Out-of-bag evaluation can make early forest development more data-efficient by avoiding one fixed holdout for every tree. It provides a useful diagnostic when data are limited and the sampling design fits the problem.

Business reports should name the metric, describe the sampling approach, and state whether the score influenced model choices. Explain any related-record or timing concerns. Treat the result as one source of evidence, especially when deployment differs from the population represented in the training collection.

Check your understanding

A tree's bootstrap sample includes an example three times. Can that tree contribute to the example's out-of-bag prediction?

Answer: No. Any inclusion makes that tree ineligible for that example. Repeated inclusion does not create a separate out-of-bag copy.

Related terms

Bagging
Random forest
Ensemble
Validation set
Data leakage
Generalization
bootstrap sampling
permutation importance

Learn next

Create a table with six examples and four imagined bootstrap bags. Mark which trees are eligible for each example, then calculate one out-of-bag average. Check whether any example has no eligible tree and explain how that limits evaluation.

Sources

Google, Decision Forests course, Out-of-bag evaluation: https://developers.google.com/machine-learning/decision-forests/out-of-bag

The primary course explains per-example tree exclusion and out-of-bag permutation importance. The numerical example, grouped-record case, and exercise are original teaching material; they do not reproduce a software experiment.

Previous: Out-of-bag error Next: Out of control