Is out-of-bag error the average of individual tree errors?

Usually not. The standard ensemble estimate first combines eligible predictions for each row, then scores those predictions.

Can it be computed without bootstrap exclusion?

The usual procedure requires knowing which observations were omitted from each member's training sample; ordinary full-data fitting provides no such excluded rows.

Does adding trees guarantee lower OOB error?

No. The estimate may fluctuate, and more trees do not correct unsuitable features or invalid evaluation assumptions.

AI Glossary / Graphs and symbolic AI / Intermediate

Out-of-bag error

Definition

Out-of-bag error is an estimate of predictive error obtained by evaluating each training example using only ensemble members whose bootstrap training samples excluded that example. It is commonly used with bagged models and bootstrap-based random forests.

What it means

A bootstrap sample draws examples with replacement. Some original examples appear more than once, while others are absent. For a particular tree, the absent examples are its out-of-bag observations.

Across a forest, the excluded set changes from tree to tree. An observation can therefore receive a prediction from several trees that did not train on it. Comparing these aggregated predictions with known outcomes produces an internal performance estimate.

This differs from training error, where predictions can come from models fitted using the same observation. It also differs from one permanently held-out test set: out-of-bag evaluation reuses the available observations in different training and evaluation roles across the ensemble.

How it works

For each bootstrap-trained tree, record which original rows were omitted. Predict outcomes for those omitted rows and retain the predictions by row identifier. Classification combines eligible predictions using the ensemble's specified aggregation method; regression commonly averages them.

After aggregation, compute an appropriate metric against the observed outcomes. Classification error might be the fraction of incorrect predicted classes. Regression could use mean squared error. Software may report an out-of-bag score such as accuracy or R-squared instead of an error, so the metric label matters.

With n independent draws from n equally likely rows, a given row is excluded with probability (1−1/n)^n, approaching about 36.8% as n grows. This describes sampling, not the expected prediction error. Small ensembles may provide very few or no eligible predictions for particular rows.

Three practical examples

A forest developer tracks out-of-bag classification error as trees are added to see whether the estimate stabilizes.
A regression team compares candidate feature settings using out-of-bag prediction errors while reserving a separate final assessment.
An analyst audits per-row out-of-bag coverage to identify observations receiving too few eligible predictions for a useful estimate.

Think of it this way

Think of asking several reviewers to judge exercises they did not see while preparing. Each exercise is judged only by reviewers who were not trained on that particular exercise.

The analogy is limited because reviewers are not independently sampled decision trees, and avoiding the exact exercise does not prevent exposure to a near-duplicate. Similarly, excluding a database row does not prevent leakage through related rows or earlier preprocessing.

Worked example

Consider five invented rows, A through E, with binary outcomes 0, 1, 0, 1 and 1. Four illustrative bootstrap samples are:

T1: A, A, B, C, C; T2: B, B, C, D, D; T3: A, C, D, D, E; T4: A, A, C, E, E.

Their excluded rows are respectively {D,E}, {A,E}, {B}, and {B,D}. Suppose eligible class votes are D:1 and E:0 from T1; A:0 and E:0 from T2; B:1 from T3; B:1 and D:1 from T4.

Aggregation correctly predicts A, B and D, but incorrectly predicts E as 0. C appears in every sample and has no out-of-bag prediction. The error among the four covered rows is 1/4, or 25%; it would be misleading to report this as an estimate covering all five rows without noting C's omission.

The samples and votes are constructed for explanation, not fitted experimental results.

Illustrative case study

The fictional Willow Forecasts team sees an attractive out-of-bag score for a model trained on repeated equipment readings. A review finds that different rows describe the same equipment within short intervals. Omitting one row still leaves closely related information available to many trees.

Willow adds an evaluation that separates equipment groups and respects the intended prediction timing. It keeps out-of-bag diagnostics for model development but reports them alongside the more relevant deployment assessment.

This fictional case illustrates why sampling units matter. The internal estimate cannot repair a mismatch between the dataset's row structure and the business question.

Common misconceptions and mistakes

About 36.8% out of bag means 36.8% error. That percentage concerns sample exclusion, not incorrect predictions.
Every tree predicts every row for this estimate. Only trees that excluded a row contribute to that row's out-of-bag prediction.
OOB evaluation prevents all leakage. Duplicates, related observations and preprocessing fitted across all rows can still contaminate assessment.
A good OOB score guarantees future performance. Distribution changes and repeated model selection remain concerns.

Business use

Out-of-bag estimates provide useful feedback without assigning one fixed validation subset in a conventional bootstrap ensemble. They can support development decisions when observations and sampling assumptions match the task.

A business report should state the metric, bootstrap settings, observation unit and eligible-prediction coverage. For imbalanced classes, overall error may hide poor performance on the important minority class; examine suitable class-specific measures too.

Repeatedly selecting settings against the same internal estimate makes it part of model development. Use an appropriately independent evaluation for claims about a final chosen system, especially when data are grouped or ordered in time.

Check your understanding

In the worked example, why is C absent from the 25% error calculation?

Answer: Every tree's training sample contains C. No eligible out-of-bag prediction exists for it, so treating C as correctly predicted would invent evidence and distort the denominator.

Related terms

Bagging
Random forest
Validation set
Data leakage
Bootstrap sampling
generalization error
classification error
regression metric

Learn next

Build a small table showing which rows enter each bootstrap sample. Calculate excluded sets, aggregate only eligible predictions and report coverage before computing error. Then identify any repeated entities or time relationships that require a different evaluation split.

Sources

Breiman's original random-forest paper describes out-of-bag estimation: https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf

Scikit-learn demonstrates tracking out-of-bag errors as an ensemble grows: https://scikit-learn.org/stable/auto_examples/ensemble/plot_ensemble_oob.html

The five-row calculation and Willow Forecasts case are original examples. Their purpose is to clarify aggregation, missing coverage and evaluation boundaries.

Previous: Out-group homogeneity bias Next: Out-of-bag evaluation