Does each tree make a complete final prediction?

Usually each tree contributes to the ensemble's score; the final output uses the combined model.

Are the trees always shallow?

Shallow trees are common, but depth and structure are configurable and implementation-dependent.

Why use a learning rate below one?

It scales each new contribution and changes the tradeoff between incremental fitting, model size and generalization; it is not a universal guarantee against overfitting.

AI Glossary / Mathematics and optimization / Intermediate

Gradient boosted (decision) trees

Definition

Gradient boosted decision trees are an ensemble model built by adding decision trees in stages, with each new tree trained to improve the current ensemble under a chosen loss. The trees contribute to a combined prediction rather than acting as independent final answers. Gradient information from the loss guides what the next tree should learn. The abbreviation GBDT is common, and the supplied glossary also uses GBT.

What it means

A single tree divides the feature space into regions and assigns an output to each region. Gradient boosting combines many such corrections, usually controlling their contribution with a learning rate. Later trees depend on the ensemble already built, which differs from training many unrelated trees and simply averaging them. For squared-error regression, the next tree can learn residuals. For other losses, the training targets are derived from loss gradients and should not be described as ordinary numerical residuals without qualification.

How it works

Begin with an initial prediction, often a constant chosen for the loss. At each stage, calculate how the loss changes with the current prediction for each training example. Fit a tree to an appropriate negative-gradient target or use a related gradient-and-curvature formulation. Add a scaled contribution from the tree to the ensemble. Tree depth, leaf constraints, sampling, regularization and the number of stages control the model's flexibility and cost. Training normally stops according to a specified limit or validation-based rule. Classification ensembles may accumulate scores that are transformed into probabilities rather than adding probabilities directly.

Three practical examples

A tabular regression model can combine shallow trees to represent nonlinear interactions among demand-related features.
A classification experiment can use an appropriate classification loss and evaluate discrimination and probability calibration on held-out data.
A model-development team can inspect validation performance as trees are added and stop before additional complexity consistently harms the chosen evaluation criterion.

Think of it this way

Think of revising a draft in a sequence of focused edits. Each new edit addresses weaknesses in the current version, and the final result includes all accepted changes. The analogy is limited because a boosted tree follows numerical targets derived from a loss rather than understanding the content. An edit that improves the training draft can still make the result less suitable for new readers, just as overfitting remains possible.

Worked example

Consider regression data with one feature x = (1, 2, 3, 4) and targets y = (1, 1, 5, 5). Use half squared error as the training loss and initialize the ensemble to the target mean, F_0(x) = 3. Negative gradients with respect to predictions are y − F_0 = (−2, −2, 2, 2). A one-split tree with split x ≤ 2.5 predicts −2 on the left and 2 on the right. With learning rate 0.5, the updated predictions are 3 + 0.5(−2) = 2 on the left and 3 + 0.5(2) = 4 on the right. The ordinary mean squared error falls from 4 to 1. The next negative-gradient targets are (−1, −1, 1, 1). If the next tree uses the same split and predicts those leaf values, predictions become 1.5 and 4.5, reducing mean squared error to 0.25. This example uses half squared error for gradient targets and reports ordinary MSE as a metric, so the factor convention is explicit. A learning rate of one would fit this unusually simple dataset in the first tree, but that does not establish that one is the best learning rate for realistic tasks or new observations.

Illustrative case study

In a fictional forecasting study, Cedar Retail compares a boosted-tree model with a simpler baseline. The team uses time-appropriate validation, checks whether features are available at prediction time and tracks performance as trees are added. A training-loss reduction is not accepted as proof of forecasting improvement. Analysts also examine errors for smaller product groups and measure inference cost. The case describes an evaluation process without claiming that boosting outperforms the baseline or produces business savings.

Common misconceptions and mistakes

Gradient boosted trees are not random forests, although both combine trees. Boosting builds dependent corrections, while random-forest training follows a different ensemble strategy. Not every boosting loss leads to ordinary residual targets. More trees can overfit, and a small learning rate does not remove the need to tune the number and complexity of trees. Feature importance is not a causal explanation. The ensemble's output scale depends on the task and implementation, so summing classification probabilities from individual trees is generally an incorrect description of the model.

Business use

Treat boosted trees as a candidate modeling approach and compare them on a relevant validation design. Document the loss, feature handling, tree complexity, learning rate and stopping policy. Check missing-value and categorical-feature behavior for the actual implementation. Evaluate prediction latency and memory use as the ensemble grows. Keep leakage checks and subgroup evaluation visible. If probabilities drive decisions, assess calibration and decision thresholds separately from ranking performance. A flexible tabular model can still learn misleading patterns when the training data or target definition are unsuitable.

Check your understanding

Why does the first worked tree predict −2 and 2 rather than the original targets 1 and 5?

Answer: It is learning corrections to the current constant prediction 3. For the chosen half-squared-error loss, those negative-gradient corrections equal target minus current prediction.

Related terms

Learn next

reproduce the two-stage example with learning rate 0.25 and compare the prediction sequence. Then examine a classification loss and identify why its gradient targets differ from squared-error residuals. Track validation quality alongside training loss as the number of trees increases.

Sources

Sources: Google, Gradient Boosted Decision Trees, https://developers.google.com/machine-learning/decision-forests/intro-to-gbdt ; Google, Machine Learning Glossary, https://developers.google.com/machine-learning/glossary . The four-observation boosting calculation and fictional retail study are original; loss-gradient and reported-MSE scaling conventions are stated explicitly.

Previous: Gradient-based method Next: Gradient boosting