AI Glossary / Statistics and probability / Intermediate

Regression tree

Definition

A regression tree is a tree-structured predictive model for a numerical outcome. It routes an observation through a sequence of conditions on its predictors and produces a numerical prediction at the terminal leaf. In a standard squared-error tree, each leaf predicts the mean outcome of its training observations. Splits are chosen to reduce a regression loss, and complexity controls limit how finely the data are partitioned. Other regression-tree losses can lead to different leaf predictions.

What it means

A tree represents the predictor space as regions with separate predictions rather than fitting one global slope. It can capture interactions because later conditions depend on earlier branches. Standard regression trees produce piecewise-constant predictions, which can jump at split boundaries. Their rules can be easy to inspect, but a deep tree may be difficult to understand and unstable. The fitted structure reflects a predictive criterion, not automatically a causal explanation or a natural classification of the world.

How it works

Start with training observations and choose a loss and allowed split types. Evaluate candidate splits, select a useful division and repeat within each child node. Stop according to criteria such as minimum leaf size, maximum depth or insufficient improvement, or prune a larger tree. Use appropriate validation to select complexity. For a new observation, apply the fitted conditions in sequence and return its leaf prediction. Keep preprocessing and missing-value handling consistent with the implementation used during fitting.

Three practical examples

Example 1: A service model predicts task duration by separating requests according to complexity and workload. Example 2: A property model predicts price from location and physical characteristics using conditional branches. Example 3: A demand model uses calendar and product features to create groups with different expected sales. These examples illustrate prediction structures. If a leaf contains few observations or unusual cases, its average can be unreliable despite the apparent simplicity of the rule.

Think of it this way

Think of a flowchart that ends with a number instead of a category label. Each question narrows the group of past cases used to predict the new case. The final number can be an average of those cases. The flowchart is useful only if those groups continue to predict well; splitting until every past case has its own box usually sacrifices that reliability.

Worked example

Suppose x = [1, 2, 3, 4] and outcomes are [2, 4, 10, 12]. A one-leaf squared-error tree predicts the overall mean seven and has training squared error 68. A split at x ≤ 2.5 gives a left leaf with mean three and a right leaf with mean eleven. The new squared error is (2 - 3) squared + (4 - 3) squared + (10 - 11) squared + (12 - 11) squared = 4. A new case with x = 2.4 receives prediction three, while x = 2.6 receives eleven. A case with x = 100 also receives eleven from this simple tree, because it follows the same right branch. This shows both the abrupt boundary and the limited extrapolation behavior of constant-leaf trees.

Illustrative case study

Fictional case study: Hazel Retail builds a regression tree for daily product demand. A deep version has very low training error but unstable predictions for small product groups. Analysts compare leaf-size settings on later periods and inspect boundary cases before deployment. They retain a baseline forecast for comparison. This fictional case illustrates validation and does not claim that a tree is always preferable to a linear, probabilistic or time-series model for demand.

Common misconceptions and mistakes

Common mistakes include confusing regression trees with classification trees, assuming predictions vary smoothly, and interpreting the first split as a definitive measure of causal importance. Greedy splitting can miss better global structures. Correlated predictors may replace one another across refits. Constant-leaf trees do not naturally extend a trend beyond their training range. Leakage, inappropriate validation and small leaves can create impressive-looking performance that does not survive use on new observations.

Business use

Business use: Regression trees can provide accessible conditional predictions and help reveal candidate interactions. Evaluate their accuracy and stability against the intended business loss, and inspect how many observations support each leaf. Explain abrupt prediction changes near thresholds. Avoid using one fitted tree as a policy rule without considering uncertainty and causal evidence. Where ensembles improve prediction, communicate that they combine many trees and may no longer have the simplicity of one flowchart.

Check your understanding

Why does x = 100 receive eleven in the example?

Answer: The tree routes it to the same terminal region as every x above 2.5 and has no fitted slope within that leaf.

Does the reduction from 68 to four prove future accuracy?

Answer: No; it measures training fit and requires independent evaluation for generalization.

Related terms

Recursive partitioning builds tree regions
while boosting builds an ensemble sequentially

Learn next

Calculate all candidate splits for a tiny numerical dataset and compare their squared errors. Inspect predictions immediately on either side of a boundary and outside the training range. Then increase minimum leaf size and compare held-out performance. Study how resampling changes the tree before treating its exact branching structure as stable knowledge.

Sources

Sources: Scikit-learn developers, Decision Trees, regression and practical considerations: https://scikit-learn.org/stable/modules/tree.html ; R rpart documentation: https://www.stat.ethz.ch/R-manual/R-devel/library/rpart/help/rpart.html . These primary software sources describe regression-tree fitting and complexity. The four-observation split, boundary predictions and fictional retail case are original illustrations.

Previous: Regression to the mean Next: Regression with ARIMA errors