Can information gain exceed the parent entropy?

Not for this standard entropy-reduction calculation, because the weighted child entropy is nonnegative.

Is zero gain always a useless feature?

Not necessarily. A feature may become useful after another split or through an interaction not visible in a single immediate partition.

Do all decision trees use information gain?

No. Other criteria include Gini impurity reduction and regression-specific objectives.

AI Glossary / Graphs and symbolic AI / Intermediate

Information gain

Definition

Information gain measures a reduction in uncertainty. In decision-tree learning, it is the entropy of the class labels at a parent node minus the weighted average entropy of the labels in the resulting child nodes.

What it means

A decision tree asks questions about input features to separate examples into more informative groups. A useful question can make the class label easier to predict. Information gain quantifies that improvement using entropy, a measure of uncertainty in the class distribution.

A group containing equal numbers of two classes has greater entropy than a group dominated by one class. A group containing only one class has zero class entropy. The gain is measured relative to the distribution at the node being split, so it depends on the data reaching that node.

This entry emphasizes the decision-tree meaning. Information-theoretic gain is related to mutual information, but it should not be confused with revenue gained, causal influence or the decision-theoretic value of collecting evidence.

How it works

Calculate the parent node's class entropy. For class probabilities p, entropy is the negative sum of p × log2(p), with zero-probability terms treated as contributing zero. Using base-two logarithms expresses entropy in bits.

Apply a candidate split and calculate the class entropy in each child. Weight each child's entropy by its fraction of the parent examples, then subtract that weighted average from the parent entropy. The weighting matters: a tiny pure child should not count as heavily as a large mixed child.

Compare candidate splits under the tree's constraints. Larger gain indicates greater reduction of class uncertainty in the observed data. It does not establish that the split will generalize well. Criteria such as gain ratio, constraints on leaf size and later evaluation address different concerns about selecting and using a split.

Three practical examples

A classification-tree trainer compares feature thresholds by how much they separate the observed labels.
A teaching tool visualizes why a split with one large mixed group can have less gain than expected from its small pure group.
A model reviewer checks whether an identifier receives a suspiciously large training gain by isolating individual records.

Think of it this way

Think of sorting a mixed collection of labeled folders into trays. A question is informative when the resulting trays make the labels less uncertain.

The analogy illustrates class separation. It does not say that every neat arrangement predicts future folders well: sorting each known folder into its own tray can look perfect while teaching little about a new folder.

Worked example

Weighting unequal child groups

Suppose a node contains eight examples: four positive and four negative. Its entropy is −0.5 log2(0.5) − 0.5 log2(0.5) = 1 bit.

A proposed split creates a left child with four positive and two negative examples, and a right child with two negative examples. The left child's positive proportion is 4/6 and its negative proportion is 2/6. Its entropy is approximately 0.9183 bits. The right child is pure, so its entropy is zero.

The weighted child entropy is (6/8) × 0.9183 + (2/8) × 0, approximately 0.6887 bits. Information gain is therefore 1 − 0.6887 = 0.3113 bits.

A simple unweighted average would give 0.4592 bits of child entropy and incorrectly inflate the gain to about 0.5408. The six-example child must carry three times the weight of the two-example child.

For comparison, splitting by a unique record identifier into eight singleton children would produce zero child entropy and gain 1 bit. That perfect training separation is not evidence that the identifier will classify unseen records usefully.

Illustrative case study

The fictional Maple Classification team trains a tree to organize service requests. One candidate feature is a unique case reference, and it creates very pure training groups.

A reviewer asks how a new case reference could support a meaningful prediction. The team removes the identifier from the candidate predictors and examines other features available at the actual prediction time. It also checks whether any feature was created only after the request's final outcome became known.

Maple compares complete models on held-out requests instead of choosing a deployment solely from the largest individual split gain. The case illustrates why a valid numerical calculation can still support an unsuitable modeling choice. It does not claim that all high-cardinality features are useless.

Common misconceptions and mistakes

Child entropies should be averaged equally. They must be weighted by the relevant example proportions, or corresponding sample weights when those are used.
High gain proves a causal relationship. It measures statistical separation in the supplied data.
A pure split guarantees good predictions on new examples. It can reflect memorization or leakage.
Information gain and gain ratio are identical. Gain ratio normalizes gain using information associated with the split's branch distribution.

Business use

Information gain helps explain why a tree selected a question at a particular node. For review, show the parent distribution, child sizes and class proportions alongside the gain; the scalar alone hides how the separation occurred.

Pay attention to predictors with many distinct values and to information unavailable when predictions will be made. The business objective may also weight errors differently from the entropy criterion. Evaluate the resulting tree against that objective rather than assuming a high-gain split necessarily improves operational outcomes. Data quality and representative evaluation remain part of the evidence.

Check your understanding

Why does the worked example weight the mixed child's entropy by 6/8?

Answer: Six of the eight parent examples enter that child. Its uncertainty therefore applies to three quarters of the observed cases after the split.

Related terms

Learn next

Recalculate the gain if each child contains two positive and two negative examples. Compare the result with the worked split, then explain why merely dividing the data does not necessarily reduce uncertainty about the class.

Sources

Google's machine-learning glossary defines decision-tree information gain and its weighted entropy calculation: https://developers.google.com/machine-learning/glossary/df#information_gain

Quinlan's original discussion of improved continuous-attribute handling also describes gain and gain-ratio split criteria: https://arxiv.org/pdf/cs/9603103

The eight-example calculation and Maple Classification case are original teaching examples.

Previous: Information field theory Next: Information gain ratio