Can discrete conditional entropy be negative?

Not under the usual Shannon definition. Continuous differential entropy is a different quantity.

Is it always symmetric?

No. What remains uncertain about X after Y can differ from what remains uncertain about Y after X.

Does zero imply a perfect practical predictor?

It implies deterministic dependence under the specified distribution, but an implemented predictor may lack the correct rule or face a different deployment distribution.

AI Glossary / Mathematics and optimization / Intermediate

Conditional entropy

Definition

Conditional entropy measures the average uncertainty remaining about one random variable after another is known. For discrete variables, H(X|Y) averages the entropy of X within each possible value of Y, weighted by that value's probability. Using base-two logarithms expresses the result in bits. It is a property of a joint probability distribution.

What it means

Suppose you want to predict an outcome X and can observe a clue Y. The clue may reveal everything, nothing or part of what you need. Conditional entropy summarizes the uncertainty left after receiving that clue.

It averages over all possible clues. It is therefore different from H(X|Y=y), the entropy after one particular observed clue. A rare clue can leave more uncertainty than the original average, even though conditioning does not increase discrete entropy on average. Information here means statistical uncertainty, not usefulness, truthfulness or business value by itself.

How it works

For each value y with positive probability, calculate the conditional probabilities p(x|y). Its entropy is −Σₓ p(x|y)log₂p(x|y), treating zero-probability terms as zero. Average these values using p(y).

Equivalently, H(X|Y)=−Σₓ,ᵧp(x,y)log₂p(x|y). The chain rule gives H(X,Y)=H(Y)+H(X|Y). Mutual information satisfies I(X;Y)=H(X)−H(X|Y), describing the average uncertainty reduction supplied by Y.

For discrete variables with finite entropies, 0≤H(X|Y)≤H(X). Zero means X is determined by Y with probability one. Independence leaves H(X|Y)=H(X). These statements should not be transplanted carelessly to differential entropy for continuous variables, whose numerical values have different properties and can be negative.

Three practical examples

1. Feature evaluation: assess how much uncertainty about a category remains after a feature is observed, while accounting for how the probabilities were estimated.

2. Decision trees: compare the weighted uncertainty in child groups after a proposed split, rather than averaging each child's entropy equally regardless of size.

3. Communication: reason about the uncertainty in a transmitted symbol after receiving a noisy observation, helping distinguish reliable information from residual ambiguity.

Think of it this way

Think of guessing a hidden card after receiving a clue. A clue naming the exact card leaves no uncertainty. A clue saying only its color may still leave several possibilities.

The analogy assumes a probability model for the cards and clues. Counting possibilities alone is insufficient when some outcomes are more likely than others. Also, a misleading or corrupted clue must be represented through its actual statistical relationship, not treated as guaranteed knowledge.

Worked example

Consider an original binary-outcome example. Half the cases have Y=A and half have Y=B. Within group A, X is always 0. Within group B, X is equally likely to be 0 or 1.

The entropy in group A is zero bits; in group B it is one bit. Therefore H(X|Y)=0.5×0+0.5×1=0.5 bits.

Without Y, X=0 has probability 0.75 and X=1 probability 0.25. Its entropy is −0.75log₂0.75−0.25log₂0.25, approximately 0.8113 bits. Knowing Y therefore supplies mutual information of about 0.8113−0.5=0.3113 bits.

The joint outcomes (0,A), (0,B) and (1,B) have probabilities 0.5, 0.25 and 0.25, giving joint entropy 1.5 bits. Since H(Y)=1, the chain rule confirms H(X|Y)=1.5−1=0.5.

Receiving the particular clue Y=B leaves one bit, which is greater than the unconditional 0.8113 bits. This does not violate the average inequality: the equally likely clue Y=A leaves zero. All calculations are hand-checked from the stated model, not estimates from a real dataset.

Illustrative case study

Fictional case: a customer-support team evaluates whether issue type helps predict the department that should receive a ticket. An analyst initially averages the entropy of a tiny issue group and a large issue group equally.

The team corrects the calculation to weight each group by its frequency. It also checks performance on held-out tickets because small training groups can appear artificially predictable. The final report describes the remaining uncertainty in bits alongside routing accuracy and the cost of misrouting, recognizing that those operational measures answer different questions.

Common misconceptions and mistakes

Do not reverse the conditioning direction casually. H(X|Y) generally differs from H(Y|X), even though mutual information is symmetric.

Do not interpret half a bit as a 50% error rate. Entropy and classification error are related but are not numerically interchangeable.

Do not confuse conditional entropy with conditional cross-entropy under an imperfect predictive model. Expected log loss can include additional error from using the wrong probabilities. Also, a feature that appears informative may leak the answer or reflect a historical artifact rather than provide usable information at prediction time.

Business use

The measure can help compare information available to a system and diagnose where uncertainty remains. It does not by itself determine which feature to collect or which model to deploy.

Use representative data, state whether logarithms produce bits or nats, and document estimation choices for rare outcomes. Compare the information gain with collection costs and actual decision performance. Statistical dependence does not establish causality: observing a clue can improve prediction without changing that clue having any effect on the outcome.

Check your understanding

Why must the two conditional entropies in the worked example be weighted by one half?

Answer: Each conditioning group occurs with probability one half. With unequal group probabilities, those actual probabilities would determine the average instead.

Related terms

Learn next

change the probability of group A in the worked example and recompute the weighted uncertainty. Then compare conditional entropy under the true distribution with a model's expected log loss when it predicts incorrect probabilities.

Log Loss

Sources

Sources: dit's primary information-measure documentation, conditional-entropy formula, examples and mutual-information identity read: https://dit.readthedocs.io/en/latest/measures/shannon.html . The explanation uses discrete Shannon entropy; it does not claim identical properties for differential or quantum conditional entropy. Worked distribution and fictional case are original, and no software output is asserted.

Previous: Conditional effect Next: Conditional expectation