Can mutual information be negative?

The population quantity cannot. Particular summands may be negative, and numerical estimates can behave differently depending on their estimator.

Is it limited to linear relationships?

No. It compares full distributions and can detect nonlinear dependence.

Does high mutual information prove accurate predictions?

No universal accuracy percentage follows. The remaining target uncertainty, available data, model, and chosen error measure all matter.

AI Glossary / Mathematics and optimization / Intermediate

Mutual information

Definition

Mutual information measures statistical dependence between two random variables by comparing their joint distribution with the distribution they would have if independent. For discrete variables, it is also the average reduction in uncertainty about one variable after observing the other, measured using entropy. It is symmetric and nonnegative, but it does not establish causation.

What it means

If learning X changes what we know about Y, the variables share information in this statistical sense. Mutual information can capture relationships that a linear correlation coefficient misses. It concerns the full joint distribution rather than only a straight-line trend.

Its units depend on the logarithm: base two gives bits, while the natural logarithm gives nats. A score of one bit is not one percentage point of prediction accuracy. The usefulness of shared information for a particular decision also depends on the target, its uncertainty, the prediction method, and the cost of different errors.

How it works

For discrete variables, sum p(x,y)log[p(x,y)/(p(x)p(y))] over possible pairs, with zero-probability terms contributing zero. The numerator describes observed joint probability; the denominator describes the independent combination of the marginals.

An equivalent discrete expression is I(X;Y)=H(Y)−H(Y|X), where conditional entropy averages remaining uncertainty over possible X values. Reversing X and Y gives the same mutual information. At the population level, mutual information is zero exactly when the variables are independent.

For continuous variables, suitable density integrals or more general probability definitions replace the discrete sum. Estimating mutual information from a finite dataset is a separate statistical problem. Discretization, estimator choices, limited samples, and high dimensionality can affect the estimate, so a calculated sample score should not be mistaken for an exact population quantity.

Three practical examples

1. Feature screening: examine whether a candidate input shares statistical information with a target, then test whether it improves a properly evaluated prediction system.

2. Communication analysis: quantify how much a noisy received signal reveals about the transmitted symbol under a specified channel model.

3. Representation research: study information shared between an input, a learned representation, and a target, while carefully distinguishing theoretical quantities from practical estimators.

Think of it this way

Think of receiving a clue about a hidden category. Before the clue, several possibilities are plausible; afterward, the average uncertainty may be smaller. Mutual information measures that average change using a precise information scale.

The analogy does not mean every individual clue reduces uncertainty in every realized case. The reduction is averaged over the joint distribution. It also does not make the clue a cause of the hidden category: two observations can share information because both reflect something else.

Worked example

Let X and Y be binary variables with joint probabilities p(0,0)=0.4, p(1,1)=0.4, p(0,1)=0.1, and p(1,0)=0.1. Both marginals are balanced, so every product p(x)p(y) equals 0.25.

Using base-two logarithms, I(X;Y)=0.8log2(1.6)+0.2log2(0.4). The first contribution is approximately 0.54246 and the second is approximately −0.26439, giving 0.27807 bits. Individual terms can be negative even though the complete mutual information is nonnegative.

The entropy interpretation agrees. H(Y)=1 bit. Given X, Y equals X with probability 0.8 and differs with probability 0.2, so H(Y|X)=−0.8log2(0.8)−0.2log2(0.2), approximately 0.72193 bits. Subtracting gives 0.27807.

Now consider two independent fair bits A and B, with target T=A XOR B. Knowing A alone leaves T fair, so I(A;T)=0; the same holds for B. Knowing the pair determines T exactly, so I((A,B);T)=1 bit. Screening features individually would miss their joint usefulness in this example.

Illustrative case study

Fictional case study: a subscription service uses mutual information to shortlist potential inputs for a renewal model. One variable has an unusually high score: a status field written after the renewal outcome is known.

The team removes that field from the prediction-time dataset because it would be unavailable when a decision is required. It then tests the remaining candidates on held-out data and considers interacting features rather than relying only on individual rankings. The case demonstrates that a dependence score can be mathematically valid while the feature is operationally unsuitable.

Common misconceptions and mistakes

Do not interpret mutual information as a causal effect or a signed direction. It is symmetric, so the same score describes information shared in either ordering.

Do not assume that a larger raw score across unrelated targets means a universally better predictor. Target uncertainty and the decision problem matter.

Do not equate zero linear correlation with zero mutual information. Nonlinear dependence can remain. Conversely, estimated positive mutual information in a small sample is not automatic proof of a stable relationship. Feature selection itself must be confined to training data within the evaluation procedure.

Business use

Mutual information can help analysts investigate candidate signals, redundant measurements, and surprising data relationships. Its value is strongest when combined with clear prediction timing, data-quality checks, and task-specific evaluation.

Document the estimator, logarithm base, preprocessing, sample size, and whether variables are treated as discrete or continuous. If a ranking drives expensive collection or product choices, test its stability across relevant samples and periods. Avoid presenting an information score as a direct forecast of revenue, accuracy, or decision quality.

Check your understanding

Why can two individually uninformative features still be valuable together?

Answer: Their interaction can reveal the target. In the XOR example, either bit alone leaves one bit of uncertainty, but both bits together determine the target completely.

Related terms

Entropy
Conditional entropy
Feature Selection
Information gain
Data leakage
KL divergence
statistical independence
conditional mutual information

Learn next

recompute the binary example with all four joint probabilities equal to 0.25 and verify that mutual information becomes zero. Then compare individual feature screening with a method that can detect interactions, using separate evaluation data.

Sources

Sources: Aalto Dictionary of Machine Learning, mutual information: https://aaltodictionaryofml.github.io/ADictML_Main.pdf ; John Duchi, Lecture Notes on Statistics and Information Theory, Section 2.1: https://web.stanford.edu/class/ee377/lecture-notes.pdf . Primary definitions and entropy/KL relationships were read. The explanation qualifies the supplied glossary's prediction wording: mutual information is not itself an accuracy guarantee. Calculations and case study are original.

Previous: Mutex Next: Mutual preferential independence