Is total correlation ordinary correlation?

No. It is an information-theoretic measure based on the joint distribution, not a linear association coefficient.

The theoretical quantity is nonnegative. Numerical or estimation procedures can produce artifacts that require investigation.

Does zero pairwise dependence imply zero total correlation?

No. Higher-order dependence can remain, as the modulo-3 example demonstrates.

AI Glossary / Mathematics and optimization / Advanced

Total correlation

Definition

Total correlation measures statistical dependence among several random variables by comparing the sum of their individual entropies with their joint entropy. For discrete variables, it is TC(X1,…,Xn)=ΣH(Xi)−H(X1,…,Xn). It is nonnegative and equals zero exactly when all the variables are mutually independent, assuming the relevant information quantities are well defined.

What it means

If variables are independent, describing each separately requires the same total information as describing the whole collection jointly. Dependencies allow the joint description to be shorter than the sum of separate descriptions. Total correlation measures that difference in information units.

Despite its name, it is not a sum of Pearson correlation coefficients and is not restricted to linear relationships. It also differs from some other multivariate information measures that can be negative. With two variables it equals mutual information, but with three or more variables it captures dependence involving the collection as a whole.

How it works

Start from a joint probability distribution and obtain each marginal distribution. Calculate their entropies using a consistent logarithm base, then subtract joint entropy from the sum. Base-2 logarithms express the result in bits.

Equivalently, total correlation is the Kullback - Leibler divergence from the joint distribution to the product of its marginals. The product represents a hypothetical distribution with the same individual behavior but complete independence. This explains nonnegativity and the zero-at-independence property.

In real data, the joint distribution must be estimated. High-dimensional tables become sparse, and entropy-estimation errors can distort the result. Continuous-variable versions require suitable density or divergence estimators; simply binning values changes the question and introduces choices that must be assessed. Dependence is not, by itself, evidence about causal direction.

Three practical examples

1. Feature analysis: investigate whether a group of model inputs contains dependence beyond what pairwise linear correlations reveal.

2. Representation learning: assess or penalize dependence among learned latent coordinates when a particular method aims for factorized representations.

3. Probabilistic modeling: examine the information lost by an independence approximation to a joint distribution, while checking estimation uncertainty.

Think of it this way

Think of several reports that overlap in what they say. Reading them as one coordinated package can avoid some repetition compared with treating each report independently. Total correlation measures the corresponding information difference.

The analogy must include dependencies that appear only when reports are combined. There need not be one repeated sentence common to every report. A statistical dependence measure also does not determine which report caused another or whether any information is useful for a particular decision.

Worked example

Consider an original three-variable construction. X and Y are independent, each uniformly taking values 0, 1, or 2. Define Z=(X+Y) modulo 3. There are nine equally likely triples, one for each choice of X and Y.

Each variable is uniform on three values, so H(X)=H(Y)=H(Z)=log2(3)≈1.585 bits. The joint entropy is log2(9)=2log2(3)≈3.170 bits, because knowing X and Y fixes Z.

Therefore total correlation is 3log2(3)−2log2(3)=log2(3)≈1.585 bits. Yet every pair is independent. For example, each specified X,Z pair occurs once among the nine equally likely triples, with probability 1/9, equal to (1/3)(1/3).

This shows why zero pairwise mutual information does not establish mutual independence of all three variables. Under the product of marginals, all 27 triples would be equally likely; the actual distribution permits only nine. For each permitted triple, the probability ratio is (1/9)/(1/27)=3, so the divergence is also log2(3).

The calculation is exact for the stated distribution. It is not an estimated score from a dataset or evidence of any causal relationship.

Illustrative case study

Fictional case: Birch Analytics checks three categorical features before using an independence-based model. Pairwise association checks show no relationship, but the data-generation specification contains a deterministic three-way rule.

The team calculates total correlation on the known toy distribution and identifies the dependence missed by pairwise checks. For real observations, it then examines sample size, estimator behavior, and held-out model performance rather than treating the toy result as a measured production finding. The case illustrates a modeling diagnostic, not a guaranteed feature-selection improvement.

Common misconceptions and mistakes

Do not add pairwise correlations and call the result total correlation. The worked construction has independent pairs but positive total correlation.

Do not interpret the measure as information uniquely shared by every variable. Several different dependency structures can produce the same total value, so one scalar does not fully explain their organization.

The quantity is not automatically bounded between zero and one, and changing logarithm base changes its units. Also, estimated nonzero dependence may reflect finite-sample bias, preprocessing, or model misspecification rather than a robust relationship in the population.

Business use

The measure can help assess independence assumptions used in probabilistic and representation models. Its usefulness depends on whether the estimated dependence affects the downstream objective and whether the estimator is credible for the available data.

Compare against appropriate independent baselines or resampling procedures and report uncertainty. Avoid removing features solely because they are dependent: shared information may still be operationally useful. Conversely, a low estimated value is not proof of independence when data are insufficient to detect complex interactions.

Check your understanding

Why is the joint entropy of the worked triple log2(9), rather than log2(27)?

Answer: X and Y provide nine equally likely combinations, and Z is fixed by them. Only nine triples are possible, so the third variable does not add another independent three-way choice.

Related terms

Mutual information
Shannon entropy
joint entropy
Kullback–Leibler divergence
mutual independence
pairwise independence
multivariate dependence
latent representation

Learn next

Enumerate the nine triples in the worked example and verify every marginal and pairwise distribution. Compare it with three independent ternary variables and three identical copies of one ternary variable. Then study how total correlation differs from interaction information and dual total correlation before choosing a multivariate dependence measure.

Dual total correlation

Sources

Primary source: dit documentation, “Total Correlation”, https://dit.readthedocs.io/en/latest/measures/multivariate/total_correlation.html . The entropy and divergence definitions and independence property were read. The modulo-3 derivation is an independently worked teaching construction; no dataset estimation or software execution is claimed.

Previous: Torus interconnect Next: Total functions