Can DTC be positive when every pair has zero mutual information?

Yes. The parity example has DTC 2 bits and zero pairwise mutual informations.

Can DTC be negative for a finite discrete distribution?

The population quantity cannot; a negative computed estimate calls for checking the estimator and calculation.

Does zero DTC mean only pairwise independence?

No. For the finite-valued setting described here, it characterizes mutual independence.

AI Glossary / Mathematics and optimization / Advanced

Dual total correlation

Definition

Dual total correlation (DTC) is an information-theoretic measure of dependence among several random variables. For finite-valued variables, it equals their joint entropy minus the sum of each variable's conditional entropy given all the others. It reduces to ordinary mutual information when there are two variables.

What it means

DTC describes a property of the whole joint distribution. It can reveal dependence that is invisible in pairwise comparisons. Its value is expressed in bits when entropy uses base-two logarithms. It is not a percentage, a signed linear correlation coefficient, or a direct measure of whether one variable causes another.

How it works

For variables X1 through Xn, calculate DTC = H(X1,...,Xn) - sum_i H(Xi|X without Xi). Each conditional term measures the uncertainty remaining in one variable after all the others are known. DTC is nonnegative and is zero exactly for mutual independence. Total correlation uses a different formula: sum_i H(Xi) minus the joint entropy. Neither formula is obtained by simply adding pairwise correlations. This entry uses finite discrete distributions, avoiding complications associated with continuous variables and entropy definitions.

Three practical examples

A fictional data engineer checks three binary fields linked by a parity rule. Each pair can appear independent even though knowing two fields determines the third. Examining only pairwise associations would miss this relationship.
A fictional reporting system duplicates one binary status into three columns. DTC detects dependence, but its numerical value differs from the parity example. The measure should therefore be interpreted alongside the actual construction of the fields.
A fictional analyst compares three independently generated control flags with production flags. The independent construction provides a known reference with DTC zero. A finite sample estimate may deviate from that reference, so estimation error must be separated from the theoretical value.

Think of it this way

Think of a three-part combination rule. Any one part may tell you nothing about another part, yet two parts together can fix the third. Pairwise checks ask about pairs; DTC uses the whole arrangement. The analogy highlights a dependency pattern, not a claim that a single number completely explains every relationship in a dataset.

Worked example

Let A and B be independent fair bits, and set C=A XOR B. The four possible triples are 000,011,101,110, each with probability one quarter. Their joint entropy is 2 bits because four equally likely outcomes require two bits to distinguish.
Each individual variable is fair, with entropy 1 bit. Any two variables determine the third: A and B determine C, A and C determine B, and B and C determine A. Therefore H(A|B,C), H(B|A,C), and H(C|A,B) are all zero. DTC is 2-0-0-0=2 bits.
Every pair is nevertheless independent. For example, the four possible pairs (A,C) each occur with probability one quarter. Thus I(A;C)=1+1-2=0, and the other pairwise mutual informations are also zero. Total correlation is 1+1+1-2=1 bit, different from DTC.
Now compare three copies of one fair bit: A=B=C. Only 000 and 111 occur, each with probability one half. Joint entropy is 1 bit, and each leave-one-out conditional entropy remains zero. DTC is therefore 1 bit, while total correlation is 3-1=2 bits.
Finally, three mutually independent fair bits have joint entropy 3, with each conditional entropy 1. Their DTC is 3-1-1-1=0. These three constructions give distinct answers without any need to estimate probabilities from a finite dataset.

Illustrative case study

In a fictional data-quality workshop, a team considers the parity construction as three fields in a processing record. A pairwise association report shows zero mutual information for every pair. The first interpretation is that none of the fields is related to the others.
The reviewer enumerates the four allowed records and calculates DTC=2 bits. The team then checks reconstruction: any two fields recover the missing field exactly under the stated rule. The missing relationship is a group dependency, not a small pairwise effect overlooked by rounding.
A second dataset contains copied status fields and gives DTC=1 bit. The team resists calling it half as useful or half as dependent in every practical sense. Both constructions allow exact recovery of one missing field, but their joint uncertainties and dependence structures differ. The workshop documents the rules and entropy calculations alongside the scores. This example demonstrates interpretation, not a measured improvement in a deployed AI model.

Common misconceptions and mistakes

Pairwise independence does not imply mutual independence. The parity example proves this directly.
DTC and total correlation are not interchangeable: their values reverse between the parity and copied-bit constructions.
A value of 2 bits is not a correlation coefficient exceeding an allowed range. DTC uses information units rather than the scale of a Pearson coefficient.
The score alone does not identify a causal direction or determine which feature to remove. Predictive usefulness depends on the task and the other available information.
Theoretical probabilities and empirical frequency estimates are different inputs. Estimating a large joint distribution from limited observations can create misleading numerical results even when the formula is implemented correctly.

Business use

Use DTC as one diagnostic when relationships across several fields matter. Start with explicit variable definitions and a defensible joint-distribution estimate. In the parity example, a data-quality test could check the actual parity constraint as well as calculating a summary score. In the copied-field example, lineage information explains why the columns agree. Report the underlying assumptions and relevant dependence patterns so decision makers do not treat a larger score as automatically better data or a more valuable model.

Check your understanding

Why is joint entropy 2 bits in the parity example?

Answer: There are four equally likely triples.

Why are all three conditional entropies zero there?

Answer: Knowing any two bits determines the third.

What are DTC and total correlation for three copies of one fair bit?

Answer: DTC is 1 bit and total correlation is 2 bits.

Related terms

Total correlation
Mutual information
Conditional entropy
Entropy estimation
joint entropy
mutual independence
pairwise independence
higher-order dependence
information theory

Learn next

Replace the deterministic parity bit with a parity bit that is independently flipped with a specified probability. Enumerate the resulting joint probabilities and recalculate the entropy terms. Observe how introducing noise changes the group dependence while preserving the distinction between pairwise and joint analysis.

Sources

Tim Austin, Multi-variate Correlation and Mixtures of Product Measures, Kybernetika 56(3), 2020, definitions and basic properties: https://www.kybernetika.cz/content/2020/3/459/paper.pdf The enumerated binary constructions, calculations, and data-quality scenarios are explanatory illustrations, independently checked from their stated probabilities.

Previous: Dual problem Next: DualDICE