How many entries are needed for three variables?

Seven when the empty subset is omitted, or eight when its fixed zero entry is included.

Does the vector uniquely determine the distribution?

No. The two explicit constructions have different distributions but the same vector.

Is the all-zero vector entropic?

Yes. Constant variables provide a construction: every subset has zero uncertainty.

AI Glossary / Mathematics and optimization / Advanced

Entropic vector

Definition

An entropic vector records the joint entropies of all nonempty subsets of a collection of discrete random variables with finite entropies. A numerical vector is entropic when some joint distribution actually produces all of its entries. For n variables, the usual representation has 2 to the power n minus 1 coordinates, each labeled by a subset.

What it means

The vector summarizes how uncertainty changes when variables are considered individually and together. Its coordinates must arise from the same joint distribution; they cannot be freely selected as unrelated scores. It preserves many information-theoretic relationships, but it generally does not identify a unique probability distribution or explain what the variables mean in the real world.

How it works

Specify the variables, their joint distribution, the subset ordering, and the logarithm base. For each nonempty subset, form its marginal distribution and compute its entropy. An alternative convention includes the empty subset with entropy zero, giving 2 to the power n coordinates. Entropic vectors satisfy monotonicity and submodularity inequalities. Those necessary constraints do not generally provide a complete characterization. An almost entropic vector belongs to the closure of the entropic region; this is a limiting notion distinct from direct realization by a distribution.

Three practical examples

A fictional information-analysis exercise compares a duplicated status field with an independently generated flag. The entropy vector makes the duplicate relationship visible through the joint entropy, even though all three individual fields are equally unpredictable.
A fictional network-design team expresses information requirements using entropies of several messages and their combinations. A proposed list of entropy values must be mutually consistent before it can represent an actual message distribution.
A fictional data-quality review checks whether reported single-field and combined-field uncertainties are plausible together. If a discrete field has entropy 1 bit but a pair containing it is reported to have entropy 0.5 bits, the report violates a necessary monotonicity condition.

Think of it this way

Think of a catalog that lists the uncertainty of every possible bundle of variables. A bundle containing a duplicate may add no uncertainty, while a bundle containing an independent field can add more. The catalog is more informative than a list of individual scores, but it is still a summary rather than a copy of every probability in the underlying dataset.

Worked example

Let X be a fair bit, let Y=X, and let Z be another fair bit independent of X. The four triples 000,001,110,111 each occur with probability one quarter.
Use the coordinate order H(X), H(Y), H(Z), H(X,Y), H(X,Z), H(Y,Z), H(X,Y,Z), with logarithms to base two. The three individual entropies are each 1 bit.
The pair (X,Y) has only two equally likely outcomes, 00 and 11, so H(X,Y)=1. Each of (X,Z) and (Y,Z) has four equally likely outcomes, giving entropy 2. The full triple also has four equally likely outcomes, so its entropy is 2.
The entropic vector is therefore (1,1,1,1,2,2,2). Its realizability is established by the explicit distribution above, not merely by checking a few inequalities.
Several quantities follow directly. H(Y|X)=H(X,Y)-H(X)=0. I(X;Y)=H(X)+H(Y)-H(X,Y)=1. I(X;Z)=1+1-2=0. Also H(Z|X,Y)=2-1=1, so knowing the duplicated pair does not reduce uncertainty about Z.
Now change Y to 1-X while leaving X and Z independent fair bits. The possible triples become 010,011,100,101, each with probability one quarter. Every entropy coordinate remains the same. The vector therefore cannot distinguish equality from this deterministic bit reversal, even though the records themselves differ.

Illustrative case study

In a fictional analytics workshop, a team receives the seven-number vector from the example without its coordinate labels. One analyst assumes the fourth entry describes H(X,Z), while another assumes it describes H(X,Y). Their conclusions about which fields are redundant disagree.
The team fixes the coordinate order and reconstructs the seven entries from the four equally likely records. It then verifies the conditional and mutual-information calculations rather than treating the bare list as self-explanatory.
A later review compares the Y=X and Y=1-X constructions. Both have the same entropic vector, but one requires copying a bit and the other requires reversing it. The team records this limitation: entropy summaries can describe deterministic dependence without supplying the particular transformation. The workshop is an explanatory scenario, not evidence that an entropy vector alone is sufficient for production data repair.

Common misconceptions and mistakes

An entropic vector is not a probability vector. Its entries need not sum to one; the example's seven entries sum to 10 bits, and that sum is not the entropy of the triple.
Individual entropies do not determine the joint entropy. Three fair bits may be copied, independent, or related in other ways.
Passing a selection of entropy inequalities does not automatically prove realizability. An explicit construction proves the worked example, but more general characterization problems require stronger reasoning.
Coordinate order matters. A correct collection of numbers attached to the wrong subsets produces incorrect conclusions.
Entropic and almost entropic describe different mathematical conditions. One concerns actual realization; the other permits limits of realizable vectors.

Business use

Entropy vectors can organize information constraints and reveal inconsistencies in proposed uncertainty summaries. Keep the subset labels, units, and source distribution or estimation method alongside every vector. For the worked example, the vector identifies that Y adds no uncertainty given X, but additional knowledge is required to know whether Y copies or reverses X. Where probabilities come from samples, investigate estimation uncertainty and data coverage before interpreting small differences between entropy coordinates as dependable structural evidence.

Check your understanding

Why is H(X,Y)=1 in the example?

Answer: Y is completely determined by the one fair bit X.

Why is H(X,Y,Z)=2 rather than 3?

Answer: Only X and Z contribute independent fair bits.

What additional information distinguishes the two constructions with the same vector?

Answer: The actual relation Y=X versus Y=1-X, or the corresponding joint probabilities.

Related terms

Marginal distribution
Conditional entropy
Mutual information
Submodularity
Entropy function
joint entropy
entropy region
almost entropic vector
information inequality

Learn next

Construct the vector for three mutually independent fair bits in the same coordinate order. Then construct one for a fair-bit parity relation. Compare individual, pairwise, and triple entries rather than comparing only their sums. State the joint distribution for each construction so realizability remains explicit.

Sources

Qi Chen and Raymond W. Yeung, Partition-Symmetrical Entropy Functions, introduction and entropy-region definitions: https://arxiv.org/pdf/1407.7405 The binary constructions, coordinate calculations, and analytics-workshop scenario are explanatory illustrations, independently checked from the stated distributions.

Previous: Entity-Relationship Model Next: Entropically regularized optimal transport