Must the critic be a neural network?

No. Any suitable scoring function can define the bound.

Are results measured in bits?

Natural logarithms give nats; divide by log(2) to express the same information in bits.

Does an optimal critic always emerge from training?

No. Function restrictions, finite data and optimization can prevent that.

AI Glossary / Probabilistic machine learning / Advanced

Nguyen-Wainwright-Jordan lower bound

Definition

The Nguyen-Wainwright-Jordan (NWJ) lower bound is a variational expression that bounds Kullback-Leibler divergence from below using a real-valued scoring function. Applied to the joint distribution of two variables and the product of their marginals, it gives a lower bound on mutual information. Neural networks can supply the scoring function, often called a critic.

What it means

Mutual information measures statistical dependence. Computing it directly may require distributions that are unavailable in a useful explicit form. NWJ replaces that calculation with expectations of critic scores under two sampling schemes: genuinely paired observations and independently combined observations.
The population expression is a lower bound. A finite sampled estimate is a random quantity and can exceed the true mutual information. This distinction matters whenever a reported number is interpreted as a guaranteed minimum rather than an estimate of a lower-bound objective.

How it works

Let P denote the joint distribution and Q the product of its marginals. For a critic f with the required finite expectations, the expression is L(f) = E_P[f] - E_Q[exp(f - 1)]. It satisfies L(f) <= KL(P || Q), which equals mutual information in this setting.
An unrestricted optimal critic is 1 plus the log density ratio P/Q, where the ratio is defined. A restricted neural critic may not represent this function or reach it through training. The exponential term penalizes large scores under Q and can make estimation sensitive to rare, large contributions.
For a fixed critic evaluated on appropriate fresh samples, sample averages can estimate the population objective without bias. Optimizing and evaluating on the same finite data introduces additional selection concerns. Independent evaluation and repeated estimates help distinguish useful evidence from a favorable sampled value.

Three practical examples

Representation research: scientists compare how strongly learned features retain dependence with a selected target. They report the critic family and sampling procedure alongside the estimated objective.
Estimator validation: an analyst tests code on a small distribution with known mutual information before applying it to continuous data. Exact enumeration reveals sign or normalization mistakes.
Training diagnostics: a team monitors the exponential penalty separately from the joint score. Sudden spikes may explain unstable gradients that a single combined average conceals.

Think of it this way

Imagine a judge scoring genuine partnerships and randomly assembled pairs. The judge earns credit for scoring genuine partnerships highly, but pays an exponentially growing penalty for high scores on random combinations. A good judge finds evidence of association without declaring every pair convincing. The scoring rule is statistical, not a test of causation.

Worked example

Consider an original binary example. Assign joint probabilities P(0,0) = P(1,1) = 0.4 and P(0,1) = P(1,0) = 0.1. Both marginals are uniform, so Q gives probability 0.25 to each pair. Density ratios are 1.6 for matching bits and 0.4 for mismatching bits.
The exact mutual information is 0.8 log(1.6) + 0.2 log(0.4), approximately 0.192745 nats. Choose the ideal critic f = 1 + log(P/Q). Its scores are approximately 1.470004 on matches and 0.083709 on mismatches.
The joint expected critic score is 1.192745. The exponential penalty under Q is 0.5 times 1.6 plus 0.5 times 0.4 = 1. Therefore L(f) = 0.192745, achieving equality. A constant critic of 1 instead gives 1 - exp(0) = 0. It remains a valid population lower bound but misses the dependence. A constant critic of 0 gives -1/e, approximately -0.367879; a negative bound does not mean negative mutual information.
Finally, estimate the ideal critic's objective using just one independent draw from P and one from Q. If the P draw matches and the Q draw mismatches, the estimate is 1 + log(1.6) - 0.4, approximately 1.070004. This exceeds the true mutual information. That event has probability 0.8 times 0.5 = 0.4 under the stipulated independent sampling. The population guarantee survives, but it does not apply to every realized estimate. The toy critic uses known probabilities to demonstrate the mathematics; it was not fitted from data.

Illustrative case study

Fictional company Willow Research studies dependence between equipment readings and a learned representation. Its analysts first verify an NWJ implementation on the binary example. They then train critics on one partition and evaluate them on independent records, repeating the evaluation to inspect variation.
When one estimate rises sharply, the team examines both expectations and influential samples before treating the change as stronger evidence of dependence. It also checks whether the representation preserves information useful for the actual prediction task. This fictional workflow makes no claim about measured equipment performance or a deployed information estimator.

Common misconceptions and mistakes

NWJ does not guarantee that each minibatch value lies below true mutual information. A negative objective does not establish independence. A large objective does not show that one variable causes another, and retaining more information is not automatically beneficial for a business task.
Do not omit the minus one inside the exponential while keeping the same critic definition. An equivalent shifted convention exists, but its full formula must be transformed consistently. Also, pairs used for the product distribution must follow the intended independence construction; retaining genuine pairings would estimate a different quantity.

Business use

This is primarily a specialist research tool for dependence estimation and representation learning. Its practical value depends on critic capacity, sampling and estimation stability. Teams should avoid presenting a single training score as a certified information measure. For operational decisions, compare the resulting representations on relevant outcomes and preserve enough experimental detail to reproduce the estimate.

Check your understanding

Why is the toy exponential expectation exactly 1?

Answer: At the ideal critic it integrates the density ratio P/Q under Q.

Why can the one-sample estimate exceed mutual information?

Answer: Sampling variation affects both estimated expectations.

What does a zero value from the constant critic prove?

Answer: Only that this critic gives a zero lower bound, not that the variables are independent.

Related terms

Mutual information
Kullback-Leibler divergence
Variational inference
Density ratio estimation
Critic network
InfoNCE
Monte Carlo estimation

Learn next

Enumerate joint and product probabilities for a small table, then compare exact expectations with repeated sampled estimates. Study how critic restrictions and evaluation reuse affect conclusions.

Sources

Poole and colleagues, On Variational Bounds of Mutual Information, 2019, Section 2.2 and Equation 7, explains the Nguyen-Wainwright-Jordan bound and its normalization: https://proceedings.mlr.press/v97/poole19a/poole19a.pdf . The binary calculation and fictional case are original illustrations. Equation 7 and critic-normalization discussion independently consulted; exact and sampled objectives recalculated.

Previous: Neyman - Pearson lemma Next: No-good