Is information projection the same as ordinary projection?

No. The divergence defines a different geometry from Euclidean distance.

Does it always mean maximizing entropy?

With a uniform positive reference, minimizing D(P||Q) is equivalent to maximizing entropy over the same feasible set; a nonuniform reference changes that relationship.

Must it be computed iteratively?

No. The grouped-total example has a direct solution, although more complicated constraints may require iterative methods.

AI Glossary / Mathematics and optimization / Intermediate

Information projection

Definition

Information projection, often called I-projection, finds the probability distribution in an allowed set that minimizes a specified Kullback-Leibler divergence from a reference distribution. Here the convention is P*=argmin over allowed P of D(P||Q), where Q is fixed. Stating that direction matters: reversing the divergence generally changes the problem and can change the answer.

What it means

Suppose an existing probability model describes several possible outcomes, but new requirements specify totals or averages that the model must satisfy. Many revised distributions may meet those requirements. Information projection chooses among them using a precise measure of departure from the reference model.
The answer is a constrained probability model, not an assertion that the reference or the new requirements are true. Calling the change minimal always means minimal under the chosen divergence and feasible set. Another measure, such as squared coordinate distance, can recommend a different adjustment.

How it works

For a finite set of outcomes, calculate D(P||Q)=sum of P(i) log(P(i)/Q(i)), with zero-probability terms handled by their limiting convention. Impose normalization, nonnegativity, and the substantive constraints, then minimize this objective. A positive P(i) paired with Q(i)=0 gives infinite divergence.
With strictly positive Q and a nonempty closed convex feasible set of finite probability vectors, a unique minimizer exists. Some constraints permit a direct solution; others need numerical optimization. For fixed totals across disjoint groups, scale the reference probabilities within each group to meet its required total while preserving their internal proportions.

Three practical examples

A planning analyst can reconcile a distribution across demand scenarios with newly stipulated aggregate scenario probabilities.
A research team can adjust a contingency-table model to prescribed margins and examine how those constraints alter individual cells.
A machine-learning experiment can use divergence-based projection as a component of a constrained optimization procedure, checking that its convention matches the algorithm's objective.

Think of it this way

Think of rearranging a set of weighted jars to meet required totals on several shelves while changing the relative allocation as little as a chosen rule permits. The rule determines what little means. The analogy stops at the geometry: KL divergence is asymmetric and is not ordinary physical distance between jars.

Worked example

Consider an original three-outcome model Q=(0.2,0.3,0.5). A new requirement says the first two outcomes together must have probability 0.8. Therefore the third must have probability 0.2. Use natural logarithms, so divergence is measured in nats.
The first group's original total is 0.2+0.3=0.5. Multiply its two probabilities by 0.8/0.5=1.6. The third outcome is multiplied by 0.2/0.5=0.4. The proposed projection is P*=(0.32,0.48,0.2). All entries are nonnegative, they sum to one, and the first group's total is exactly 0.8.
The first two probabilities retain their original ratio of 2:3. To see why this is optimal, write any feasible distribution as P=(0.8r,0.8(1-r),0.2), where 0<=r<=1. Its divergence separates into a fixed term for changing group totals, plus 0.8 times the divergence between the within-group distributions (r,1-r) and (0.4,0.6). That additional term is nonnegative and equals zero at r=0.4. This establishes the minimum without a numerical solver.
At the optimum, D(P*||Q)=0.8 ln(1.6)+0.2 ln(0.4), approximately 0.192745. A feasible alternative, P=(0.4,0.4,0.2), has divergence 0.4 ln(2)+0.4 ln(4/3)+0.2 ln(0.4), approximately 0.209074. Equalizing the first two probabilities introduces an extra change that the aggregate constraint did not require.
For comparison, minimizing squared coordinate distance to Q under the same constraint adds 0.15 to each of the first two coordinates, producing (0.35,0.45,0.2). That is a different projection under a different geometry. Neither calculation establishes which distribution best predicts future observations; predictive validity requires additional evidence.

Illustrative case study

In a fictional capacity-planning study, North Quay Analytics starts with three demand scenarios and the reference probabilities from the example. A planning committee requires the first two scenarios to receive a combined weight of 0.8 for a sensitivity exercise. The analyst records the projected weights and shows their effect on expected resource needs.
The team labels the exercise as a conditional scenario analysis. It does not present the committee's stipulated total as newly observed market data. Keeping the original model, revised constraints, and projection rule together makes the decision traceable.

Common misconceptions and mistakes

Do not treat KL divergence as symmetric or assume every author uses the same projection terminology. Write down which distribution occupies each argument. Do not ignore zero reference probabilities when new constraints require positive mass there. A solver returning a vector does not prove that inconsistent constraints are feasible.
Also distinguish uniqueness under stated finite convex assumptions from a universal guarantee over arbitrary model families. Nonconvex restrictions can introduce several local candidates. A mathematically valid projection can still preserve bias or error in the reference distribution.

Business use

Business value comes from disciplined reconciliation: the analyst can explain which totals changed, which relative proportions were retained, and what mathematical criterion selected the result. Useful review records include the reference model, constraint origins, support restrictions, objective direction, and numerical tolerance. Decision makers should also inspect sensitivity to alternative constraints instead of interpreting one optimized distribution as unavoidable.

Check your understanding

Why is (0.4,0.4,0.2) not the projection in the example even though it meets the required total?

Answer: It changes the first group's internal proportions unnecessarily and has higher D(P||Q) than (0.32,0.48,0.2).

Related terms

Kullback-Leibler divergence
Convex optimization
Maximum entropy
Probability simplex
Constraint satisfaction
Relative entropy
Iterative proportional fitting
Information geometry

Learn next

Change the required first-group total to 0.6. Derive the projected vector, verify normalization, and compare its divergence with that of an equal split within the group. Then explain why setting the reference probability of the third outcome to zero would require special attention.

Sources

Primary source: Csiszar and Shields, Information Theory and Statistics: A Tutorial, especially I-projections and iterative scaling: https://www.renyi.hu/~csiszar/Publications/Information_Theory_and_Statistics:_A_Tutorial.pdf . Numerical probabilities, comparison, and fictional planning case are original.

Previous: Information model Next: Information repository