Why is min-entropy called minimum?

It is the limiting smallest value in the nonnegative-order Rényi family for a finite distribution.

Can the entropy exceed log2(n) for n outcomes?

Not for a normalized finite discrete distribution at nonnegative orders; the uniform distribution reaches that bound.

Does H0 count categories that were merely allowed but have zero probability?

No. It counts the positive-probability support, so model support and observed categories should not be confused.

AI Glossary / Mathematics and optimization / Intermediate

Rényi entropy

Definition

Rényi entropy is a family of measures of uncertainty or diversity in a probability distribution, controlled by an order parameter usually written α. Different orders give different emphasis to common and rare outcomes. Shannon entropy appears as the limiting case when α approaches one. The definition below concerns a finite discrete distribution and uses base-two logarithms, so values are measured in bits.

What it means

A distribution describes how probability is divided among possible outcomes. Rényi entropy turns that distribution into a number, but the order must be stated for the number to be meaningful. Smaller nonnegative orders give relatively more influence to the presence of low-probability outcomes; larger orders emphasize concentration in high-probability outcomes. This makes the family useful for examining several aspects of diversity. It does not mean that one order is universally more accurate or more appropriate than another.

How it works

For probabilities p1 through pn and α > 0 with α ≠ 1, calculate Hα = log2(Σ pi^α)/(1 − α). Zero probabilities contribute zero in this range. The limit at α = 1 is Shannon entropy, −Σ pi log2(pi). At order zero, the limiting value is the logarithm of the number of outcomes with positive probability. At order two, the expression becomes −log2(Σ pi²), often called collision entropy. As the order tends to infinity, it becomes −log2(max pi), called min-entropy. Choose the order and logarithm base before calculation, and distinguish probabilities assumed by a model from frequencies estimated from a finite sample.

Three practical examples

An analyst can compare category diversity at several orders to see whether a large number of rare categories masks domination by a few common ones.
A randomness-source evaluation can use min-entropy to focus on the most likely output, while separately accounting for estimation uncertainty and dependence between outputs.
A machine-learning experiment can use an explicitly chosen entropy order as a descriptive feature of a normalized distribution. Its usefulness must be tested against the intended task; changing the order does not automatically improve a model.

Think of it this way

Think of viewing a crowd through different lenses: one emphasizes how many groups are represented, another how evenly people are spread among them, and another the largest group. Rényi orders offer different summaries of the same probabilities. The analogy is limited because these are precise mathematical functions, and the categories themselves must be defined consistently. Renaming, merging or splitting categories can change the result.

Worked example

Take probabilities [1/2, 1/4, 1/4]. Three outcomes have positive probability, so H0 = log2(3), approximately 1.585 bits. Shannon entropy is −(1/2)(−1) − 2(1/4)(−2) = 1.5 bits. For order two, the sum of squared probabilities is 1/4 + 1/16 + 1/16 = 3/8. Therefore H2 = −log2(3/8), approximately 1.415 bits. The most likely outcome has probability 1/2, giving H∞ = 1 bit. The collision interpretation can be checked directly: two independent draws match as the first outcome with probability 1/4, as the second with probability 1/16 and as the third with probability 1/16, totaling 3/8. The corresponding effective number 2^H2 is 8/3, approximately 2.667, which need not be an integer. If the three outcomes were equally probable, every listed order would give log2(3). The differences here arise from uneven probabilities, not from changing the number of categories between calculations.

Illustrative case study

In a fictional project, Cedar Catalogue reviews the distribution of items across subject categories. A simple category count looks broad, but the analysts also calculate Shannon and order-two summaries to show whether a small set dominates. They keep the category definitions fixed and present the underlying frequencies alongside the scores. They do not equate diversity with catalog quality or assume that a higher entropy necessarily serves customers better. The example illustrates interpretation rather than a measured commercial result.

Common misconceptions and mistakes

Rényi entropy is not Rényi divergence: entropy summarizes one distribution, whereas divergence compares distributions under its own definition. An order-two value is not generally the Shannon coding rate. The ordinary Shannon chain rule cannot simply be copied to every other order. Equal entropy values do not imply identical distributions, and relabeling outcomes leaves entropy unchanged. For finite discrete distributions, increasing the order cannot increase Rényi entropy. Continuous-variable versions require different care and do not inherit every intuitive statement about counting discrete outcomes.

Business use

Use a specified order to answer a specified concentration or uncertainty question. A dashboard should show the order, units, category scheme, observation window and estimation method. For small samples, rare categories may be missed and common-category probabilities may be uncertain. A high empirical entropy does not establish unpredictability to someone with additional information, and a histogram alone can hide strong sequential dependence. When comparing periods or groups, check that sampling effort and categorization are compatible. Retain the underlying distribution so a changed score can be traced to actual probability shifts.

Check your understanding

For probabilities [1/2, 1/4, 1/4], is the probability that two independent draws match equal to H2?

Answer: No. The match probability is 3/8; H2 is its negative base-two logarithm, approximately 1.415 bits.

Related terms

Rényi divergence
Probability distribution
Shannon entropy
collision probability
min-entropy
effective diversity

Learn next

calculate H0, H1, H2 and H∞ for [0.8, 0.1, 0.1] and compare them with the worked example. Then create a sequence with a balanced marginal histogram but predictable alternation to see why distributional diversity alone does not establish sequence unpredictability.

Sources

Sources: Tobias Fritz, Characterizations of Shannon and Rényi Entropy, author-hosted lecture slides, https://tobiasfritz.science/2015/entropy.pdf ; NIST, Min-Entropy glossary, https://csrc.nist.gov/glossary/term/min_entropy . Rényi's 1961 original-paper archive was located but could not be opened. The probability calculations are original; the worklist's damaged accent remains flagged for metadata correction.

Previous: Rényi divergence Next: Reparameterization trick