Is MDL the same as choosing the best training fit?

No. It also accounts for the cost of the model or model class used to obtain that fit.

Must it use an ordinary file compressor?

No. Statistical coding constructions can define description lengths without a general-purpose compression program.

Does MDL identify the true model?

Not necessarily. It selects among available descriptions under stated assumptions and can be useful even when every candidate is an approximation.

AI Glossary / Graphs and symbolic AI / Intermediate

Minimum description length

Definition

Minimum description length, or MDL, is a principle for choosing models by how compactly they describe observed data. It balances the information needed to specify a model with the information needed to describe the data using that model. A useful model captures regularity without spending excessive description length on complexity.

What it means

A model that memorizes every observation may fit perfectly but require a long description. A very simple model may be cheap to describe but leave much of the data unexplained. MDL evaluates the combined description rather than rewarding either simplicity or fit alone.
The description length is measured relative to a coding method. It is not the number of words in a model explanation or the size of a screenshot. Practical MDL involves explicit statistical or coding choices; it does not require finding the shortest possible computer program for the data, an idealized problem that is generally uncomputable.

How it works

A basic two-part illustration minimizes L(M) + L(D given M): the length of the model description plus the length of the data description conditional on that model. The descriptions must contain enough information for the intended decoding, including relevant parameters and conventions.
For probabilistic models, assigning higher probability to observed data corresponds to shorter idealized code lengths through negative log probability. More refined MDL formulations use universal coding approaches and need not appear as a simple separately transmitted model and residual file. The underlying comparison still concerns complete, comparable descriptions rather than fit measured without a complexity cost.

Three practical examples

A model-selection exercise can compare a small decision structure with a more detailed structure that captures additional patterns.
A pattern-discovery system can assess whether adding a rule saves more description length than the rule costs to specify.
A statistical analysis can use an MDL criterion to compare model structures while making its coding assumptions explicit.

Think of it this way

Think of sending someone a patterned dataset. You might send a short rule plus exceptions instead of listing everything. A complicated rule is worthwhile only if it saves enough elsewhere. The analogy is limited because formal MDL needs precise decodable representations, and statistical versions can encode data through probability models rather than ordinary prose rules.

Worked example

Consider three invented candidate descriptions of the same dataset under a fixed coding setup. All necessary overhead is assumed included, and each description allows the same data to be recovered at the agreed precision.
Model A is simple: its model description costs 20 bits, while describing the data given A costs 190 bits. Total length is 210 bits. Model B costs 60 bits to specify and leaves a conditional data description of 120 bits, totaling 180 bits. Model C costs 150 bits to specify but reduces the remaining data description to 60 bits, totaling 210 bits.
The two-part criterion selects B because 180 is the smallest total. A has the shortest model description but fails to capture enough structure. C encodes the data most compactly once its model is known, but the model's extra complexity consumes the saving.
Moving from B to C adds 90 model bits and saves only 60 data bits, worsening the total by 30. If a revised C instead reduced its conditional data description to 20 bits without changing its 150-bit model cost, its total would be 170 and it would win. The principle does not reject complexity categorically; complexity must earn its description cost.
These bit counts are assumed teaching values, not measurements from a compressor. If one description silently relies on a large external lookup table while another includes its table, the comparison is unfair unless that shared information is explicitly part of the coding setup. Likewise, changing the precision at which numeric data is reconstructed changes the task and can change code lengths.

Illustrative case study

In a fictional forecasting project, Hazel Models compares a compact trend model with a large table of special-case corrections. The correction table fits the historical observations closely, but the first comparison counts only the remaining errors and ignores the table's description.
The team revises the accounting to include the correction structure and its values. It then evaluates future-data performance separately. This invented case illustrates the need for complete description accounting; it does not claim that MDL alone establishes the correct causal explanation or guarantees a particular forecasting result.

Common misconceptions and mistakes

MDL does not mean always choose the smallest model. It chooses according to a total description criterion, so a larger model can win when it captures enough additional regularity.
Do not substitute arbitrary penalties and call them MDL without explaining their coding interpretation. Also, parameter count alone does not capture every aspect of model complexity, and finite-precision choices matter in two-part descriptions. A shorter description is not proof of causation or of a uniquely true model. Practical success also depends on the candidate model classes and the description scheme being suitable for the data.

Business use

The principle is useful when teams need a disciplined explanation for balancing flexibility against overfitting. Require the analysis to state what is encoded, which model classes are compared, and how lengths are calculated. Include assumptions about precision and shared information.
Use domain-relevant evaluation alongside the selection criterion. A model can describe observed data efficiently yet fail when the data-generating environment changes or when an important outcome has little representation in the sample. For reporting, distinguish a formal MDL calculation from a general preference for simplicity. Both can be useful, but they provide different levels of justification.

Check your understanding

Why does model C lose to B in the first worked comparison despite its shorter conditional data description?

Answer: C saves 60 data bits but requires 90 additional model bits. Its total is therefore 30 bits longer than B's total.

Related terms

Model selection
Overfitting
Kolmogorov complexity
Regularization
Occam's razor
code length
information theory
universal coding
negative log likelihood

Learn next

Create a small table of model-description and conditional-data lengths, then identify which changes can alter the winner. Next, study the connection between probability and code length before comparing basic two-part MDL with refined universal-coding formulations.

Sources

Peter Grünwald, A Tutorial Introduction to the Minimum Description Length Principle, original tutorial: https://arxiv.org/pdf/math/0406077

Previous: Minimum chi-square estimation Next: Minimum edit distance