AI Glossary / AI and machine learning / Advanced
Minimum Message Length
Definition
Minimum Message Length is a model-selection principle from information theory: the best model is the one that gives the shortest total description of the data, counting both the model's own complexity and the data encoded with its help.
What it means
The idea turns Occam's razor into arithmetic. A model is a compression scheme: a simple model describes itself cheaply but describes the data expensively; a complex model reverses the bill. The best total message wins.
Thinking in messages makes overfitting concrete. A memorizing model needs a huge description of its own, one entry per memorized point, so its total message is long even when its fit is perfect.
The principle is a close cousin of minimum description length and of Bayesian model comparison, where the evidence for a model rewards fit and taxes complexity in exactly the same spirit.
In practice, Minimum Message Length rarely appears as software you run; it appears as the reasoning behind penalties, regularization and the habit of preferring the simplest model that explains the data.
For a business, the principle is a decision rule: when two models perform alike, prefer the one that is cheaper to state, because complexity you cannot describe concisely is complexity you cannot maintain or defend.
How it works
Frame each candidate model as a two-part message: the description of the model itself, then the data described using that model.
Score candidates by total message length: fit gains shorten the second part, complexity lengthens the first.
Prefer the candidate with the shortest total, which balances fit against complexity without a separate penalty term.
In daily work, use its descendants: regularization strength, information criteria and validation curves operationalize the same trade.
Revisit the comparison when data changes, since the shortest message today may belong to a different model after a regime shift.
Three practical examples
- Choosing between a ten-rule scoring sheet and a thousand-parameter model whose extra fit does not repay its description cost.
- Comparing two segmentations of a customer base: twelve segments that compress purchase histories well against fifty that fit slightly better but take a manual to specify.
- Selecting polynomial degree in a trend model, where each added degree must earn its description length in compression of the residuals.
Think of it this way
Think of explaining a sales pattern to a colleague: one sentence about seasonality plus a few exceptions beats either 'it is random' or reciting every single day. The best explanation is the shortest complete one.
Worked example
two models, one message
Suppose two churn models tie on validation: a twelve-rule scorecard and an ensemble of five hundred trees. The figures are illustrative.
Describing the scorecard takes a page; describing the ensemble takes a warehouse of parameters, and Minimum Message Length says the scorecard is the better theory of the data.
The practical consequences agree: the scorecard is audited in an afternoon, explained to the regulator in a meeting, and monitored with simple drift checks.
When the ensemble's fit edge is real and large, the calculus flips: the extra fit shortens the data's description enough to pay for the model's own.
Illustrative case study
A fictional analytics team kept adding features because each one shaved a point off training error, ending with a model nobody could fully list.
On new data it performed no better than its fifty-feature ancestor, and the team's own documentation could not justify a third of the inputs.
The invented lesson: every addition must pay rent in unexplained data. If you cannot say briefly what a model is, you do not have a model; you have a pile.
Common misconceptions and mistakes
- It is just a fancy name for simplicity. It is a precise accounting: fit and complexity priced in the same currency of description length.
- It always prefers the simplest model. It prefers the shortest total message, which can be a complex model when the data genuinely has structure.
- It is a competitor to validation. They agree in spirit: validation estimates the trade empirically, and the message-length view explains why the trade exists.
Business use
When models tie on performance, choose the one you can describe shortest: it will be cheaper to run, audit and defend.
Treat unexplained complexity as technical debt with an interest rate, because it is exactly that.
Use the principle in vendor reviews: ask for the short description of the model, and weight the answer's clarity as evidence.
Apply the same accounting to your own features and rules: each must pay for its description in performance.
People also ask
What does Minimum Message Length actually minimize?
The combined length of describing the model and describing the data with the model's help: complexity plus misfit, in one currency.
Is it the same as minimum description length?
They are sibling formulations of the same principle, developed in parallel with closely related conclusions.
Do I ever compute it directly?
Rarely; its influence arrives through regularization, information criteria and validation practice.
How does it relate to Occam's razor?
It is the razor made quantitative: simplicity counts only as far as it still explains the data.
Check your understanding
How does Minimum Message Length turn Occam's razor into a decision rule?
Answer: By pricing both the model's complexity and its misfit in one currency of description length, so the best model is the shortest complete explanation, neither the simplest regardless of fit nor the best-fitting regardless of complexity.
Related terms
Learn next
AI for Business Owners helps you weigh model complexity against maintainability in procurement and build decisions: https://aiforbusinessandlife.com/academy/ai-for-business-owners. [C2]
More in AI and machine learning
- Meta-learning - Meta-learning, often called learning to learn, is training a model on many tasks so it adapts to a new task from very few examples, rather than starting...
- Meta-optimization - Meta-optimization is the use of an optimization process to improve the configuration or behavior of another optimization process.
- Mini-batch - A mini-batch is the small slice of training data, commonly 32 to a few thousand examples, used for each weight update instead of the whole dataset or a...
- Minority class - The minority class is the rare outcome in a classification dataset: the few fraud cases among millions of honest ones, the few churners among loyal...
- Missing values - Missing values are the empty cells in a dataset: measurements never taken, questions never answered, sensors that failed, customers who skipped a field...
- Mixture of subspaces - A mixture of subspaces is a modeling approach that treats high-dimensional data as points drawn from several lower-dimensional flat regions, each cluster...
Sources
[S13] Minka, Statistical Learning/Pattern Recognition Glossary; [S11] Murphy, Probabilistic Machine Learning: An Introduction. Full URLs are in the source ledger. Verified 11 October 2026.
Previous: Minimum information standard Next: Minimum norm estimation