AI Glossary / AI and machine learning / Intermediate
Bayesian information criterion
Definition
The Bayesian information criterion, or BIC, is a model-selection score that balances fit to observed data against model complexity. In a common form, BIC = -2 log(L_hat) + k log(n), where L_hat is the maximized likelihood, k the parameter count and n the sample size. Lower values are preferred within a valid comparison.
What it means
Adding parameters can improve a model's fit to the data used to estimate it. That improvement may reflect useful structure, but it may also capture noise. BIC applies a penalty so that extra complexity must earn its place through a sufficient improvement in fit.
The score is comparative. Its absolute value is not a percentage accuracy or a direct measure of business value. Models need to be fitted to comparable data using compatible likelihood definitions for their scores to be meaningfully compared.
The name reflects its derivation as an approximation related to Bayesian model evidence under particular assumptions. It is not a universal test of truth and should not be applied mechanically to every model architecture or dataset.
How it works
Fit each candidate model and obtain the maximized log-likelihood. Count the parameters as appropriate for that model and the criterion being used. Apply the complexity penalty k log(n) and add it to the negative twice log-likelihood term.
The fit term rewards models that assign greater likelihood to the observations. The penalty grows with the number of parameters and, for fixed k, with the logarithm of sample size. Compare the resulting scores, with smaller BIC preferred.
The approximation depends on statistical conditions. Dependent observations, nonstandard models and ambiguous effective parameter counts can require specialized treatment. Inspect residuals and predictive performance as well. A criterion can select the best of a weak candidate set without making the selected model adequate.
Three practical examples
- An analyst compares several suitable time-series model specifications fitted to the same observations.
- A mixture-model experiment uses BIC as one input when choosing among candidate numbers of components.
- A modeling review checks whether an extra predictor improves likelihood enough to offset its parameter penalty.
Think of it this way
Think of evaluating a proposed explanation by asking both how well it fits the evidence and how much machinery it introduces. Extra detail should contribute something useful.
The analogy is limited because BIC uses a particular mathematical penalty. An intuitive preference for simplicity is not equivalent to calculating the criterion, and operational simplicity can involve factors the parameter count does not capture.
Worked example
Comparing two fitted models
Suppose two fictional models use the same 100 observations. Model A has maximized log-likelihood -120 and three parameters. Model B has log-likelihood -117 and five parameters. Using natural logarithms, log(100) is approximately 4.605.
A's BIC is 240 + 3 × 4.605 = 253.815. B's is 234 + 5 × 4.605 = 257.025. B fits better, but A has the lower BIC because the improvement does not offset the extra penalty.
The difference is approximately 3.21. This comparison does not establish that A predicts every future case better or that either model is suitable for deployment. It illustrates the criterion's trade-off under the stated formula and invented inputs.
Illustrative case study
A fictional distribution company compares demand models. One analyst reports that a more complex version has the smallest training error and should therefore be selected. A review adds a valid likelihood-based comparison, BIC and a held-out forecast check.
The team discovers that the extra parameters improve fit modestly but do not consistently improve forecasts. It chooses a simpler candidate for further testing and documents the comparison conditions.
The case illustrates using BIC alongside the intended purpose. It does not suggest that the simpler model must always win or that BIC replaces evaluation on future-like data.
Common misconceptions and mistakes
- The model with the best training fit always has the lowest BIC. The complexity penalty can reverse the ranking.
- A BIC value is a probability. It is a comparative score.
- Any two BIC numbers can be compared. Differences in observations or likelihood definitions can make the comparison invalid.
Business use
BIC is useful in statistical model selection when its assumptions and parameter accounting fit the problem. It provides a compact way to discuss whether extra complexity is supported by the data.
Ask the analyst which candidates were compared, whether they used the same observations and how parameters were counted. Pair the result with checks relevant to the decision, including forecast quality or error consequences. A statistically preferred model may still be difficult to maintain or unsuitable for the operational setting.
People also ask
Should BIC be high or low?
Lower is preferred among models in a valid comparison.
Does BIC always choose the smallest model?
No. A sufficiently large improvement in likelihood can justify extra parameters.
How does it differ from AIC?
Both balance fit and complexity, but use different penalties and have different theoretical motivations.
Check your understanding
Model B fits better than A but has a larger BIC on the same data. Which does BIC prefer?
Answer: A. Under the criterion, B's fit improvement does not compensate for its added complexity.
Related terms
- Model selection
- Likelihood
- Akaike Information Criterion
- Overfitting
- Parameter
- Cross-validation
- marginal likelihood
Learn next
Read likelihood and AIC. Reproduce a two-model BIC comparison, then examine whether the preferred model also meets the task's predictive and operational requirements.
More in AI and machine learning
- Batch - In machine learning, a batch is a group of training examples processed together in one step before the model updates its parameters.
- Batch size - Batch size is the number of training examples a model works through before it makes one adjustment to itself, a setting that trades speed, memory use and...
- Bayes factor - A Bayes factor compares how well two specified models predict observed data.
- Bayesian interpretation of kernel regularization - The Bayesian interpretation of kernel regularization explains how certain penalized kernel-learning methods correspond to probabilistic models with priors.
- Bayesian linear regression - Bayesian linear regression is a method for predicting a numerical outcome with a linear model while expressing uncertainty about its coefficients and other...
- Bayesian multivariate linear regression - Bayesian multivariate linear regression jointly models several numerical outcomes using linear relationships with predictors.
Sources
Scikit-learn, Linear Models, information criteria discussion: https://scikit-learn.org/stable/modules/linear_model.html. Sumio Watanabe, A Widely Applicable Bayesian Information Criterion: https://arxiv.org/abs/1208.6338. Official documentation and primary research illustrate the standard criterion and limits of regular-model approximations. The numerical comparison is original.
Previous: Bayesian inference Next: Bayesian interpretation of kernel regularization