Can two criteria choose different winners?

Yes, because they value different properties.

Does optimal mean perfect?

No. It means best under the specified comparison and feasible candidates.

Is a criterion an algorithm?

No. The criterion defines the objective, while an algorithm searches for a solution.

AI Glossary / AI and machine learning / Intermediate

Optimality criterion

Definition

An optimality criterion is the rule used to decide which candidate model or solution counts as best. In statistical modeling, it commonly scores fit, predictive performance or a balance between fit and complexity, making the meaning of best explicit.

What it means

Two models can disagree because they are trying to optimize different things. One may minimize squared prediction errors, while another minimizes absolute errors. A third may accept a slightly worse fit to obtain a simpler model.

The criterion defines the comparison. The search procedure then tries to find a candidate that performs well under it. Choosing an algorithm without specifying the criterion leaves an important part of the problem undefined.

Optimality is also relative to the candidate set and constraints. A model can be best among the models considered while still being unsuitable for the real task. This statistical use of criterion should be distinguished from mathematical optimality conditions, which test whether a candidate satisfies properties required of an optimum.

How it works

Begin by defining the target, candidate models, available evidence and any constraints. Choose a scoring rule aligned with the purpose. Examples include likelihood, squared error, absolute error and model-selection criteria that penalize complexity.

Fit or compare candidates consistently, using the correct direction: some criteria are minimized, while others are maximized. Keep data splits and preprocessing comparable.

After selecting a candidate, evaluate it on information not used for selection when the goal is future prediction. Sensitivity analysis can show whether a small change in the criterion or its weights changes the winner. Such changes may reveal a genuine trade-off rather than a software error.

Three practical examples

A forecasting team chooses between models using a loss that reflects the consequences of large errors.
A statistical analyst compares candidate explanations using a stated fit-and-complexity criterion.
An AI product team sets a latency constraint and then selects the most accurate eligible model on a held-out evaluation.

Think of it this way

Think of judging a competition. A contest for fastest completion can select a different winner from one rewarding precision or a weighted combination of the two.

The analogy explains why the rules matter, but a modeling criterion is not automatically fair or complete. Important effects may be absent from the score, and optimizing a convenient proxy can move a system away from the outcome people actually value.

Worked example

Different losses can select different models

Suppose Model A has absolute prediction errors of 0, 0 and 6 on three evaluation cases. Model B has errors of 3, 3 and 3.

Under mean absolute error, A scores (0 + 0 + 6)/3 = 2, while B scores 3. A is preferred because its total absolute error is smaller.

Under mean squared error, A scores (0 + 0 + 36)/3 = 12, while B scores (9 + 9 + 9)/3 = 9. B is preferred because squared error penalizes A's large miss more heavily.

There is no contradiction. The criteria express different preferences about the distribution of errors. A business should choose based on what errors mean in its application, rather than switching metrics after seeing which one makes a favored model win. Three cases are far too few for a reliable production comparison; they only illustrate the arithmetic.

Illustrative case study

Fictional case: selecting a demand model

Ash Grove Supply, an invented distributor, compares forecasts using a familiar average-error metric. Operations staff explain that occasional large underestimates create disproportionate disruption, which the original score does not adequately reflect.

The team defines the decision objective more carefully, distinguishes forecasting from inventory policy and evaluates candidate models on a separate period. It reports trade-offs instead of calling one model best without qualification.

The case illustrates criterion design. It does not claim a real cost saving or establish a universal loss function for inventory management.

Common misconceptions and mistakes

The best training fit is not automatically the best predictor. A more flexible model can improve fit by learning noise. Criteria that penalize complexity and independent validation address related concerns but are not interchangeable in every setting.

An optimality criterion is not the same as an optimization algorithm or a stopping tolerance. The criterion defines what is valued; the algorithm searches; the stopping rule determines when computation ends.

Business use

Businesses should make criteria explicit before comparing AI systems, suppliers or model versions. This reduces disputes caused by different teams using different meanings of success.

Document the metric, direction, weighting, constraints and evaluation population. Include important subgroup outcomes and operational limits rather than allowing one aggregate score to conceal unacceptable failures. Where objectives conflict, present the trade-off and identify the decision owner. Check whether the criterion is only a proxy for the actual business outcome. Even a mathematically optimal result can be disappointing if the underlying objective was poorly chosen.

Check your understanding

A model has the lowest training error but worse performance on untouched data. Has lowest training error established the best production choice?

Answer: No. The training criterion was optimized on seen examples. Production selection needs evidence aligned with future use, including independent evaluation.

Related terms

Learn next

Learn next how a model-selection criterion differs from a final test metric. Then write a one-sentence definition of best for a real comparison, including the relevant population, error costs and operational constraints.

Model selection

Sources

Primary references: NIST parameter-estimation guidance, https://itl.nist.gov/div898/handbook/pmd/section4/pmd43.htm ; least squares, https://www.itl.nist.gov/div898/handbook/pmd/section4/pmd431.htm ; maximum likelihood, https://www.itl.nist.gov/div898/handbook/eda/section3/eda3652.htm . NIST guidance on the least-squares objective and maximum-likelihood estimation was consulted. Error calculations and case are original.

Previous: Optimal transport Next: Optimism in the face of uncertainty