Does ERM always overfit?

No. Its generalization depends on the hypothesis class, sample, assumptions, and learning setup. Overfitting is a risk, not an inevitable outcome.

Can different algorithms implement ERM?

Yes. Closed-form solutions, direct search, and iterative optimization can pursue suitable empirical-risk objectives.

Does minimizing loss make a model fair?

No. The loss and data may omit fairness concerns, and acceptable deployment requires broader assessment of the system and its consequences.

AI Glossary / Governance, ethics and safety / Intermediate

Empirical risk minimization

Definition

Empirical risk minimization, or ERM, is the learning principle of choosing a model from an allowed set to minimize average loss on the training examples. The data supplies examples, the loss defines what counts as a mistake, and the hypothesis class defines the available models. ERM describes an objective, not a guarantee that the selected model will work well on new data.

What it means

The principle turns learning into a comparison: which permitted prediction rule performs best under the chosen scoring rule on the observed training set? A simple application may compare a few candidate rules directly. A large neural network may use iterative optimization to seek parameters with low average loss. The choices made before optimization matter greatly. If the loss ignores an important consequence or the data excludes an important population, minimizing that objective will not automatically repair the omission. The best training score is best only within the stated problem.

How it works

Specify a hypothesis class, a training dataset, and a loss function. For each candidate, its empirical risk is the average of the losses it incurs on the training examples. ERM selects a minimizer, or practical training seeks an approximate solution when exact optimization is difficult. Different algorithms can pursue the same objective, and some objectives have multiple minimizers. Regularized ERM adds a penalty or constraint to influence the solution and help control fitting behavior. Evaluation on suitably held-out data checks whether the learned rule generalizes beyond the examples used to choose it.

Three practical examples

A small classifier chooses among several thresholds by counting training mistakes and selecting the threshold with the lowest rate.
Linear regression fits coefficients by minimizing average squared prediction error on observed outcomes.
A neural classifier is trained to reduce average cross-entropy loss with an iterative optimizer. Its optimization method and model class differ from the threshold example, but both can follow an empirical-loss objective.

Think of it this way

Think of choosing a recipe by scoring several versions in a tasting session and selecting the best under the agreed criteria. The result depends on which recipes were allowed, who tasted them, and how scores were assigned. The analogy is limited because machine learning can search enormous parameter spaces and overfit subtle details of the examples. It nevertheless makes clear that a winning score does not establish universal suitability.

Worked example

Suppose four training cases have labels 0, 0, 1, and 1. Candidate A predicts 0, 0, 0, and 1, making one mistake. Candidate B predicts 0, 1, 1, and 1, also making one mistake. Candidate C predicts 0, 0, 1, and 1, making none. Under zero-or-one loss, their empirical risks are 1/4, 1/4, and 0, so ERM over this three-candidate class selects C. Now consider four new cases with labels 0, 1, 0, and 1. If C predicts 0, 0, 1, and 1 on them, its new-sample error is 2/4, or 50 percent. This does not contradict its zero training risk. The training rule selected C for the observed sample; it did not promise correctness elsewhere. The example also depends on the candidate class: if C were not allowed, A and B would tie as training minimizers.

Illustrative case study

Fictional case: Spruce Routing compares a compact message classifier with a more flexible one. The flexible model achieves fewer training errors, but a reserved evaluation reveals mistakes on messages from a new channel. The team checks data coverage, reviews its model choices, and evaluates regularization and additional representative training examples. It does not deploy solely on the basis of the lowest training loss. This invented case illustrates why ERM belongs within a larger development and evaluation process rather than serving as the entire acceptance decision.

Common misconceptions and mistakes

ERM is not the same as gradient descent. ERM states what to minimize; gradient descent is one possible way to optimize suitable objectives. The method does not require zero training error, nor does every optimizer find a global minimum. A large hypothesis class can fit irrelevant details, while a very restrictive class may miss useful patterns. Reducing average loss can also conceal uneven errors across subgroups. Finally, adding a regularization term changes the optimization objective: its total value should be distinguished from the empirical loss itself when reporting results.

Business use

Ask what loss is being minimized and why that loss is appropriate for the task. Review the training population and permitted model family, not just the optimizer settings. Maintain separation between data used for fitting, tuning, and final evaluation so performance claims remain interpretable. Compare relevant error types and operating costs alongside the average loss. When two models have similar evaluation performance, other requirements such as maintainability, explanation, and resource use may influence selection. Document limitations that optimization cannot resolve, including weak labels, missing scenarios, or an unsuitable decision purpose.

Check your understanding

Two candidate models tie for the lowest training loss. Does ERM alone necessarily identify a unique winner?

Answer: No. Both are empirical-risk minimizers in the candidate class. A tie rule, additional constraints, or other justified selection criteria are needed to choose one.

Related terms

Learn next

Create three simple prediction rules for a tiny labeled dataset and calculate their average losses. Change the loss or restrict the candidate set, then observe whether the chosen rule changes. Finally, test the selected rule on examples that were not used to choose it.

Sources

Aalto Dictionary of Machine Learning, empirical risk minimization: https://aaltodictionaryofml.github.io/ADictML_Main.pdf Shalev-Shwartz and Ben-David, Understanding Machine Learning, sections 2.2–2.3: https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf

Previous: Empirical risk Next: Empowerment (artificial intelligence)