Can more data eliminate approximation error?

Not for a fixed class and fixed population under this definition.

Does enlarging a nested class increase its ideal minimum risk?

No; it retains the smaller class's options.

Is approximation error always above Bayes risk?

Under the excess convention it is the difference above optimal risk; another convention includes that baseline in the named quantity.

AI Glossary / Statistics and probability / Advanced

Hypothesis class approximation error

Definition

Hypothesis class approximation error is the limitation imposed by restricting prediction to a particular family of functions. Under a common convention, it is the difference between the best achievable population risk within that class and the best achievable risk without that restriction. Some texts instead call the class's minimum risk itself approximation error, including irreducible error. The convention, loss function, and data distribution must therefore be stated.

What it means

A learning algorithm cannot select a predictor that its allowed hypothesis class cannot represent. If a class contains only straight lines, even unlimited data and perfect optimization may not recover a genuinely curved relationship under the chosen loss.
This is a representational limitation. It differs from estimation error caused by learning from finite data and from optimization error caused by failing to find a sufficiently good solution. In practice these effects are difficult to separate exactly because the true data-generating distribution and best possible predictor are usually unknown.

How it works

Let R(h) denote a predictor's expected loss on the population. For a hypothesis class H, define R_H* as the infimum of R(h) over h in H. Define R* as the unrestricted optimal risk for the same prediction problem and information available.
The excess-risk convention gives approximation error R_H* − R*. A learned predictor's excess risk can then be written as R(h_hat) − R* = [R_H* − R*] + [R(h_hat) − R_H*]. The second term measures how far the learned predictor is above the best member of its class; further decompositions require care about the learning and optimization setup.
If one class contains another, its best achievable risk cannot be higher, because all the smaller class's choices remain available. That statement concerns an ideal population optimum, not guaranteed performance after training on a finite dataset.
Choose or enrich a class using domain knowledge and validation. Additional features can change what the class can represent, while regularization and optimization affect the solution actually obtained. More observations alone do not remove a fixed class's representational limit, although they can improve estimation within it.

Three practical examples

A linear predictor cannot exactly represent a curved relationship unless its features or function class are expanded appropriately.
A shallow decision-tree class may be unable to express distinctions that require more splits.
A model using insufficient inputs can face an information limit even when its function family is highly flexible, requiring careful separation of representation and missing-information issues.

Think of it this way

Think of trying to draw a circle using only one straight line. More measurements can tell you where to place the line, but they cannot turn the permitted shape into a circle. Expanding the allowed shapes addresses a different problem from measuring the points more accurately.

Worked example

Consider an invented population where x is equally likely to be −1, 0, or 1 and y = x². Use squared-error loss and restrict predictions to affine functions h(x) = ax + b.
The population risk is [(1 + a − b)² + b² + (1 − a − b)²] / 3. Expanding and grouping gives [2(1 − b)² + 2a² + b²] / 3. This is minimized at a = 0 and b = 2/3.
The three squared errors are then 1/9, 4/9, and 1/9, giving risk (6/9) / 3 = 2/9, approximately 0.2222. A class that includes h(x) = x² achieves zero risk, so the affine class's approximation excess is 2/9.
Now add independent zero-mean noise with variance 0.1 to y. Under squared loss, the unrestricted optimal predictor remains x², but its risk is 0.1. The affine class's minimum risk becomes 2/9 + 0.1, approximately 0.3222. Its excess above the unrestricted optimum remains 2/9. This makes the terminology distinction visible: the class minimum and its approximation excess are different quantities when irreducible noise is present.

Illustrative case study

In this fictional case, Cedar Forecasting uses a linear model for a process with a known curved response. Collecting more observations reduces variability between fitted models, but a systematic pattern remains in residuals.
The team tests a justified nonlinear feature and compares results on held-out data. It also checks whether measurement error, missing inputs, or distribution changes could explain the pattern. A more flexible model is not accepted simply because its training error falls.
The report describes evidence consistent with a limitation of the original class, while acknowledging that real data do not reveal the exact population approximation error as neatly as the teaching example.

Common misconceptions and mistakes

Approximation error is not the observed training error. A model can fit its sample well and still have poor population performance. Conversely, a nonzero training loss does not prove that the hypothesis class is inadequate; optimization or noisy observations may contribute.
Another mistake is assuming that a larger class always yields a better learned predictor. Its ideal minimum risk cannot increase under class inclusion, but finite-data estimation and computation can make the fitted result worse.
The term also has competing conventions. Comparing numbers without checking whether irreducible risk has been subtracted can create a false disagreement.

Business use

The concept helps teams decide whether to investigate model capacity, features, data quantity, or training quality. It discourages repeatedly collecting data when the central limitation may be the permitted representation.
Use learning curves, residual analysis, and controlled comparisons as diagnostics rather than exact decompositions of unknown population error. Evaluate the operational trade-offs of a richer class, including stability, interpretability, computation, and performance on new data.

Check your understanding

What is the best affine predictor in the example?

Answer: The constant 2/3.

What is its noiseless squared-error risk?

Answer: 2/9.

What changes when noise variance 0.1 is added?

Answer: Both optimal risks rise by 0.1, while the approximation excess remains 2/9.

Related terms

Bayes risk
Estimation error
Inductive bias
Underfitting
Model capacity
hypothesis class
population risk
optimization error

Learn next

Learn next how approximation and estimation interact in model selection. Compare error-decomposition conventions and study why empirical validation cannot directly reveal every component of population risk.

Model selection

Sources

Shalev-Shwartz and Ben-David, Understanding Machine Learning: From Theory to Algorithms, Section 5.2 and its convention note: https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf

Previous: Hypothesis Next: Hypothesis prior