Can bias and variance both decrease?

Yes. Better information, additional data or a different method can improve both; the trade-off describes particular comparisons, not an absolute prohibition.

Does the simple formula apply to every loss?

The familiar additive decomposition is for squared-error settings under stated assumptions. Other losses require different treatment.

Is overfitting identical to variance?

They are related ideas, but overfitting is not a direct measurement of the variance component.

AI Glossary / Governance, ethics and safety / Intermediate

Bias-variance trade-off

Definition

The bias - variance trade-off describes how a learning method's systematic prediction error and sensitivity to its training sample can interact. A change that reduces one component may increase the other, so the useful choice is judged by performance on new data under the intended conditions.

What it means

Here, bias concerns how the average fitted prediction differs from the underlying target relationship across repeated training samples. Variance concerns how much those fitted predictions change when the sample changes. These are statistical concepts, not measures of social unfairness or the offset parameter in a neural-network layer.

A very rigid model may miss a real curved relationship in a similar way across samples. A very flexible model may follow incidental fluctuations, changing its predictions substantially from one sample to another. Neither behavior is desirable merely because it lowers training error.

The familiar trade-off is a useful framework, not a universal law that larger models must always generalize worse after a particular point. Modern learning systems can show more complicated patterns, so actual validation remains essential.

How it works

For squared-error regression under a standard setup, expected prediction error can be decomposed into squared bias, variance and irreducible noise. The expectation concerns repeated training datasets and new outcomes, with assumptions about their distribution. The noise term reflects outcome variation not predictable from the available inputs.

Model flexibility, regularization, training sample size and ensemble methods can alter the components. Stronger regularization often increases bias while reducing variance in a given setting. Averaging suitably varying predictors can reduce prediction variability, although its effect depends on their dependence and individual behavior.

In ordinary projects, the components are rarely known exactly because the true relationship and repeated independent datasets are unavailable. Teams compare candidates using suitable validation and inspect learning curves or stability as supporting evidence. A train - test error gap can be informative, but it does not by itself provide an exact bias - variance decomposition.

Three practical examples

A demand team compares a rigid straight-line model with a more flexible predictor using held-out periods rather than choosing the best training fit.
An analyst tunes regularization to reduce unstable estimates caused by a small and noisy training sample.
A modeling team averages several fitted trees and evaluates whether the ensemble improves prediction stability and new-data error.

Think of it this way

Think of repeatedly estimating a target location from different noisy measurements. One procedure consistently points a little away from the target; another swings widely between attempts.

The analogy represents systematic displacement and variability, but a single real-world model is usually trained only on the data available. The target relationship is also not fully visible, so observed training and test scores cannot be read as exact measurements of the two components.

Worked example

Compare the total, not one component

In an invented squared-error regression example, suppose the irreducible noise contribution is 4 for all three candidate methods. A rigid model has squared bias 9 and variance 1, giving expected error 9 + 1 + 4 = 14.

A moderately flexible model has squared bias 4 and variance 3, giving 11. A highly flexible model has squared bias 1 and variance 10, giving 15. In this teaching setup, the middle candidate has the lowest total even though it has neither the lowest bias nor the lowest variance.

The numbers are assumed component values, not quantities inferred from one training run. They illustrate one possible pattern, not a rule for every neural network or dataset. A larger training sample or a different regularization method could change the comparison and its preferred candidate.

Illustrative case study

Fictional Birch Deliveries predicts packing time from order characteristics. A complex model fits past records extremely closely, but predictions change sharply when a small set of records is replaced. A simpler model is steadier but consistently misses the extra time associated with mixed-item orders.

The team adds a justified interaction feature and compares regularization settings using an evaluation design that respects order timing. It examines both performance and stability, then checks the chosen method on a separate assessment period.

The case illustrates that “simpler” and “more complex” are incomplete prescriptions. Better features, more informative data and appropriate regularization may improve the result. The team describes measured validation performance rather than claiming to know the exact theoretical decomposition.

Common misconceptions and mistakes

Bias means the model's ethical unfairness. The term here has a statistical meaning.
Variance means variation among outcome values. Here it refers to variation in fitted predictions across training samples.
Lowest training error gives the best trade-off. New-data performance is the relevant comparison.
More complexity always increases variance and worsens generalization eventually. That classical pattern is not universal.
More data removes irreducible noise. The noise remaining under fixed information cannot simply be fitted away.

Business use

Businesses can use the framework to understand why a model that appears excellent during development may be unstable in operation. Ask whether the evaluation represents future use and whether the chosen model depends excessively on a small number of training records.

Avoid using the trade-off as a slogan to justify a predetermined model size. Compare reasonable candidates, examine the effect of additional data and features, and preserve an independent assessment. The business choice also includes latency, maintenance and the consequences of errors. A small improvement in a statistical score may not settle those wider considerations.

Check your understanding

Two models have the same noise contribution. Model A has squared bias 2 and variance 8; model B has squared bias 4 and variance 3. Which has the lower expected squared error under this decomposition?

Answer: B, because 4 + 3 = 7 is lower than 2 + 8 = 10 before adding the shared noise.

Related terms

Learn next

Read regularization and cross-validation. Compare two model settings using a valid new-data evaluation and examine prediction stability. Keep observed evidence separate from theoretical component values that you have not measured.

Sources

Scikit-learn, Single estimator versus bagging: bias-variance decomposition: https://scikit-learn.org/stable/auto_examples/ensemble/plot_bias_variance.html. Belkin and colleagues, Reconciling modern machine learning practice and the bias-variance trade-off: https://arxiv.org/abs/1812.11118. These support the classical decomposition and its modern limitations; numbers and business scenarios are original illustrations.

Previous: Bias testing Next: Biased estimator