Why does the loss target a quantile?

Its asymmetric slopes balance the probability mass below and above the prediction at the chosen level when expected loss is minimized.

Can pinball loss assess several quantiles?

Yes. Score each declared quantile and state how any combined summary weights them. Check that predicted quantiles are ordered consistently.

Is a high quantile always preferable?

No. It answers a different question and may imply greater resource allocation. The appropriate target depends on the use.

AI Glossary / Causal inference and experimentation / Intermediate

Pinball loss

Definition

Pinball loss is an asymmetric error measure used to train and evaluate quantile predictions. It penalizes underprediction and overprediction at different rates determined by a chosen quantile level. Minimizing its expected value targets that quantile of the outcome distribution. For example, a 90th-percentile forecast uses a loss that penalizes underprediction more heavily than equally large overprediction. Lower pinball loss is better for the specified quantile and evaluation data.

What it means

A conventional point forecast often aims at a central value. A quantile forecast answers a different question: what value should a specified proportion of outcomes fall at or below, under the modeled conditions?

For instance, a 0.90 quantile forecast of delivery time aims to represent a level exceeded by approximately 10% of outcomes when the model is well calibrated in the relevant setting. It is not a promise that a particular delivery will meet that time.

Pinball loss provides a way to judge such forecasts that respects their intended asymmetry. The name refers to the bent, piecewise-linear shape of the loss graph. It does not involve simulating a physical pinball machine.

How it works

Choose a quantile level alpha between zero and one, such as 0.90. Let the actual outcome be y and the predicted quantile be q. If y is above q, the loss is alpha times the amount of underprediction, alpha × (y − q).

If q is above y, the loss is (1 − alpha) times the amount of overprediction, (1 − alpha) × (q − y). An exact match contributes zero. Average the losses across evaluation examples to obtain mean pinball loss.

At alpha = 0.50, both directions receive weight 0.50, so this convention gives half the absolute error. At higher alpha, underprediction becomes more costly relative to overprediction. Evaluate the same quantile level on the same data when comparing models, and inspect calibration and operational usefulness alongside the average loss.

Three practical examples

1. Delivery planning: a logistics model predicts an upper quantile of travel time, allowing planners to assess a forecast intended to cover most journeys.

2. Inventory forecasting: a retailer predicts a demand quantile reflecting an asymmetric planning preference between shortages and excess stock, while modeling the actual business costs separately.

3. Capacity estimation: a service team evaluates a high-quantile forecast of workload to support planning for busier periods rather than only typical demand.

Think of it this way

Think of a measuring rule that charges different rates for falling short and going too far. If falling short is more costly, its side of the rule is steeper. The analogy is limited because the quantile level defines a statistical target; choosing a high level does not automatically translate every real operational cost into the correct loss or guarantee adequate capacity.

Worked example

Suppose the target is the 0.80 quantile. An actual outcome of 10 with a prediction of 8 is an underprediction of 2, giving loss 0.80 × 2 = 1.6. Predicting 12 for that same outcome overpredicts by 2, giving loss 0.20 × 2 = 0.4.

Now consider three actual outcomes, 8, 10, and 14. Candidate A predicts 10 for each. Its losses are 0.20 × 2 = 0.4, zero, and 0.80 × 4 = 3.2. Mean loss is 3.6/3 = 1.2.

Candidate B predicts 12 for each. Its losses are 0.8, 0.4, and 1.6, for mean loss 2.8/3, approximately 0.93. Under this specified evaluation, B has lower pinball loss. These three invented observations do not establish good calibration or the optimal forecast for a population; they only demonstrate how the asymmetric scoring rule compares the candidates.

Illustrative case study

Fictional case: Maple Deliveries previously assessed every forecast using mean absolute error, including a model intended to predict a high percentile of journey duration. Its reviewers notice that this treats early and late deviations symmetrically despite the forecast's different purpose.

The team adds pinball loss at the declared quantile level and separately checks how often actual durations exceed the predicted value. It also reviews performance for different routes, because an acceptable overall exceedance rate can conceal weak coverage in one area. The case illustrates evaluation alignment, not a real measured improvement or a recommended universal service target.

Common misconceptions and mistakes

Pinball loss is not a probability and its value is not bounded by one. Its units follow the target variable under the stated weighting. A score of 2 for hours cannot be directly compared with a score of 2 for minutes without accounting for scale.

Do not interpret an alpha value of 0.90 as “90% accurate.” It specifies the quantile being targeted. Likewise, low average loss does not by itself demonstrate correct conditional coverage in every subgroup.

The loss at alpha = 0.50 is half mean absolute error under this common convention, not numerically identical to it. Some presentations use different scaling, so check definitions before comparing reported values. Different quantile levels also have different targets and should not be ranked as interchangeable forecasts.

Business use

Pinball loss helps teams evaluate predictions designed for asymmetric planning needs. It can make the distinction between a median forecast and an upper planning estimate explicit in model selection and reporting.

Business users should agree on what the quantile represents and how it will influence decisions. If the forecast is used to set inventory or staffing, evaluate the resulting operational policy too. A useful statistical score does not capture every constraint, cost, or consequence of acting on the forecast.

Check your understanding

At alpha = 0.75, a prediction exceeds the actual outcome by 4. What is the pinball loss?

Answer: It is (1 − 0.75) × 4 = 1. An underprediction of the same size would incur 0.75 × 4 = 3.

Related terms

Learn next

Calculate pinball losses for the same four forecasts at alpha values 0.50 and 0.90. Explain which errors receive more weight in each case and why a lower score at one level does not automatically make it the right planning target.

Sources

Scikit-learn, Metrics and scoring, Pinball loss: https://scikit-learn.org/stable/modules/model_evaluation.html#pinball-loss

Scikit-learn, mean_pinball_loss: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_pinball_loss.html

The primary documentation provides the convention and quantile interpretation. Calculations, fictional case, and exercise are original teaching material.

Previous: Pilot survey Next: Pink noise