Can the fit decrease instead?

Yes, with the opposite constraint.

Is it only for probabilities?

No; responses may be other real-valued quantities.

AI Glossary / Statistics and probability / Advanced

Isotonic regression

Definition

Isotonic regression fits values subject to an ordering constraint, usually requiring fitted responses not to decrease as an input increases. In its common least-squares form, it chooses the ordered fitted values that minimize weighted squared differences from the observations. It imposes monotonicity without requiring a straight line or a particular smooth curve.

What it means

Sometimes domain knowledge supports an ordering even though the exact relationship is unknown. More of an input should not produce a lower expected response, but observations may fluctuate because of noise. Isotonic regression finds a fit consistent with the proposed ordering.
Nondecreasing does not mean strictly increasing: several inputs can receive the same fitted value. The constraint describes the fitted relationship, not a guarantee that every future observation will obey the order or that the input causes the response.

How it works

Order observations by their input values and specify the direction and positive weights. For increasing regression, minimize the sum of weight times squared residual, subject to each fitted value being no greater than the next.
The pool-adjacent-violators algorithm merges neighboring blocks whose fitted averages violate the required order. A merged block receives its weighted mean. Further merging may be necessary because fixing one violation can expose another.
The fitted values at observed inputs often contain constant blocks. Predictions between observed inputs require an extension rule. Scikit-learn's IsotonicRegression interpolates between fitted points, producing a piecewise-linear prediction function; the ordering constraint alone does not uniquely specify interpolation or extrapolation.

Three practical examples

An AI team maps a model's scores to estimated event probabilities using separate calibration data, when a monotone relationship is appropriate.
An operations analyst fits expected processing time against an ordered workload measure without assuming that each additional unit has the same effect.
A researcher estimates an ordered response across graded input levels while allowing noisy observations to depart from that order.

Think of it this way

Arrange platforms from left to right so that none is lower than its predecessor. When neighboring platforms violate the rule, combine them at a shared height that balances their observations. Some platforms become level, rather than being forced into a continuous upward slope.

Worked example

Consider an original unweighted example with ordered inputs 1, 2, 3, 4, and 5 and observed responses 1, 4, 2, 3, and 5. Require a nondecreasing fitted sequence.
The response drops from four at the second input to two at the third. Pool those two observations. Their average is three, giving the candidate fit 1, 3, 3, 3, and 5. It satisfies the ordering constraint, so no further change is needed.
The squared residuals are zero, one, one, zero, and zero, totaling two. The original responses have zero fitting error but violate the constraint; they are not an admissible solution. A constant fit of three is admissible but has squared error ten.
Why is three the right common value for the conflicting pair? If both receive value c, their contribution is (4 minus c) squared plus (2 minus c) squared. Expanding gives 2 times (c minus 3) squared plus 2, minimized at c = 3. The neighboring responses one and three permit that choice.
Weights can change the answer. Give the response four a weight of three and every other observation a weight of one. Initially pooling four and two gives a weighted mean of (3 times 4 plus 2)/4 = 3.5.
That value exceeds the next observed response of three, so the block must expand. Pooling responses four, two, and three with weights three, one, and one gives (12 plus 2 plus 3)/5 = 3.4. The resulting fit is 1, 3.4, 3.4, 3.4, and 5.
Its weighted squared error is 3 times 0.6 squared plus 1.4 squared plus 0.4 squared = 3.2. Retaining the unweighted fit at three would give weighted error four. Weighting changes the balance of discrepancies, not just the visual appearance of the curve.
For a small calibration illustration, take increasing model scores 0.1, 0.2, 0.3, and 0.4, with observed binary outcomes 0, 1, 0, and 1. The fitted probabilities at these scores are 0, 0.5, 0.5, and 1. The conflicting middle outcomes are pooled.
Their total squared error is 0.5, giving an average Brier loss of 0.125 on these four calibration observations. This is a fitting calculation, not an independent performance result. Four observations provide very weak evidence about future probabilities, especially fitted endpoints of zero and one.
Under linear interpolation, the fitted value at score 0.15 is 0.25, halfway between the fitted values at 0.1 and 0.2. A stepwise extension could give a different answer while matching the same fitted observations. An input beyond 0.4 also needs an explicitly chosen handling rule; the training fit alone does not justify extrapolation.

Illustrative case study

Fictional company Maple Scoring wants more interpretable probabilities from an existing classifier. It fits a monotone calibration map using data excluded from classifier training and evaluates the result on another suitable held-out set.
The team checks whether improvements persist across relevant groups and later periods. It documents flat regions and endpoint behavior instead of treating the fitted map as guaranteed calibration. This fictional workflow does not claim an observed accuracy improvement.

Common misconceptions and mistakes

Monotone does not mean linear or strictly increasing.
Sorting response values is not the same as fitting them against fixed input positions.
A constrained fit cannot establish a causal dose-response relationship.
Better calibration-set fit does not prove better future calibration.
A monotone mapping can create ties, so it need not preserve every ranking-based metric exactly.

Business use

Use isotonic regression when the ordering constraint is defensible and a flexible fit is useful. Validate the direction, weights, data split, interpolation, and handling of unseen input ranges. For probability calibration, compare held-out calibration and predictive performance rather than assuming monotonicity alone makes outputs trustworthy.

Check your understanding

What replaces the unweighted pair four and two?

Answer: Three and three.

Why does the weighted block expand?

Answer: Its initial mean 3.5 exceeds the next response three.

Does the example Brier loss measure independent test performance?

Answer: No.

Related terms

Monotonicity
Weighted least squares
Probability calibration
Brier score
pool-adjacent-violators algorithm
constrained regression
interpolation

Learn next

Study weighted pooling, tied input values, and held-out calibration assessment before deploying a monotone score-to-probability map.

Sources

Scikit-learn, Isotonic regression: https://scikit-learn.org/stable/modules/isotonic.html . SciPy, isotonic_regression: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.isotonic_regression.html . Scikit-learn, Probability calibration: https://scikit-learn.org/stable/modules/calibration.html . Numerical examples and company scenario are original illustrations.

Previous: Isophote Next: Isotropic position