Must the new model contain the old model?

The movement calculation can compare two sets of predictions, but that does not guarantee valid inference or useful interpretation.

Does positive NRI mean every group improves?

No. The nonevent component in the worked example is negative.

AI Glossary / Statistics and probability / Advanced

Net reclassification improvement

Definition

Net reclassification improvement, or NRI, compares how two risk-prediction models move observations upward or downward in predicted risk. It adds an event component and a nonevent component. Category-based NRI uses movement between specified risk categories; continuous NRI uses the direction of risk changes without requiring category crossings. NRI is a sum of differences in proportions, not itself a proportion.

What it means

Upward movement counts favorably among observations that experience the event; downward movement counts favorably among those that do not. The two groups contribute separately, regardless of their relative sizes. A positive total can therefore coexist with worsening performance on another measure. NRI should not be described as the percentage of people whose predictions became correct.

How it works

For completely observed binary outcomes, subtract the proportion moving down from the proportion moving up among events. Among nonevents, subtract the proportion moving up from the proportion moving down. Add the two components. Specify risk thresholds for categorical calculations and how ties are handled. For continuous NRI, unchanged risks contribute neither an upward nor a downward movement.

Three practical examples

An analyst compares two fraud-risk models using prespecified review categories. The report shows event and nonevent movements separately so a gain in fraud detection does not hide more unnecessary reviews.
A reliability team compares predicted failure risks over the same fixed follow-up period. Units with incomplete follow-up require a method that handles censoring rather than being automatically labeled nonfailures.
An AI validation team reviews a reported continuous NRI. It checks the sizes of risk changes, calibration, and operational thresholds before interpreting a positive value as a useful improvement.

Think of it this way

Imagine keeping two scorecards: one for actual events and one for actual nonevents. Each scorecard records favorable moves minus unfavorable moves, using its own group size as the denominator. Adding the scorecards gives both groups equal prominence even when one is much larger. This is different from counting how many observations overall benefited.

Worked example

Consider an original held-out evaluation of 100 transactions, including 20 confirmed fraud events and 80 nonevents. Two fixed models assign each transaction to low or high predicted risk using the same prespecified threshold. For this example only, high means the positive classification and low means the negative classification.

Among the 20 events, eight move from low to high, two move from high to low, and ten stay in the same category. The event NRI is (8-2)/20 = 0.30. Among the 80 nonevents, eight move from high to low, sixteen move from low to high, and fifty-six stay in their category. The nonevent NRI is (8-16)/80 = -0.10. Total NRI is therefore 0.20.

That total is not a 20-percentage-point improvement in classification accuracy. Favorable classification changes number 8+8 = 16; unfavorable changes number 2+16 = 18. The net change in correct classifications is -2 out of 100, or minus two percentage points. The positive NRI and negative accuracy change answer different questions because the denominators differ.

Suppose the old model classifies six events high and fourteen low. Four of those six remain high, while two move down; eight of the fourteen move up, while six remain low. The new model therefore has twelve true positives and eight false negatives. Sensitivity increases from 6/20 = 0.30 to 12/20 = 0.60.

Suppose the old model classifies twenty nonevents high and sixty low. Twelve of the twenty remain high and eight move down. Of the sixty low-risk nonevents, sixteen move up and forty-four remain low. The new model has twenty-eight false positives and fifty-two true negatives. Specificity decreases from 60/80 = 0.75 to 52/80 = 0.65.

Thus the sensitivity change is +0.30 and the specificity change is -0.10, matching the two-category NRI components. Accuracy changes from (6+60)/100 = 66% to (12+52)/100 = 64%. Balanced accuracy, the average of sensitivity and specificity, changes from 52.5% to 62.5%. In this two-category setting, its increase is half the NRI.

Assign a hypothetical cost of ten units to each missed fraud and one unit to each false alert. The old total is 14 times 10 plus 20 = 160 units. The new total is 8 times 10 plus 28 = 108 units. If both errors instead cost one unit, totals are 34 and 36. These invented costs demonstrate why a deployment decision needs explicit consequences; the NRI itself contains neither cost assumption.

For a separate continuous-NRI illustration, give every observation an old risk of 0.20. Set new risks to 0.200001 for events and 0.199999 for nonevents. Every movement has the favorable sign, so continuous NRI equals 1+1 = 2, its maximum. Yet no observation crosses a 0.50 action threshold. These synthetic risk vectors illustrate the metric's sensitivity to direction and disregard for movement size; they are not a fitted model or evidence of usable prediction performance.

Illustrative case study

This fictional case concerns Cedar Alerts, an imaginary fraud-review team. A presentation headlines the first example as a 20% accuracy improvement. A reviewer reconstructs the category tables and corrects the statement: NRI is 0.20, while accuracy falls by two percentage points.

The team adds calibration and workload evidence. This fictional case illustrates reporting discipline.

Common misconceptions and mistakes

NRI values range from -2 to 2, so reporting them as ordinary percentages can be misleading. Different category thresholds can change a categorical NRI, and continuous NRI is not interchangeable with categorical NRI.

A statistically positive result does not demonstrate calibration, useful decision-making, or causal benefit. Published methodological critiques identify serious interpretation and inference problems. A single NRI value should not determine whether a new model is adopted.

Business use

Require the complete reclassification tables, separate components, evaluation population, prediction horizon, and threshold definitions. Evaluate calibration and performance on data appropriate to the deployment setting. Decisions should account for false-alert workload and missed-event consequences, with uncertainty assessed through a suitable validation and analysis procedure.

Check your understanding

Why is NRI positive while accuracy falls?

Answer: Events and nonevents use separate denominators, whereas accuracy weights each transaction equally.

What does continuous NRI ignore?

Answer: The magnitude of each nonzero risk change.

Related terms

Calibration
Risk reclassification
sensitivity
specificity
balanced accuracy
decision threshold
net benefit
external validation

Learn next

Reconstruct both two-by-two classification tables and compare accuracy, balanced accuracy, and the two hypothetical cost totals before interpreting NRI.

Sources

Eisuke Inoue, nricens package documentation, nribin definitions and variants: https://cran.r-project.org/web/packages/nricens/nricens.pdf . Kathleen F. Kerr, Net Reclassification Index Statistics Do Not Help Assess New Risk Models, published abstract and figure captions: https://pubmed.ncbi.nlm.nih.gov/36378029/ . Transaction data, costs, and Cedar Alerts are original illustrations.

Previous: Nesterov accelerated gradient Next: Network Data Envelopment Analysis