Why Measure Reclassification?
When a polygenic score is added to a clinical risk model, the new model may assign different risks to the same people. A useful question is whether those changes tend to move people who experience the outcome toward higher risk and those who do not experience it toward lower risk. Net reclassification improvement (NRI) summarizes those directions. It does not, by itself, establish that the new model is better calibrated, more useful for decisions, or beneficial to patients.
Reclassification is assessed by comparing predictions from two models on the same individuals: a reference model, such as a clinical model, and an updated model that adds the polygenic score. Both predictions must target the same outcome, population, and time horizon. For a fixed-horizon analysis, an event is an outcome that occurred by that horizon, and a non-event is one that did not. If outcomes are censored before the horizon, the analysis needs methods that account for censoring rather than simply treating unobserved outcomes as non-events.
Category-Based Net Reclassification Improvement
In category-based NRI, risk boundaries are chosen before assessing reclassification. For instance, categories might be below 5%, from 5% to below 10%, and 10% or higher at a specified time horizon. The boundaries should have a defensible clinical or policy rationale; they should not be selected after inspecting which choices produce the most favorable NRI.
Events and non-events are evaluated separately because a higher predicted risk is generally a desirable direction for an event and an undesirable direction for a non-event. Let \(P(\text{up}\mid\text{event})\) denote the proportion of people with events who move upward, and define the other directional proportions in the same way.
The event component is positive when upward movements outnumber downward movements among people with events. The non-event component is positive when downward movements outnumber upward movements among people without events. The total is their sum. Each component is a difference in conditional proportions; the total is not the proportion of all individuals correctly reclassified.
Worked Example: Three Prespecified Risk Categories
Suppose a hypothetical validation cohort has 500 people: 100 experience the outcome by the target horizon and 400 do not. The reference and updated models use the same predictions’ time horizon. Categories are below 5%, 5% to below 10%, and 10% or higher. Comparing each person’s category under the two models gives these directional counts:
| Outcome group | Moved up | Moved down | Unchanged | Total |
|---|---|---|---|---|
| Event | 24 | 8 | 68 | 100 |
| Non-event | 20 | 60 | 320 | 400 |
Among those with events, the upward proportion is \(24/100=0.24\), and the downward proportion is \(8/100=0.08\). Thus,
Among non-events, downward movement is favorable in this measure. The downward proportion is \(60/400=0.15\), and the upward proportion is \(20/400=0.05\), so
The category-based NRI is \(0.16+0.10=0.26\). In this cohort, the directional balance is favorable in both outcome groups under these category boundaries. It would be incorrect to say that 26% of people were correctly reclassified: the value adds two differences in proportions calculated within separate outcome groups.
Category-Free Net Reclassification Improvement
The category-free, or continuous, NRI does not require risk categories. It counts any increase in predicted risk as upward and any decrease as downward, however small. The same event and non-event formulas apply, with direction determined by the change in predicted risk rather than a change in category.
Worked Example: Category-Free Calculation
In a second hypothetical validation cohort, both models predict the same fixed-horizon outcome for 80 people who experience it and 320 who do not. Comparing the paired risk predictions shows that among events, 30 risks increase, 10 decrease, and 40 are tied. Among non-events, 96 risks decrease, 64 increase, and 160 are tied.
The event component is
The non-event component is
Therefore, the category-free NRI is \(0.250+0.100=0.350\). The value describes the net balance of directions, not the sizes of risk changes. For example, a change from 4.00% to 4.01% counts as upward just as a change from 4% to 9% does.
This sensitivity to tiny movements is an important distinction from category-based NRI. A person’s risk can increase without crossing a category boundary, contributing to category-free NRI but not category-based NRI. Conversely, a modest change that crosses a boundary contributes to category-based NRI. The two measures answer different questions and need not agree.
Why the Formula Adds Two Conditional Components
The direction considered favorable depends on outcome status. The event component subtracts the event group’s downward proportion from its upward proportion; the non-event component reverses that order because downward movement is favorable for non-events. This can be justified by defining a directional score that is \(+1\) for a favorable movement, \(-1\) for an unfavorable movement, and \(0\) for no movement. The mean of this score among events is the event component. Its mean among non-events is the non-event component. Adding these two group-specific means gives the stated NRI.
This construction also explains why the standard total NRI does not weight the components by the proportion of events in the cohort. It adds two conditional summaries, one for each outcome group. It is therefore not an overall proportion or an average weighted by the event prevalence. A positive total can arise even when one component is negative, so both components should be reported alongside the total.
Thresholds, Event Rates, and Calibration
Category-based NRI can depend strongly on category boundaries. If almost nobody crosses the selected boundaries, its components will be near zero, even if predicted risks shift within categories. Moving a boundary can change which people count as upward or downward. Consequently, report the chosen boundaries and their rationale, and avoid presenting a result as if it were independent of that choice.
The event and non-event components are conditional on outcome status, so the standard NRI is not directly weighted by event prevalence. This matters when comparing settings with different event rates: the same pair of conditional components produces the same total NRI, even though the groups make up different proportions of the population. A total NRI should not be read as the population-wide balance of movements or as a clinical utility measure.
NRI also does not test calibration. Calibration concerns agreement between predicted and observed risks; reclassification concerns directions of change between two sets of predictions. A model can move risks in a direction that yields a favorable NRI while assigning poorly calibrated absolute risks. Conversely, a change that improves calibration need not produce a positive NRI. Calibration should therefore be evaluated separately, particularly when the meaning of a risk category depends on its absolute risk level.
Worked Example: A Threshold and Calibration Can Change the Story
Consider a deliberately simple hypothetical cohort of 100 people with a 10% event rate: 10 experience the outcome by the target horizon and 90 do not. Suppose the reference model assigns everyone 4% risk, while the updated model assigns everyone 6%. The observed event proportion is 10%, so both sets of predictions are below that proportion; the updated predictions are closer, but still understate it. Neither model separates individuals by risk.
With a 5% category boundary, every person moves upward under the updated model. Thus, the event component is \(10/10-0/10=1.00\), while the non-event component is \(0/90-90/90=-1.00\). The category-based NRI is \(1.00-1.00=0\). The category-free NRI is also zero because everyone moves upward: its event component is \(+1.00\) and its non-event component is \(-1.00\).
Now use an 8% boundary. Both predictions are below the boundary, so nobody changes category and the category-based NRI is zero because both components are zero. The category-free NRI remains zero, since the risk changes are still upward for everyone. The calculations show that the category-based result depends on the threshold, while neither NRI captures the fact that the updated predictions are closer to the observed event proportion in this simplified example. The measures summarize directional reclassification; they do not replace calibration assessment.
Reporting and Common Mistakes
A useful report states the target outcome and horizon, the population in which both models were evaluated, the risk categories if applicable, the event and non-event counts, both NRI components, and the total. It should identify the analysis as category-based or category-free. Uncertainty should also be assessed: because each person contributes paired predictions, a bootstrap for an interval should resample individuals while keeping their two predictions and outcome together.
- Mistake: “An NRI of 0.26 means 26% of people improved.” The total adds conditional directional differences from the event and non-event groups. It is not a percentage of all people.
- Mistake: “Positive NRI proves the score improves prediction.” NRI reports directional movement, not calibration, the magnitude of risk changes, or overall discrimination. Consider those properties separately.
- Mistake: “Category-based NRI is an intrinsic property of the model.” Its value depends on the risk boundaries. Report and justify them.
- Mistake: “Any increase is an important improvement.” Category-free NRI treats arbitrarily small increases as upward movement. It does not weight changes by magnitude or clinical importance.
- Mistake: “A positive total means both outcome groups benefit.” One component can be negative while the other is sufficiently positive to make the sum positive. Inspect both components.
Check Your Understanding
Answer these questions using the definitions and calculations above.
- Among 120 events, 36 move up and 12 move down. What is the event component of NRI?
- Among non-events, why does the NRI formula subtract the upward proportion from the downward proportion?
- Explain why an NRI of 0.20 does not mean that 20% of the cohort was correctly reclassified.
- How can changing category boundaries alter category-based NRI without changing the risk predictions?
- What can a favorable category-free NRI fail to reveal about the size of risk changes and the calibration of the updated model?