Why Measure Incremental Discrimination?
A polygenic score may change a person’s predicted absolute risk after it is added to a clinical model. One question is whether those updated predictions better distinguish people who experience the outcome from those who do not. Discrimination describes this ability to rank people according to outcome status. It is different from calibration, which concerns agreement between predicted and observed risks, and from clinical utility, which concerns decisions and their consequences.
The C statistic summarizes discrimination by comparing the predictions for cases with the predictions for controls. The change in C statistic after adding a polygenic score therefore asks whether the augmented model orders case-control pairs more successfully in the evaluation population. Because the statistic uses ordering, an increase does not by itself show that absolute risks are accurate or that using the score improves decisions.
Calculating the C Statistic
Suppose an evaluation sample contains \(n_1\) cases and \(n_0\) controls, and model \(m\) assigns individual \(i\) a predicted risk \(\hat r_{im}\). For any case-control pair, define its contribution as 1 if the case has higher predicted risk, 0 if the control has higher predicted risk, and \(1/2\) if their risks are tied. The sample C statistic is the average of these contributions over all \(n_1n_0\) pairs.
Here \(Y_i=1\) indicates a case and \(Y_j=0\) a control; \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. A C statistic of 0.5 indicates no better pairwise ordering than chance, while 1 indicates that every case-control pair is ordered correctly. Values below 0.5 indicate systematic ordering in the wrong direction; reversing the risk scale would reverse that ordering.
To measure incremental discrimination, calculate each model’s C statistic in the same evaluation sample and subtract:
A positive value means the augmented model correctly orders more case-control pairs on average in that sample. It does not mean that every individual’s ranking improved. The comparison is meaningful when both models target the same outcome, population, and prediction horizon, and are evaluated using the same cases and controls.
Worked Example: Comparing Ten-Year Risk Models
Consider a hypothetical external evaluation sample with three cases and three controls. Both models predict ten-year absolute risk. The table gives risks as percentages.
| Person | Outcome | Clinical model | Clinical model plus PRS |
|---|---|---|---|
| A | Case | 13% | 24% |
| B | Case | 10% | 18% |
| C | Case | 7% | 11% |
| D | Control | 11% | 12% |
| E | Control | 6% | 9% |
| F | Control | 4% | 5% |
There are \(3\times3=9\) case-control pairs. With the clinical model, case A outranks all three controls, for 3 correctly ordered pairs. Case B outranks E and F but not D, for 2. Case C also outranks E and F but not D, for 2. Thus:
With the augmented model, cases A and B each outrank all three controls. Case C, with risk 11%, outranks controls E and F, but not D, whose risk is 12%. That gives \(3+3+2=8\) correctly ordered pairs:
In this small hypothetical sample, adding the PRS increases the estimated C statistic by 0.1111. Equivalently, the augmented model correctly orders one additional pair out of the nine. This is a sample description, not evidence on its own that the increase would persist in a new population.
Time-Dependent Discrimination
When the outcome is time to an event, discrimination must be defined relative to a time horizon \(t\). A common measure is the cumulative/dynamic time-dependent AUC, also called a time-dependent C statistic. For this definition, cases are people who experience the event by time \(t\), and controls are people who remain event-free beyond \(t\). Predictions must refer to risk by that same horizon.
This measure is horizon-specific: the people classified as cases and controls, and their predicted risks, can differ across horizons. A model can discriminate well at one horizon and less well at another. For an incremental comparison, calculate both models’ time-dependent C statistics at the same \(t\), using the same target population and outcome definition.
Censoring complicates this calculation. A person censored before \(t\) cannot automatically be classified as event-free through \(t\); their status at the horizon is unknown. A complete-case calculation that simply discards such people can be biased if having observed follow-up to \(t\) is related to risk or outcome. Estimators such as inverse-probability-of-censoring weighting account for this incomplete follow-up. They require appropriate modeling of the censoring distribution, conditional independence of censoring and event time given the included covariates, and positivity: people with the relevant covariate patterns must have a nonzero probability of remaining observed through the times needed for estimation. The estimator and assumptions should be reported, rather than treating censored participants as controls by default.
Worked Example: Comparing Five-Year Discrimination
In a hypothetical sample, follow-up is complete through five years and there are no competing events. Three people experience the event by year 5 (A, B, and C), and three remain event-free beyond year 5 (D, E, and F). The models give five-year absolute-risk predictions as follows.
| Person | Status by year 5 | Clinical model | Clinical model plus PRS |
|---|---|---|---|
| A | Event by 5 years | 18% | 30% |
| B | Event by 5 years | 16% | 22% |
| C | Event by 5 years | 10% | 12% |
| D | Event-free beyond 5 years | 20% | 20% |
| E | Event-free beyond 5 years | 12% | 10% |
| F | Event-free beyond 5 years | 6% | 8% |
For the clinical model, A outranks E and F but not D, contributing 2 correct pairs. B also outranks E and F but not D, contributing 2. C outranks F but not D or E, contributing 1. Therefore:
For the augmented model, A and B each outrank all three controls, contributing 3 pairs each. C outranks E and F but not D, contributing 2. Thus:
The estimated five-year discrimination is higher for the augmented model in this hypothetical sample. Complete ascertainment through five years makes the case and control classification explicit here. If some people had been censored before year 5, their status would be unknown and a method that accounts for censoring would be needed.
Ties, Uncertainty, and Validation
The tie rule matters when predictions are rounded, categorized, or otherwise identical for some pairs. A tied case-control pair contributes one-half, not a full success or failure. The next example shows how that convention affects a model comparison.
Worked Example: Counting Tied Pairs
Suppose a hypothetical binary-outcome evaluation has two cases, C1 and C2, and two controls, N1 and N2. The clinical model predicts risks of 20% and 10% for the cases, and 20% and 5% for the controls. C1 ties N1 and outranks N2, contributing \(0.5+1=1.5\). C2 is below N1 but above N2, contributing \(0+1=1\). Hence:
Now suppose the augmented model predicts 24% and 16% for the cases, and 16% and 8% for the controls. C1 outranks both controls, contributing 2. C2 ties N1 and outranks N2, contributing \(0.5+1=1.5\). Therefore:
The augmented model’s estimated C statistic is 0.2500 higher in this sample. The arithmetic treats each tied pair as half correctly ordered; counting ties as full successes would overstate discrimination.
C statistics calculated on the data used to fit a model can be optimistic, particularly when model development includes variable selection or tuning. Prefer an independent evaluation sample or a validation procedure that accounts for model fitting. Since both models are evaluated on the same people, their C estimates are paired: uncertainty for the difference should preserve that pairing. A participant-level bootstrap can resample people and recalculate both C statistics and their difference in each resample. If the goal is to assess a model-development process rather than fixed fitted models, model fitting and tuning must also be repeated within the validation procedure.
The C statistic measures rank ordering, not the accuracy of the predicted probabilities. For example, any strictly increasing transformation of a model’s predictions preserves all pairwise rankings and therefore leaves its C statistic unchanged, even if the transformed risks are poorly calibrated. Thus, a higher C statistic does not establish that a predicted ten-year risk is close to the person’s true absolute risk. It also does not establish that treatment decisions or health outcomes improve; those questions require other assessments.
Common Mistake
Wrong reasoning: “The augmented model has a higher C statistic, so it gives more accurate absolute risks and should be used for clinical decisions.” The C statistic only summarizes case-control ordering. It does not evaluate agreement between predicted and observed risks, nor does it quantify the balance of benefits and harms at a decision threshold.
Correction: Report the change in discrimination as evidence about ranking in the stated population and at the stated horizon. Interpret it alongside calibration and, when relevant, decision-analytic evidence. Ensure that both models are evaluated on the same target population and that the analysis handles censoring and validation appropriately.
Check Your Understanding
Try these questions without looking back.
- A binary-outcome evaluation contains 4 cases and 5 controls. How many case-control pairs contribute to the sample C statistic?
- If an augmented model’s C statistic is 0.81 and the clinical model’s is 0.76 on the same sample, what is the estimated change, and what does it describe?
- At a five-year horizon, how are cumulative/dynamic cases and controls defined?
- Why should a participant censored at year 3 not automatically be counted as event-free beyond year 5?
- A strictly increasing transformation changes all predicted risks but preserves their ordering. What happens to the C statistic, and what important property of the risks does this leave unassessed?