Why Compare Models as a Pair?
A genetic score may be associated with a disease and still add little to predictions already available from age, family history, or clinical measurements. The relevant question is therefore not whether the score predicts the outcome on its own, but whether a model that includes it performs better than a clearly specified clinical-only baseline.
A fair comparison evaluates both models on the same individuals, for the same outcome and prediction horizon, in a validation population relevant to the intended use. The clinical model and the genetic-augmented model must be fitted using training data only; their predictions are then fixed before evaluating the validation outcomes. This avoids letting the validation outcomes influence either model’s predictions.
This signed convention prevents ambiguity: every reported difference should identify both the direction of subtraction and which direction is desirable. The comparison is metric-specific. A model can discriminate cases from noncases better yet have poorly calibrated risk estimates, or have a slightly smaller prediction error without a meaningful change in discrimination.
Paired Comparisons of Discrimination
For a binary outcome, the area under the receiver operating characteristic curve (AUC) summarizes discrimination: how well predicted risks rank cases above noncases. It does not assess whether a predicted risk of, for example, 20% corresponds to an observed event rate of 20%.
AUC differences are naturally paired because each person receives predictions from both models. The estimand is \(\Delta \mathrm{AUC}=\mathrm{AUC}_{\text{genetic}}-\mathrm{AUC}_{\text{clinical}}\), calculated in the same validation population. Comparing AUCs from separate samples would mix model differences with differences in the people evaluated.
Worked Example: A Paired AUC Difference With Ties
Suppose two locked models are evaluated on the same hypothetical validation sample of four cases and four noncases. The clinical model gives the cases risks 0.4, 0.6, 0.7, and 0.9 and the noncases risks 0.2, 0.4, 0.6, and 0.8. The genetic model gives the cases risks 0.4, 0.7, 0.9, and 1.0 and the noncases risks 0.2, 0.4, 0.6, and 0.8. The listed risks are hypothetical predictions, not results from a real study.
There are \(4\times4=16\) case–noncase pairs for each model. Under the clinical model, the four cases contribute 1, 2, 3, and 4 wins, respectively, for 10 correctly ranked pairs. The case risks 0.4 and 0.6 each tie one noncase, giving 2 tied pairs. Thus:
Under the genetic model, the cases contribute 1, 3, 4, and 4 wins, for 12 correctly ranked pairs. The case risk 0.4 ties one noncase, so there is 1 tied pair:
The paired difference is \(0.78125-0.6875=0.09375\), or about 0.094 AUC units. In this small hypothetical sample, the genetic model ranks case–noncase pairs better on average. The result alone does not establish that the improvement is precise or important in a target population; uncertainty and other performance dimensions still need examination.
Calibration and Prediction Error
Calibration concerns agreement between predicted risks and observed outcome frequencies. For binary outcomes, a common summary fits a logistic calibration model in the validation data:
where \(Y\) is the binary outcome, \(\hat{p}\) is the model’s predicted risk, \(a\) is the calibration intercept, and \(b\) is the calibration slope. Ideal values are \(a=0\) and \(b=1\). An intercept below zero indicates that, on average, predicted risks tend to be too high after accounting for the fitted calibration slope; an intercept above zero indicates underprediction. A slope below one commonly indicates predictions that are too extreme, while a slope above one indicates predictions that are not extreme enough. These summaries describe calibration patterns; they do not replace a calibration plot or show every local discrepancy.
A prediction-error summary for binary outcomes is the Brier score, the mean squared difference between predicted risk and outcome coded as 1 or 0. Lower values indicate smaller average squared errors. The score depends on outcome prevalence as well as prediction quality, so comparisons should use the same validation individuals and a population appropriate to the intended prediction target. If evaluation uses case-control sampling, unweighted sample metrics need not represent target-population performance; prevalence and any necessary sampling adjustment must be handled consistently.
Worked Example: Comparing Brier Scores on the Same Participants
Consider six hypothetical validation participants with outcomes \(1,1,1,0,0,0\). Suppose the clinical model predicts risks 0.7, 0.6, 0.4, 0.3, 0.4, and 0.2; the genetic model predicts 0.8, 0.65, 0.5, 0.2, 0.3, and 0.25. For each person, square the difference between predicted risk and observed outcome.
For the clinical model, the six squared errors are \(0.09, 0.16, 0.36, 0.09, 0.16,\) and \(0.04\). Their sum is \(0.90\), so:
For the genetic model, the squared errors are \(0.04, 0.1225, 0.25, 0.04, 0.09,\) and \(0.0625\). Their sum is \(0.605\), giving:
The paired difference is \(0.1008-0.150=-0.0492\) Brier-score units, favoring the genetic model on this sample. This example has three outcomes of each type, so it should not be treated as a prevalence-representative estimate for a population in which the outcome is less common. The comparison is illustrative; transport to a target population requires appropriate validation data or adjustment.
Worked Example: Reading Calibration Results in Context
Suppose calibration models fitted to a hypothetical validation cohort of 1,000 people yield an intercept of 0.12 and slope of 0.82 for the clinical model, and an intercept of 0.04 and slope of 0.90 for the genetic model. These are assumed model outputs for illustration.
The intercept difference, genetic minus clinical, is \(0.04-0.12=-0.08\). The slope difference is \(0.90-0.82=0.08\). Both models’ intercepts are above zero and slopes are below one, so neither has ideal calibration by these summaries. The genetic model’s intercept is closer to zero by 0.08 and its slope is closer to one by 0.08. This suggests improvement in these two calibration summaries, not proof that calibration is adequate at every predicted risk.
The comparisons are meaningful only if both models’ predictions were evaluated on the same cohort and the risk scale corresponds to the same outcome definition and time horizon. In particular, an intercept depends on the event frequency in the evaluated population. Calibration estimates from a case-control sample cannot automatically be interpreted as target-population calibration when the sampling has changed prevalence.
Quantifying Uncertainty in the Differences
A performance difference is estimated from a finite validation sample. Its uncertainty matters: a small observed improvement may reflect sampling variation. Because each participant contributes predictions to both models, uncertainty calculations should preserve this pairing.
The reason pairing matters can be seen from the variance identity:
The covariance term is generally not zero because both metrics are calculated from the same outcomes and participants. Treating the two estimates as if they came from independent samples discards this dependence and can give an incorrect standard error. A paired bootstrap preserves it by resampling participants and recalculating both metrics, and their difference, in every resample.
Specify the target population, outcome, prediction horizon, baseline model, genetic-augmented model, and metric. Use predictions generated without fitting or tuning on the validation outcomes.
Draw a bootstrap sample of participants, retaining each selected participant’s outcome and predictions from both models together. If observations are clustered by family or another sampling unit, resample the appropriate independent clusters rather than treating related observations as independent.
For each resample, calculate the metric for each model and subtract clinical from genetic using the same sign convention. Repeat many times to obtain a bootstrap distribution of the difference.
Use a suitable bootstrap interval, such as percentile quantiles, and report it with the point estimate, population, and metric. If uncertainty from model fitting is also part of the target, model development and tuning must be repeated within the resampling or validation procedure.
Worked Example: Interpreting a Bootstrap Interval for AUC Gain
Suppose a paired bootstrap with 2,000 resamples of a hypothetical validation cohort gives an observed AUC difference of 0.021 and a 95% percentile interval from −0.006 to 0.048. The point estimate is calculated as genetic-model AUC minus clinical-model AUC. In each resample, both AUCs use the same resampled participants and outcomes.
The estimated improvement is 0.021 AUC units, but the interval includes zero and ranges from a small negative difference to a positive difference. The data therefore do not precisely distinguish no improvement from a modest improvement at this uncertainty level. The interval is not a 95% probability statement about the fixed, true difference. It is an interval produced by a procedure designed to capture the difference across repeated samples under its assumptions.
This interval addresses sampling variability in the validation cohort, conditional on the predictions being evaluated. It does not automatically include uncertainty from choosing predictors, tuning model settings, or fitting the models. Those sources require a validation design that repeats the relevant development steps, rather than bootstrapping only the final fixed predictions.
Putting the Metrics Together
A useful report presents the clinical and genetic model estimates side by side, then gives the signed difference and its uncertainty. For binary outcomes, this commonly means AUC for discrimination, calibration intercept and slope (often accompanied by a plot), and Brier score for overall squared prediction error. State the evaluation population and outcome prevalence context, especially when sampling differs from the target population.
| Performance aspect | Example measure | Direction that usually favors the genetic model | What it does not establish alone |
|---|---|---|---|
| Discrimination | AUC difference | Positive | Accurate absolute risks or practical usefulness |
| Calibration | Intercept nearer 0; slope nearer 1 | Closer to ideal values | Agreement at every risk level |
| Prediction error | Brier-score difference | Negative | Performance in a different-prevalence population |
These measures answer different questions and should not be collapsed into a single claim that the genetic model is “better” without specifying the evidence. For example, a higher AUC with worse calibration could mean improved ranking but less reliable absolute risk estimates. A small Brier-score reduction may be driven by calibration, discrimination, or both. The magnitude, uncertainty, and context of each difference should be considered together.
Common Mistakes
- Comparing unrelated samples. The wrong reasoning is to calculate one model’s AUC in one group and the other model’s AUC in another, then call the difference incremental value. The correction is to compare predictions on the same validation participants, or use a design and analysis that explicitly accounts for the distinct samples.
- Ignoring tied AUC predictions. A tied case–noncase prediction is not a failed ranking. It contributes one half of a correctly ordered pair to standard AUC, so omitting ties or counting them as zero changes the metric.
- Treating better discrimination as better risk prediction overall. AUC measures ranking, not calibration or prediction error. Report complementary metrics and interpret each according to what it measures.
- Reporting only a point difference. The observed gain can be unstable. Give an uncertainty interval, and distinguish uncertainty conditional on fixed predictions from uncertainty that includes model development.
- Ignoring the evaluation population. Performance metrics, particularly calibration and Brier score, can change with outcome prevalence. State the population and address case-control sampling when it affects the target estimand.
Check Your Understanding
Try these questions using the definitions and conventions above.
- A case and a noncase receive identical predicted risks. How does their pair contribute to the standard AUC calculation, and why?
- If the genetic model has Brier score 0.12 and the clinical model has Brier score 0.15 on the same participants, calculate the signed difference and state which model is favored by this metric.
- Why should a bootstrap resample retain each participant’s predictions from both models together?
- A genetic model has a higher AUC but a calibration slope farther from one than the clinical model. What can and cannot be concluded from these results?
- A paired 95% interval for an AUC difference includes zero. Explain what this says about the precision of the estimated improvement without claiming that the models are identical.