Why Pair the Comparison?
Suppose two joint models produce predictions for the same people at the same prediction horizon. Their performance estimates are linked: both models are evaluated against the same observed outcomes, and some subjects will be easy or difficult for both. A comparison that ignores this pairing discards information and can give an inappropriate estimate of uncertainty.
The target is usually a difference in a performance measure in a specified validation population and at a specified horizon. The models, prediction time, outcome definition, evaluable subjects, and metric must all be held fixed across the comparison. If the models are evaluated on different cohorts or at different horizons, the difference mixes model performance with differences in the evaluation conditions.
The contrast is an estimate for the chosen metric, horizon, and validation setting; it is not a universal ranking of the models. Discrimination, calibration, and prediction error measure different aspects of performance, so a model can improve one and worsen another. The earlier material on uncertainty intervals for prediction metrics and bootstrap confidence intervals provides useful background for the uncertainty procedures used here.
Paired Uncertainty for Subject-Level Scores
For a score that can be calculated separately for each subject, let \(q_{ij}\) be the score for subject \(i\) under model \(M_j\). For example, a Brier score is a squared prediction error, so smaller values are better. Define the subject-level difference \(d_i=q_{i2}-q_{i1}\). The estimated paired difference is the mean of these within-subject differences.
When subjects are independent and the differences have finite variance, the estimated standard error is \(SE(\bar d)=s_d/\sqrt{n}\), where \(s_d\) is the sample standard deviation of the \(d_i\). A conventional \(t\)-interval is \(\bar d\pm t_{n-1,\,1-\alpha/2}s_d/\sqrt{n}\) when a normal approximation for the mean difference is reasonable. For bounded or skewed metrics, small samples, or complex censoring adjustments, a subject-level bootstrap is often preferable.
The reason pairing can improve precision follows directly from the variance of a difference. If the two metric estimates have variances \(V_1\) and \(V_2\), then
When the estimates tend to move together, their covariance is positive, and the variance of their difference is smaller than it would be if that covariance were ignored. This is why calculating two separate confidence intervals and comparing their overlap is not a valid substitute for an interval for the paired difference.
Worked Example: Paired Brier-Score Difference
In a hypothetical validation sample, both models predict event risk by the same horizon for six fully observed subjects. The table gives each subject’s Brier score; smaller is better.
| Subject | \(M_1\) | \(M_2\) | \(d_i=q_{i2}-q_{i1}\) |
|---|---|---|---|
| 1 | 0.12 | 0.18 | 0.06 |
| 2 | 0.30 | 0.24 | -0.06 |
| 3 | 0.20 | 0.16 | -0.04 |
| 4 | 0.25 | 0.31 | 0.06 |
| 5 | 0.10 | 0.08 | -0.02 |
| 6 | 0.35 | 0.27 | -0.08 |
The mean scores are \(1.32/6=0.2200\) for \(M_1\) and \(1.24/6=0.2067\) for \(M_2\). Thus
The paired differences have sample standard deviation \(s_d=0.06022\), so \(SE=0.06022/\sqrt{6}=0.02459\). With \(df=5\), the 95% \(t\)-interval uses \(t_{5,0.975}=2.571\):
The point estimate favors \(M_2\), because its Brier score is lower by 0.0133, but the interval includes zero and also includes differences favoring \(M_1\). This small example therefore gives imprecise evidence about which model has lower mean Brier score. The calculation assumes independent subjects and complete outcome ascertainment at the horizon; neither model’s scores are treated as coming from a separate sample.
Paired Bootstrap for Whole-Sample Metrics
Not every metric has a natural subject-level score whose average equals the metric. The area under a receiver operating characteristic curve, for example, depends on comparisons between subjects. Calibration summaries may also depend on the entire validation sample. For such metrics, preserve the pairing by resampling subjects, not by resampling the two models’ results separately.
In each bootstrap replicate, sample subjects with replacement from the validation cohort. Keep each selected subject’s outcome and both models’ predictions together, calculate the metric for each model on that same resample, and store their difference. Percentiles of the resulting differences can provide a confidence interval. If subjects are clustered within centers and inference is intended to generalize across centers, resample centers as clusters rather than treating all subjects as independent.
Specify the horizon, eligible validation subjects, outcome definition, metric, and order of subtraction.
Use the same subjects and outcome information for \(M_1\) and \(M_2\), then record \(Q(M_2)-Q(M_1)\).
For each bootstrap replicate, resample subjects or independent clusters while retaining both predictions and the outcome for every selected unit.
Use the bootstrap distribution to estimate uncertainty, and interpret its interval in the metric’s units and direction.
Worked Example: Comparing Discrimination
A hypothetical validation analysis evaluates discrimination for event status by a fixed horizon. The observed AUCs are 0.71 for \(M_1\) and 0.76 for \(M_2\). The paired point estimate is
Because larger AUC indicates better discrimination, this point estimate favors \(M_2\) by 0.050 AUC units. In 2,000 paired subject-level bootstrap replicates, suppose the 2.5th and 97.5th percentiles of the AUC differences are -0.010 and 0.110. The resulting 95% percentile interval is \([-0.010,\ 0.110]\). It includes zero, so the data do not establish a difference in discrimination at the 5% two-sided level using this interval. The interval is compatible with a small advantage for \(M_1\) as well as a larger advantage for \(M_2\).
This procedure is appropriate only if resampling subjects represents the sampling process of interest and the outcome status at the horizon is observed or handled appropriately. The two AUCs must be recalculated on the same bootstrap sample in every replicate. Bootstrapping each model independently would lose the dependence created by their shared subjects and outcomes.
Comparing Calibration and Prediction Error
For prediction error, use a metric with a clear direction, such as the Brier score or an integrated Brier score: lower is better. For calibration, first specify which calibration summary is being compared. One simple horizon-specific summary is the absolute difference between mean predicted risk and the observed event proportion. This summarizes calibration in the large, but it does not establish that predictions are calibrated across the full range of risk.
For censored outcomes, event status by a horizon may not be known for every subject. Inverse-probability-of-censoring-weighted scores can address this under the censoring assumptions used by the score. The same outcome handling and weights must be used for both models within each replicate. If censoring weights are estimated, a bootstrap intended to reflect their estimation uncertainty should repeat that estimation as part of the replicate’s analysis.
Worked Example: Comparing a Calibration Summary
In a hypothetical cohort of 200 subjects, event status is known for everyone at the chosen horizon, and 42 subjects have experienced the event. The observed event proportion is \(42/200=0.210\). The average predicted risks are 0.200 for \(M_1\) and 0.240 for \(M_2\). Define the calibration summary as the absolute difference between mean predicted risk and observed event proportion. Then
Smaller values indicate closer agreement for this summary, so the point estimate says that \(M_2\)'s absolute discrepancy is 0.020, or 2.0 percentage points, larger than \(M_1\)'s. Suppose a paired subject-level bootstrap gives a 95% interval for \(\Delta_C\) of \([0.003,\ 0.036]\). Under the stated summary and validation setting, this interval indicates a larger discrepancy for \(M_2\). It does not show that every risk level is better calibrated under \(M_1\), nor does it identify the direction of the discrepancy without examining the signed differences.
Assumptions, Diagnostics, and Limits
A paired comparison is meaningful when both models target the same outcome, horizon, and validation population, and predictions are available for the same evaluable units. If missing predictions or exclusions leave different subjects in the two metric calculations, the contrast may no longer be a within-subject comparison. State how the analysis handles such cases; do not silently compare metrics based on different denominators.
For standard subject-level standard errors or subject bootstrap intervals, the independent sampling units must be correctly identified. Repeated measurements from one person do not create independent subjects. When the target population includes variation between centers, center clustering may need to be represented in the resampling scheme. For survival prediction metrics, also check whether censoring before the horizon is addressed by a method whose assumptions are credible in the validation population.
Several practical checks help reveal problems:
- Confirm that the model ordering in the reported difference matches the stated sign interpretation.
- Check that both models use the same subjects, prediction horizon, event definition, and censoring procedure.
- Inspect the paired differences or bootstrap distribution for extreme values, skewness, or a few influential subjects.
- Report the point estimate and interval, not only whether an interval excludes zero.
- If comparing many horizons or metrics, describe that multiplicity and avoid presenting a selected favorable comparison as if it had been the only planned one.
A paired comparison answers a narrower question than a full evaluation of either model. A difference in AUC does not determine which model is better calibrated or has lower prediction error. A single calibration summary cannot describe calibration over all predicted risks. And uncertainty from a validation sample does not automatically include uncertainty from the model-development process: if the intended target includes refitting models on new development samples, the validation procedure must account for that additional source of variation.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For a lower-is-better score, model \(M_1\) has score 0.18 and \(M_2\) has score 0.15. Using the contrast \(M_2-M_1\), which model is favored, and what is the estimated difference?
- Why should both models’ predictions remain together when subjects are resampled for a bootstrap comparison?
- A paired 95% interval for an AUC difference \(M_2-M_1\) is \([0.02,\ 0.08]\). What does it imply, and what does it not imply?
- Why is the absolute difference between mean predicted risk and observed event proportion not a complete assessment of calibration?
Solutions. (1) The difference is \(0.15-0.18=-0.03\), so \(M_2\) is favored by 0.03 score units because lower is better. (2) The subjects share outcomes and contribute to both models’ metrics; keeping predictions paired preserves the covariance needed for uncertainty in their difference. (3) Since higher AUC is better and the interval is entirely positive, it supports higher discrimination for \(M_2\) in the stated validation setting; it does not establish better calibration or lower prediction error. (4) Mean predicted risk can match the event proportion even if predictions are systematically too low in one risk range and too high in another, so the summary does not describe calibration across the risk distribution.