Comparing Models On The Same Held-Out Subjects

Why the Comparison Must Preserve Pairing

When two prediction pipelines are evaluated on the same held-out subjects, each subject contributes an outcome and a prediction from both models. The two performance results are therefore paired: for a given subject, the models face the same outcome and often the same subject-specific difficulty. Comparing two overall scores as if they came from unrelated groups discards that pairing and can misstate uncertainty.

The target of a comparison should be specified before calculating it. For an additive subject-level loss, a useful target is the mean difference in loss across the population represented by the held-out sample. If lower loss is better, define the difference so a positive value favors model A. For a score where higher is better, use the corresponding reverse subtraction. This direction convention makes the sign of the result interpretable, but it does not make a difference practically important by itself.

For example, the Brier score for a binary outcome is the squared difference between the observed outcome, coded 0 or 1, and its predicted probability. It is a dimensionless loss, with smaller values better. With the definition above, a positive mean difference in Brier score means model A has lower average Brier loss. For an outcome measured in years, a difference in absolute prediction error would instead be measured in years.

Pairing requires that the two predictions be evaluated against the same subjects and the same outcome values. It does not mean that the models are statistically independent; indeed, their errors are commonly correlated. The held-out subjects must also be independent sampling units for the basic subject-level procedures below. If several observations come from the same patient, family, or site, treating all rows as independent can understate uncertainty.

Paired Mean Differences and Their Uncertainty

Suppose each subject contributes one difference \(d_i\), and the held-out subjects are an independent and identically distributed sample from the population to which the comparison is intended to apply. The sample standard deviation of the differences is \(s_d\), and the standard error of their mean is \(SE(\bar{d})=s_d/\sqrt{n}\). This standard error is calculated from the differences, not by separately estimating uncertainty for each model and combining the results as though the models were evaluated on different people.

The reason pairing matters can be seen from the variance of a difference. If the two subject-level losses have variances \(\sigma_A^2\) and \(\sigma_B^2\), and covariance \(\operatorname{Cov}(L_A,L_B)\), then

Because both models are scored on the same subject, their losses may have positive covariance: a difficult subject may produce a relatively large loss for both. Positive covariance reduces the variance of the difference. An unpaired calculation omits this covariance term and does not use the actual sampling structure.

A paired \(t\) interval for the mean difference is

This interval is exact when the differences \(d_1,\ldots,d_n\) are independent and identically distributed normal observations with a common mean and a common variance. Independence and normality of each difference alone are not sufficient if the differences have unequal variances or different means. For larger samples, the interval is often used as an approximation when the differences are independent and identically distributed with finite variance and no extreme observations dominate the mean. With a small sample, substantial skewness or outliers can make that approximation unreliable.

Metrics That Are Not Subject-Level Averages

Not every performance metric is the average of one score per subject. The area under the receiver operating characteristic curve (AUC), for instance, is based on comparisons between positive and negative subjects. It cannot generally be reduced to a simple average of independent subject-level AUC contributions. Applying the paired \(t\) formula to invented per-subject AUC values is not justified.

A paired subject bootstrap is one way to quantify sampling uncertainty for a difference in such metrics. Resample subjects, retaining each subject’s outcome and both models’ predictions together. For each resampled data set, calculate both metrics and their difference. The same resampled subjects and multiplicities must be used for both models in every replicate; otherwise the pairing is broken. For AUC, resamples must contain both outcome classes. A stratified bootstrap resamples within each outcome class and targets uncertainty conditional on the observed class counts; an unstratified bootstrap instead reflects variation in class counts as well, provided the replicates permit the metric to be calculated.

Worked Comparison of Paired and Unpaired Uncertainty

The effect of dependence can be substantial even when the observed model performances are similar. For a simple illustration, suppose subject-level losses have standard deviations \(0.10\) for A and \(0.09\) for B, and their within-subject correlation is \(0.80\). The covariance is \(0.80(0.10)(0.09)=0.0072\). Thus, the variance of the paired loss difference is

The standard deviation of a paired difference is \(\sqrt{0.0037}=0.06083\). For \(n=100\) independent subjects, the standard error of the mean paired difference is \(0.06083/\sqrt{100}=0.006083\). By contrast, a calculation that incorrectly treats the two model evaluations as independent gives \(\sqrt{(0.10^2+0.09^2)/100}=0.01345\). The latter is more than twice as large in this particular setting because it discards positive within-subject correlation. The direction and magnitude of the error are not guaranteed in all settings; the correct calculation follows the paired sampling design.

Assumptions, Diagnostics, and Scope

A paired comparison is informative only when the evaluation design supports it. The following checks address different potential failures:

• Confirm the pairing. Verify that each included subject has an outcome and predictions from both models, and that the same outcome definition and eligibility rules were used. Do not pair records by row position unless the subject identifiers confirm the match.
• Check the sampling unit. If observations are repeated within a patient or clustered within a site, the rows may not be independent. Use a method that resamples or models the independent units, such as resampling patients or sites as appropriate, rather than treating repeated rows as separate subjects.
• Inspect the differences, not just the model scores. For a paired \(t\) interval, examine a plot or summary of \(d_i\) for skewness, outliers, and data errors. With small \(n\), these features can undermine the normal approximation. An extreme difference may be real, so investigate it rather than deleting it automatically.
• Check the metric and its direction. Confirm that a larger difference consistently favors the intended model and that the metric is suitable for the outcome and class balance. A mean difference in a loss is not the same estimand as a difference in AUC or calibration.
• Keep the held-out evaluation separate from model development. Pairing addresses uncertainty in the comparison on shared evaluation subjects; it cannot repair leakage, test-set reuse, or model selection based on held-out outcomes.

The paired \(t\) interval or subject bootstrap generally describes sampling variability across held-out subjects conditional on the fitted models and the evaluation procedure used. It does not, by itself, include variability from drawing a new training set, refitting the pipelines, or repeating model selection and tuning. If those sources are part of the intended target, the evaluation design must reproduce them when estimating uncertainty. Likewise, if the held-out subjects are a fixed finite set and the only target is performance on exactly that set, the computed difference is descriptive for that set; a sampling interval requires a population-sampling interpretation.

The main alternative to the paired approach is an unpaired comparison, which is appropriate only when the models are evaluated on distinct independent samples and the target is defined accordingly. It is not an acceptable shortcut for shared subjects. Bootstrap procedures are useful for complex metrics, but they are not automatically reliable in tiny samples or when the resampling scheme fails to represent the sampling design. The method should follow the estimand and dependence structure, rather than the convenience of a software function.

Common Mistakes

• Using separate standard errors for two scores on shared subjects. This ignores the covariance between the model results. Calculate subject-level paired differences, or use a paired resampling procedure for a non-additive metric.
• Assuming normality alone gives an exact paired \(t\) interval. Exactness requires independent and identically distributed normal differences with a common mean and variance. Independent normal differences with unequal variances do not in general yield the usual \(t\) distribution.
• Interpreting an interval as a probability statement about the parameter. A 95% confidence procedure has 95% long-run coverage under its assumptions; it does not mean there is a 95% probability that this particular interval contains the fixed population mean.
• Confusing statistical evidence with practical value. An interval excluding zero does not establish that the difference is large enough to matter clinically or operationally. Interpret its units and compare it with a relevant practical threshold.
• Treating every row as an independent subject. Repeated measures or clustered observations can create dependence beyond the pairing of models. Resample or analyze the independent sampling units.