Uncertainty in Validation Metrics

Why Validation Metrics Need Uncertainty Intervals

A validation metric is calculated from a particular set of individuals, so another representative validation sample would generally produce a different value. A reported AUC or Brier score without uncertainty can therefore suggest more precision than the data support. When comparing two prediction models on the same people, the uncertainty calculation must also recognize that the two metrics are linked: both use the same outcomes and individuals.

The target of inference matters. A confidence interval calculated by resampling a fixed validation set describes uncertainty from sampling validation individuals, conditional on the models as fitted. It does not, by itself, include uncertainty from selecting or fitting those models. To assess the performance of a model-building procedure, the fitting and tuning steps must also be repeated within an appropriate resampling design.

Definition: A validation metric is a statistic calculated by comparing model predictions with outcomes in validation data. A confidence interval for a metric describes uncertainty about its target value under a stated sampling model. A paired comparison evaluates the difference between two metrics calculated using the same validation units, preserving the dependence between those estimates.

The three metrics here have different forms. AUC measures how often a randomly selected case is ranked above a randomly selected control. The Brier score is the mean squared difference between an observed binary outcome and its predicted probability; lower values indicate better predictions. For a quantitative outcome, explained variance is commonly summarized by the validation \(R^2\), the proportion of outcome variation accounted for by predictions relative to the sample-mean benchmark; higher values indicate better performance. The details of how these metrics are constructed are less important here than how their sampling uncertainty is estimated.

State the Sampling Unit and the Estimand

Before calculating an interval, identify the unit that can reasonably be treated as independently sampled. If validation participants are unrelated and sampled independently, the participant is usually the unit. If participants include relatives, or if several participants come from the same family or site, outcomes and predictions may be correlated within those groups. Resampling each person independently would then treat correlated observations as independent and may understate uncertainty. In that setting, the family or other appropriate independent cluster should be the resampling unit.

Also specify what is being estimated. For example, an interval for AUC may target performance in a population with a particular case-control composition or may treat the case and control groups as separately sampled. For paired model comparisons, the target is commonly the population difference between the two models’ metrics. The interval’s interpretation depends on the validation sampling design and on how the resampling reflects that design.

Confidence Intervals and Paired Differences

For two metrics \(\hat{\theta}_A\) and \(\hat{\theta}_B\), the estimated difference is \(\hat{\Delta}=\hat{\theta}_B-\hat{\theta}_A\). Its variance is

\operatorname{Var}(\hat{\Delta}) = \operatorname{Var}(\hat{\theta}_B) + \operatorname{Var}(\hat{\theta}_A) - 2\operatorname{Cov}(\hat{\theta}_A,\hat{\theta}_B).

The covariance term is not optional: it reflects the fact that the same individuals contribute to both estimates. Usually, the covariance is positive because some validation samples make both models look better or worse. Ignoring that covariance can produce an incorrect standard error for the difference. The following short derivation shows why.

Justification: Write the estimated difference as \(\hat{\theta}_B-\hat{\theta}_A\). The variance of a difference satisfies \(\operatorname{Var}(U-V)=\operatorname{Var}(U)+\operatorname{Var}(V)-2\operatorname{Cov}(U,V)\). Substituting the two metric estimates gives the displayed formula. Thus, the variance of a paired difference cannot generally be found by adding the two variances alone.

For AUC, a standard approach is DeLong’s method, which estimates the sampling variances of AUCs and their covariance using the case-control comparisons that form the AUC estimates. For a single AUC, a large-sample interval is often formed as estimate plus or minus a normal critical value times its estimated standard error. For a paired AUC difference, use the estimated variance of the difference, including the covariance. This approach assumes independent sampling units within the case and control groups and a sample large enough for the normal approximation to be adequate.

For other metrics, a paired bootstrap is broadly useful. Resample validation units with replacement, retaining each selected unit’s outcome and predictions from both models together. Recalculate both metrics and their difference in every resample. The variation in the bootstrap differences estimates uncertainty in the paired comparison. A percentile interval uses the lower and upper quantiles of those differences; for a 95% interval, these are the 2.5th and 97.5th percentiles. If participants are clustered, resample clusters and retain all members of each selected cluster.

Define the comparison.
State the metric, the direction of subtraction, the population target, and whether the comparison is conditional on fitted models.

Choose the sampling unit.
Use individuals only when they can reasonably be treated as independent. Use families or other independent clusters when observations are dependent within groups.

Preserve pairing.
For each unit selected in a resample, keep its outcome and predictions from both models together. Recalculate both metrics and their difference.

Summarize and interpret.
Report the point estimate and interval in context. An interval containing zero does not establish equal performance; it indicates that the data do not rule out a zero difference at that interval level.

Worked Examples

Worked Example: Paired Comparison of AUC

In a hypothetical validation sample of 100 cases and 100 controls, Model A has estimated AUC 0.74 and Model B has estimated AUC 0.78. A paired DeLong analysis gives standard errors 0.035 for Model A and 0.032 for Model B, and estimated covariance 0.000720 between the AUC estimates. Estimate a 95% confidence interval for the difference, defined as Model B minus Model A.

The estimated difference is \(0.78-0.74=0.0400\). Its estimated variance uses the paired covariance:

\widehat{\operatorname{Var}}(\hat{\Delta}) = 0.032^2+0.035^2-2(0.000720) = 0.001024+0.001225-0.001440 = 0.000809.

Thus, the standard error is \(\sqrt{0.000809}=0.02844\). Using 1.96 as the normal critical value, the margin of error is \(1.96(0.02844)=0.05575\), and the interval is \(0.0400\pm0.05575\), or approximately \([-0.01575,\ 0.09575]\).

The point estimate favors Model B by 0.0400 AUC units, but the interval includes zero. These data do not provide clear evidence of a difference at the 5% two-sided level. This is an estimate of the AUC difference for the stated validation target, not evidence that the models are equivalent.

Worked Example: A Paired Brier-Score Difference

In a hypothetical independent validation sample of 40 unrelated people, the per-person Brier-score difference is defined as Model B’s squared prediction error minus Model A’s. Suppose the 40 differences have mean \(\bar{d}=-0.0180\) and sample standard deviation \(s_d=0.0600\). Estimate a 95% confidence interval for the mean difference. The negative sign would favor Model B because a lower Brier score is better.

The paired analysis uses the differences within people, rather than treating the two sets of scores as independent. The standard error of the mean difference is

SE(\bar{d})=\frac{s_d}{\sqrt{n}} = \frac{0.0600}{\sqrt{40}} = 0.009487.

With \(df=39\), the two-sided 95% \(t\) critical value is approximately 2.023. The margin is \(2.023(0.009487)=0.01919\), giving

-0.0180\pm0.01919 = [-0.03719,\ 0.00119].

Under the usual paired-\(t\) assumptions—that the individual differences are independent across people and their mean is adequately approximated by the \(t\) procedure—this interval includes zero. The estimated difference favors Model B by 0.0180 Brier-score units, but the data do not establish a difference at the 5% two-sided level. If these 40 participants included related individuals, the independence assumption would need to be addressed, for example by a cluster-aware method.

Worked Example: Cluster Bootstrap for Explained Variance

Suppose a hypothetical validation data set contains 120 people from 60 families, with two related people per family. Model A has validation \(R^2=0.315\) and Model B has validation \(R^2=0.347\). The estimated difference, Model B minus Model A, is \(0.347-0.315=0.0320\). To account for within-family dependence, an analyst resamples 60 families with replacement, including both people whenever a family is selected, and recomputes both \(R^2\) values and their difference in each of 2,000 bootstrap samples.

Suppose the 2.5th and 97.5th percentiles of the resulting 2,000 paired differences are \(-0.0120\) and \(0.0730\). The 95% percentile bootstrap interval is therefore \([-0.0120,\ 0.0730]\). The point estimate favors Model B by 0.0320 \(R^2\) units, but the interval includes zero, so the validation data do not clearly establish a difference. Resampling people independently would have broken the family structure and would not match the stated sampling design.

This interval describes sampling uncertainty for the fitted models on new validation families, conditional on those fitted models. It does not include uncertainty from retraining or retuning the models; that would require repeating those steps as part of the resampling procedure.

Assumptions, Interpretation, and Common Pitfalls

A confidence interval procedure is only as credible as its sampling assumptions. DeLong intervals rely on independent sampling units and an adequate large-sample approximation. A paired-\(t\) interval relies on independent differences across units and a sufficiently well-behaved sampling distribution of their mean. Bootstrap intervals rely on the resampling scheme representing the data-generating design; for instance, a family bootstrap is appropriate only when families are the independent clusters relevant to the target.

A 95% confidence interval is a procedure that would cover the target value in about 95% of repeated samples under its assumptions. It does not mean there is a 95% probability that the fixed target lies in this particular computed interval. Likewise, an interval containing zero is not proof that the models perform equally. It indicates that the observed data leave zero difference among values compatible with the procedure at its stated confidence level.

Ignoring shared individuals. Wrong reasoning: calculate separate standard errors for the two models and combine them as if their estimates were independent. Correction: use a paired method that estimates the covariance, or resample both models’ predictions together.
Resampling the wrong unit. Wrong reasoning: resample each participant separately even though several participants are related. Correction: resample independent clusters, preserving all observations within each selected cluster.
Confusing validation uncertainty with training uncertainty. Wrong reasoning: interpret an interval conditional on fixed fitted models as covering variation from retraining and tuning. Correction: state the target; repeat model fitting and tuning within resampling if uncertainty from those steps is part of the target.
Reading overlap of separate intervals as a paired test. Wrong reasoning: infer whether the model difference is significant by checking whether two individual confidence intervals overlap. Correction: calculate an interval directly for the paired difference, because its uncertainty depends on covariance.

Key takeaway: For a comparison on shared validation individuals, estimate uncertainty for the difference itself and preserve the pairing. Choose a resampling or analytic method that matches the validation design, especially when participants are related.

Check Your Understanding

Try these questions before moving on.

Why is the covariance between two AUC estimates relevant when both models were evaluated on the same cases and controls?
For a Brier-score difference defined as Model B minus Model A, which sign favors Model B, and why?
A validation sample includes several siblings from each family. What should be resampled in a cluster bootstrap, and what should remain together?
What does a 95% confidence interval for a paired metric difference containing zero tell you, and what does it not prove?
Does a bootstrap interval that resamples validation individuals while keeping the fitted models fixed include uncertainty from refitting those models? Explain.