Uncertainty in Validation Performance Comparisons

Why Validation Scores Need an Uncertainty Measure

A validation score summarizes how a model performed on a particular set of observations. It is an estimate, not a fixed property of the model: a different validation sample from the same target population would generally produce a different score. That variation matters when a score is used to assess predictive performance or to describe a difference between two models. A small difference in observed scores may be less informative than a larger difference if the validation estimate is highly variable.

Bootstrap resampling provides a practical way to estimate this variability from the observed validation data. The key design choice is the resampling unit. If validation observations are independent policies, resample policies. If several records belong to the same policy or risk and are dependent, resample the whole policy or risk cluster, retaining all its records together. Resampling dependent records one at a time can understate uncertainty.

Bootstrap Procedure for a Validation Score

Suppose there are \(n\) independent validation units and unit \(i\) has loss score \(L_i\) and, when appropriate, weight \(w_i\). An exposure-weighted aggregate score is

The weights and the score definition must match the intended performance measure. For example, an average policy-level score gives each policy equal weight, whereas an exposure-weighted score gives more influence to units with more exposure. The bootstrap must recompute the same statistic, including its weights, in every replicate.

For bootstrap replicate \(b\), draw \(n\) units with replacement from the observed units, and calculate \(T^{*(b)}\). If \(B\) replicates are generated, the bootstrap standard error is the sample standard deviation of those replicate statistics:

This standard error describes the spread of bootstrap scores. It is not the standard deviation of the bootstrap scores divided by \(\sqrt{B}\); that latter quantity concerns Monte Carlo error in estimating the bootstrap mean from a finite number of replicates. In practice, use enough replicates that Monte Carlo noise is small relative to the uncertainty being reported.

A percentile interval is an approximate interval, not a guarantee of exact coverage. It can be affected by bias, skewness, a small number of independent units, or a score with unusual boundary behavior. A large number of bootstrap replicates reduces simulation noise but does not create more independent information than the original validation sample contains.

Why the Bootstrap Standard Deviation Estimates Uncertainty

The bootstrap treats the observed validation units as an empirical approximation to the population from which they were sampled. Resampling from that empirical distribution mimics the act of drawing another validation sample. The resulting variation in \(T^{*}\) approximates the variation that \(T\) would have across repeated samples.

The variance formula follows directly from the usual sample variance applied to the bootstrap statistics. If the empirical bootstrap distribution assigns probability \(1/B\) to each replicate, its center is \(\bar{T}^{*}\); the squared deviations from that center measure how much the statistic changes across resamples. Dividing their sum by \(B-1\) gives the sample variance of the \(B\) simulated values, and taking its square root gives their standard deviation. As \(B\) increases, this calculation more stably estimates the spread of the bootstrap distribution. It does not remove uncertainty due to having observed only a finite validation sample.

Paired Uncertainty in Score Differences

When two models are scored on the same validation units, uncertainty in their difference should preserve that pairing. Let \(L_{Ai}\) and \(L_{Bi}\) be the losses for models A and B on unit \(i\). Define \(\Delta=T_A-T_B\). If lower scores are better, a positive \(\Delta\) favors model B. For each bootstrap replicate, draw one resample of validation units and calculate both model scores on those same units; then subtract the scores. The standard deviation of those replicate differences estimates uncertainty in \(\Delta\).

The reason pairing matters is that the two scores on the same unit are often related. For simple averages, the variance of their difference is

Using common resamples retains this covariance. Resampling separately for each model discards the pairing and estimates a different, generally less appropriate quantity. The paired bootstrap estimates uncertainty in the difference on common validation units; it does not by itself establish that the models or their validation periods are otherwise comparable.

Resampling Policies or Risk Clusters

A policy can contribute multiple records, such as multiple coverages, locations, or periods. If those records are dependent, the independent unit for the bootstrap is the policy, not the record. Draw policies with replacement and include all records belonging to each selected policy. When a policy is selected more than once, its full set of records appears that many times in the replicate. Recompute the score from the resampled records, including any exposure weighting.

The same principle applies when the independent unit is a broader risk cluster, such as an insured account whose policies share common experience. The resampling unit should reflect the dependence structure relevant to the validation sample. If the data have important time dependence or a sampling design that is not captured by simple independent clusters, an ordinary cluster bootstrap may not be adequate; the resampling method must be adapted to that design.

Interpretation and Practical Limits

A bootstrap standard error describes estimated sampling variation under the chosen resampling scheme. It does not automatically include uncertainty from fitting the model. If the goal is to include training-sample variability, each replicate must reproduce the relevant model-fitting process as well as the validation calculation, with care to preserve the separation between training and validation information. If the fitted models are held fixed and only validation units are resampled, the estimate is conditional on those fitted models.

The bootstrap also relies on the validation units being representative of the population to which the score is intended to apply. It cannot correct an unrepresentative validation sample, immature outcomes, or a mismatched score definition. For rates or other statistics with strong skewness or few independent clusters, inspect the replicate distribution rather than relying on a standard error alone. A percentile interval can communicate asymmetry, but its interpretation remains approximate.

• State whether lower or higher scores are preferred.
• Use the same score definition and weighting in the original calculation and every replicate.
• For model differences on common units, calculate paired replicate differences.
• Resample whole independent clusters whenever records within a cluster are dependent.
• Report the resampling unit, number of replicates, and whether model fitting was repeated.