Why a Validation Metric Needs an Uncertainty Estimate
A validation statistic is calculated from a particular set of participants. In a small cohort, a few unusual outcomes or influential observations can shift the statistic enough to change which PRS appears strongest. Reporting only the observed metric hides this sampling variability. Reporting an uncertainty interval or a resampling distribution makes clear how much the result depends on the validation sample.
The object of interest must be specified before an interval is calculated. For example, one might ask about the population correlation between a fixed score and a continuous phenotype, or about the difference in a chosen performance metric between two fixed scores in the population represented by the validation cohort. These are not the same estimand. Nor is uncertainty from sampling validation participants the same as uncertainty from fitting or selecting the PRS. Nested validation addresses aspects of tuning and model selection; here the emphasis is precision of performance estimates once the evaluation procedure and metric have been specified.
A confidence interval does not assign a probability to the fixed parameter being inside the particular interval that was calculated. Rather, for a 95% confidence procedure, repeated samples and intervals constructed by that procedure would cover the target parameter about 95% of the time under its assumptions. In a specific validation analysis, the interval communicates the range of parameter values reasonably compatible with the data and procedure.
A Correlation Interval for a Small Cohort
For a fixed PRS and a continuous outcome, the sample correlation \(r\) is a common summary of linear association. Under independent sampling and an approximately bivariate-normal population distribution for score and outcome, Fisher's transformation gives an approximately normal sampling distribution:
An approximate 95% interval on the transformed scale is \(z \pm 1.96SE(z)\). Applying the inverse transformation, \(\tanh(\cdot)\), to both endpoints returns an interval for the population correlation. This approximation is most credible when observations are independent and the joint distribution is not dominated by severe skewness or influential points. It can be imprecise for very small samples or correlations close to the boundaries of -1 and 1.
Worked Example: An Uncertain Correlation in 40 Participants
Suppose a hypothetical validation cohort contains \(n=40\) unrelated participants, and a prespecified PRS has sample correlation \(r=0.30\) with a continuous phenotype. Assume the participants are sampled independently from the target population and the score-outcome distribution is reasonably compatible with the approximation above.
First transform the observed correlation: \(z=\operatorname{atanh}(0.30)=0.3095\). The standard error is \(SE(z)=1/\sqrt{40-3}=1/\sqrt{37}=0.1644\). Thus the transformed 95% interval is
Transforming the endpoints back gives \((\tanh(-0.0127),\tanh(0.6317))\), approximately \((-0.0127,\;0.559)\). The observed correlation is positive, but this interval includes zero and is wide: the sample does not estimate the population correlation precisely. If performance is summarized as the squared correlation in this simple linear-association setting, the observed value is \(r^2=0.30^2=0.090\). Squaring the correlation interval requires care because the interval crosses zero; the corresponding range of squared values is approximately \(0\) to \(0.559^2=0.313\). This is not a general confidence interval formula for every definition of predictive \(R^2\).
Comparing Two Scores: Preserve the Pairing
When two PRSs are evaluated on the same participants, their metric estimates are dependent. A participant with an unusual outcome can affect both scores' metrics. Therefore, uncertainty in their difference should preserve each participant's outcome and both predictions together. Resampling the two score datasets independently would break this pairing and estimate the wrong variability.
A paired nonparametric bootstrap approximates repeated sampling by drawing \(n\) participant records with replacement from the validation cohort. Each selected record carries its outcome and predictions for every score being compared. The metric is recalculated for each score on that same resample, and their difference is recorded. Percentiles of the resulting bootstrap differences can form an approximate interval. This approach does not require a normal formula for the metric, but it still relies on the validation participants being suitable independent sampling units and can perform poorly when the cohort is extremely small or the metric is unstable.
Specify the metric, direction of improvement, participants, and whether predictions are fixed or the whole fitting procedure will be repeated.
Draw records with replacement, keeping outcomes and all compared predictions together. Use the same resampled records for each score.
For each bootstrap sample, calculate \(\Delta^* = M_A^* - M_B^*\), where \(M\) is the prespecified metric and larger values are defined to be better.
Use the bootstrap distribution to report an interval for the contrast and, if useful, the frequency with which each score ranks highest. Interpret both as resampling summaries, not as probabilities that a model is truly best.
Worked Example: A Paired Bootstrap Difference in R-squared
In a hypothetical cohort of 36 participants, two fixed PRSs are evaluated using the same prespecified \(R^2\) metric. Score A has \(R^2=0.124\), and score B has \(R^2=0.106\). The observed difference is
This is an observed advantage of 0.018, or 1.8 percentage points of \(R^2\). In 2,000 hypothetical paired bootstrap resamples, suppose the 2.5th and 97.5th percentiles of the differences are 0.001 and 0.035. The percentile interval is therefore \((0.001,\;0.035)\), equivalent to 0.1 to 3.5 percentage points.
Each resample uses the same selected participant records to calculate both \(R^2\) values before subtracting. Under the independent-participant sampling assumption, the interval suggests that the observed advantage is small and its size is uncertain; the interval is just above zero for this particular bootstrap procedure. It does not establish that A will outperform B in every new cohort. Because the lower endpoint is close to zero and bootstrap intervals can be unreliable in very small samples, the result should not be presented as a decisive ranking.
Why Rankings Can Change Across Resamples
A ranking discards information about the size and uncertainty of metric differences. If several PRSs have nearly equal validation metrics, small changes in the sampled participants can reorder them. Their metric estimates are often correlated because they are evaluated on the same outcomes and may produce similar predictions. A small observed gap can therefore coexist with frequent reversals in resamples.
Ranking stability can be summarized by repeating the paired bootstrap and recording which score has the largest metric in each resample. This is a descriptive stability measure conditional on the observed cohort, the resampling scheme, and the scores being compared. It is not a posterior probability that a score is the population winner, and it does not include all uncertainty from model development, tuning, or transport to a different population.
Worked Example: A Fragile Winner Among Three PRSs
Consider three fixed PRSs evaluated in a hypothetical validation cohort of 48 independent participants. Their observed \(R^2\) values are 0.112 for A, 0.108 for B, and 0.101 for C. A is the observed winner, but its lead over B is only \(0.112-0.108=0.004\), or 0.4 percentage points.
Suppose 2,000 paired bootstrap resamples produce A as the top-ranked score 1,080 times, B 820 times, and C 100 times. The respective observed top-rank frequencies are \(1080/2000=0.540\), \(820/2000=0.410\), and \(100/2000=0.050\). A is ranked first most often, but it is not first in 46.0% of the resamples. The sensible conclusion is that A leads in this cohort while the ranking, especially between A and B, is unstable. The 54.0% frequency is not a 54.0% probability that A is truly the best PRS.
Assumptions, Diagnostics, and Alternatives
The resampling unit must match the source of sampling. The simple participant bootstrap assumes participant records are independent and exchangeable for the target population. If participants are related, clustered by site, or repeatedly measured, a naive participant-level bootstrap may understate uncertainty; resampling should respect the relevant clusters or sampling design. If a metric depends on outcome groups, resampling within outcome strata may be appropriate when the goal is to preserve the observed group composition. That choice changes the target of inference and should be stated.
Also specify whether the PRS predictions are fixed. Bootstrapping fixed predictions estimates variability from drawing new validation participants conditional on those predictions. If the goal is uncertainty for a pipeline that includes fitting or tuning, that pipeline must be repeated within the resampling procedure, with any necessary separation of training and evaluation data preserved. Reusing the evaluation sample to choose a winner and then presenting its unadjusted interval as if the choice had been made in advance can make the reported performance optimistic. Nested validation is relevant when tuning and selection are part of the procedure.
| Approach | Useful when | Main caution |
|---|---|---|
| Analytic interval, such as Fisher's correlation interval | The metric has a standard sampling approximation and its assumptions are credible | Approximation can be poor with very small samples, influential observations, or a mismatched distribution |
| Paired bootstrap interval for a metric difference | The scores are measured on the same participants and a direct contrast is the target | Resampling cannot create information absent from a sparse or unrepresentative cohort |
| Point estimates and rank frequencies | Describing how rankings change under repeated resampling of the observed cohort | Rank frequencies are not probabilities that a model is truly superior |
Useful diagnostics include inspecting the spread of bootstrap estimates, checking whether a small number of participants repeatedly drive extreme values, and examining how often a resample cannot support the metric calculation. For instance, a metric may be undefined in resamples lacking one required outcome category. Frequent undefined resamples are evidence that the data are too sparse for a stable estimate, not a reason to silently discard the problem. Report the cohort size, interval method, resampling unit, number of replicates, and any failed replicates.
A small cohort limits what any interval can deliver. Bootstrap samples reuse the observed records; they do not reveal unobserved population patterns. An interval may be wide, unstable across interval methods, or deceptively narrow when participants are dependent or the target population is poorly represented. No interval corrects bias caused by a mismatched validation population, an unsuitable phenotype, or leakage between model development and evaluation.
Common Pitfall
Wrong reasoning: “Score A has the highest observed metric, and it is ranked first in 54% of bootstrap samples, so there is a 54% probability that A is the best score.”
Correction: The observed metric identifies the winner in this particular cohort. The 54% is the fraction of resamples from that cohort in which A ranked first under the chosen procedure. It describes sensitivity of the ranking to resampling; it is not a probability about which score is best in the population. Report the metric differences and their uncertainty, not only the winner's name.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A correlation estimate has a wide confidence interval that includes zero. What does that indicate about precision, and what does it not prove?
- Two PRSs are evaluated on the same participants. Why should bootstrap resamples keep each participant's outcome and predictions for both scores together?
- In 1,000 paired bootstrap resamples, score B ranks first 620 times. Give the appropriate interpretation and one interpretation to avoid.
- Name one assumption that can make a participant-level bootstrap unsuitable, and describe a corresponding adjustment.
- What additional step is needed if uncertainty from tuning the PRS, rather than only validation-sample variation conditional on fixed predictions, is part of the target?
Solutions. (1) The wide interval indicates that the population correlation is estimated imprecisely under the interval's assumptions. Including zero does not prove that the population correlation is exactly zero. (2) The two metrics are dependent because they share outcomes and participants; resampling records together preserves that pairing and estimates uncertainty in their difference appropriately. (3) B ranks first in 62.0% of resamples from this cohort under the specified procedure. This is not a 62.0% probability that B is truly the best score. (4) Related or clustered participants violate the independent-record assumption; resampling families or clusters, rather than individuals, can respect the sampling unit. (5) Repeat the relevant fitting and tuning steps within a procedure that preserves separation between training and evaluation data; bootstrapping fixed predictions alone estimates only conditional validation-sample variability.