Why Uncertainty Needs More Than One Number
A polygenic risk score (PRS) is often presented as a number for each person. That number may be useful for ranking people, but it does not by itself state the probability that a person will develop a disease. Moreover, a reported score or risk estimate can be uncertain for different reasons: the score weights were estimated, performance was measured in a finite validation sample, and calibration may rely on a finite or unrepresentative target-population sample. These are distinct uncertainties and should not be collapsed into one vague statement that “the PRS has uncertainty.”
The earlier material on discrimination, calibration, and recalibration distinguishes ranking from absolute-risk prediction. Here the focus is how to quantify uncertainty around those quantities and communicate what an interval does—and does not—cover. A confidence interval for a performance metric describes uncertainty about a population-level quantity. An interval for an individual’s predicted risk describes uncertainty in the estimated risk model at that person’s covariate values; it is not an interval for whether that person will or will not experience the outcome.
Uncertainty in an Estimated Score
Let \(g_i\) be the vector of allele counts for person \(i\), and let \(\hat{\beta}\) be the vector of estimated per-allele weights on a specified score scale. The estimated score is \(\hat{S}_i=g_i^\top\hat{\beta}\). Conditional on the person’s observed genotype, uncertainty in this score comes from uncertainty in the weights, not from repeatedly observing that person’s fixed genotype. If the weight estimator is approximately multivariate normal with covariance matrix \(\widehat{\operatorname{Cov}}(\hat{\beta})\), then the estimated variance of the score is \(g_i^\top\widehat{\operatorname{Cov}}(\hat{\beta})g_i\).
This calculation requires a covariance estimate appropriate to the weights used in the score, including their dependence. Ignoring covariance is generally unjustified: correlated weight estimates can increase or decrease the score variance. A normal-approximation interval is \(\hat{S}_i\pm1.96SE(\hat{S}_i\mid g_i)\) for an approximate 95% confidence interval. It reflects uncertainty in the estimated weights under the model and sampling assumptions; it does not include uncertainty from model misspecification, a different target population, or a later calibration step unless those sources are included in the calculation.
Worked Example: Weight Uncertainty for a Two-Variant Score
Suppose a hypothetical score uses two allele counts \(g_i=(1,2)^\top\), with estimated weights \(\hat{\beta}=(0.10,0.20)^\top\). The weights are on a log-odds scale. Assume their estimated covariance matrix is \(\begin{pmatrix}0.0004&0.0001\\0.0001&0.0009\end{pmatrix}\), and that the joint normal approximation is reasonable.
The estimated score is \(1(0.10)+2(0.20)=0.50\) log-odds units. Its estimated variance is \(1^2(0.0004)+2^2(0.0009)+2(1)(2)(0.0001)=0.0044\). The last term is the covariance contribution.
Thus \(SE=\sqrt{0.0044}=0.06633\). The approximate 95% interval is \(0.50\pm1.96(0.06633)=0.50\pm0.1300\), or \([0.370,0.630]\) log-odds units. This interval concerns the score implied by the estimated weights for these fixed allele counts. It does not mean that the person’s disease risk is between 0.370 and 0.630; these are score-scale values, not probabilities.
Uncertainty in Validation Performance
A validation estimate is also a sample statistic. For example, an AUC estimated in a validation sample varies across samples even if the scoring algorithm is held fixed. A confidence interval quantifies uncertainty about the corresponding population AUC under a specified sampling scheme. It is not an interval for the AUC of one individual, nor does it indicate uncertainty in the individual’s outcome.
A common approach is to resample validation participants and recompute the performance measure in each resample. The resampling scheme must match the sampling design and estimand. If cases and controls were sampled separately, resampling within outcome groups estimates uncertainty conditional on the observed class counts; it does not capture uncertainty in population prevalence. If participants were sampled from the target population, resampling individuals can reflect variation in both class counts and measured performance, subject to the sample size and design. Repeated measures, family structures, or site clusters require resampling at the independent sampling-unit level rather than treating every record as independent.
Worked Example: A Validation AUC Interval
In a hypothetical validation sample of 200 participants—50 cases and 150 controls—an estimated AUC is 0.720. Suppose a participant-level bootstrap that resamples cases and controls separately gives an estimated standard error of 0.035. Assume the bootstrap distribution is sufficiently symmetric for a normal approximation to be reasonable.
The approximate 95% confidence interval is \(0.720\pm1.96(0.035)\). Since \(1.96(0.035)=0.0686\), the interval is \([0.6514,0.7886]\), or about 0.651 to 0.789.
In context, this interval expresses sampling uncertainty about the AUC for the population of cases and controls represented by this validation design. Because the bootstrap sampled cases and controls separately, it does not estimate uncertainty in disease prevalence. A different resampling design could be needed for a different target estimand. The interval also does not show that 95% of individuals will have correctly ranked outcomes; AUC summarizes pairwise ranking across cases and controls.
The normal approximation is convenient but not mandatory. A percentile or other bootstrap interval may be more appropriate when the bootstrap distribution is skewed or bounded, as AUC is. Whichever interval is reported, state the sample, resampling unit, and whether outcome-group counts were fixed. A narrow interval can still be misleading if the validation sample is not representative of the population to which the results will be applied.
Target-Population Sampling and Absolute Risk
Absolute-risk estimates require a risk model and a defined outcome horizon and population. For a logistic model, let \(x_i\) contain the person’s predictors, including any PRS term, and let \(\theta\) be the fitted coefficient vector. The estimated linear predictor is \(\hat{\eta}_i=x_i^\top\hat{\theta}\), and the estimated risk is \(\hat{r}_i=\operatorname{expit}(\hat{\eta}_i)=1/(1+\exp(-\hat{\eta}_i))\). If the coefficients are estimated with covariance matrix \(\widehat{\operatorname{Cov}}(\hat{\theta})\), an approximate standard error for the linear predictor is \(\sqrt{x_i^\top\widehat{\operatorname{Cov}}(\hat{\theta})x_i}\). This includes coefficient uncertainty under the fitted model, but not necessarily uncertainty from sampling a different target population or choosing a different model.
Because the expit function is increasing, the endpoints of an interval for \(\eta_i\) can be transformed to obtain an interval for the corresponding fitted risk. This produces an asymmetric interval on the probability scale in general. If the risk model was recalibrated using a target sample, uncertainty in the recalibration parameters should be included as well. In addition, a target sample’s estimated prevalence is itself uncertain; a case-control validation sample with deliberately fixed case and control counts does not directly estimate that prevalence.
Worked Example: From a Linear Predictor to an Absolute-Risk Interval
Consider a hypothetical person evaluated with a logistic risk model for a specified disease horizon. Suppose the estimated linear predictor is \(\hat{\eta}=-3.00+0.50(1.20)=-2.40\). Assume the estimated standard error of this linear predictor is 0.15765, incorporating the fitted model’s coefficient covariance for this person’s covariate values. Use a normal approximation and treat the model form and horizon as fixed.
The estimated absolute risk is \(\operatorname{expit}(-2.40)=1/(1+\exp(2.40))\approx0.08317\), or about 8.32%. The approximate 95% interval on the linear-predictor scale is \(-2.40\pm1.96(0.15765)=[-2.709,-2.091]\).
Transforming both endpoints gives \(\operatorname{expit}(-2.709)\approx0.06244\) and \(\operatorname{expit}(-2.091)\approx0.10997\). The approximate risk interval is therefore 6.24% to 11.0% for the specified horizon, conditional on the model and the assumptions used to obtain the standard error. It is not a statement that there is a 95% probability that this person’s fixed, unknown risk lies in this particular interval, and it does not predict whether the person will develop disease.
Worked Example: Sampling Uncertainty in Target Prevalence
Suppose a hypothetical random sample of 200 people from the intended target population contains 8 people with the disease by the specified horizon. The sample proportion is \(\hat{p}=8/200=0.040\), or 4.0%. Because there are only 8 observed cases, a simple symmetric normal interval can be unreliable. Use the Wilson interval with \(z=1.96\), assuming independent representative sampling and accurate outcome classification.
The Wilson center is \((\hat{p}+z^2/(2n))/(1+z^2/n) =(0.040+3.8416/400)/(1+3.8416/200)\approx0.04867\). Its half-width is \(\{z\sqrt{\hat{p}(1-\hat{p})/n+z^2/(4n^2)}\}/(1+z^2/n)\) \(=\{1.96\sqrt{0.040(0.960)/200+3.8416/(4(200)^2)}\}/1.019208 \approx0.02826\).
The resulting interval is approximately \([0.0204,0.0769]\), or 2.04% to 7.69%. This is an interval for target-population prevalence under the stated sampling assumptions, not an individual risk interval. If prevalence informs the baseline risk or calibration intercept, its uncertainty can affect absolute-risk estimates and should be propagated through that calibration rather than ignored.
Combining Uncertainty and Checking Assumptions
Uncertainty components can be dependent. For example, the same participants may contribute to a calibration fit and to a performance estimate, or score weights may have been estimated using data related to the validation sample. Adding separate standard errors as though all components were independent can then give an incorrect result. The variance identity \(\operatorname{Var}(A+B)=\operatorname{Var}(A)+\operatorname{Var}(B)+2\operatorname{Cov}(A,B)\) shows why the covariance term matters.
A useful justification for a full-pipeline bootstrap is to repeat the relevant sampling process and refit every estimated step within each resample: score weights if their estimation is part of the target procedure, calibration, and the reported performance or risk summary. The spread of the resulting estimates reflects the combined sampling variability represented by that resampling design. This approach is not a cure for bias: it cannot make a nonrepresentative sample representative, repair outcome misclassification, or account for model choices that were not repeated or specified.
Specify whether the report concerns a score, a population performance metric, prevalence, or an individual’s model-based absolute risk.
State which estimated weights, validation participants, or target-population observations contribute uncertainty, and which quantities are treated as fixed.
Use an analytic approximation when its distributional assumptions are credible; use a design-appropriate bootstrap when it better captures the fitting process and sampling structure.
Give the interval with its estimand, outcome horizon where relevant, target population, and key assumptions. Explicitly separate relative score from absolute risk.
Diagnostics include checking whether the estimated standard error is plausible across reasonable resampling schemes, whether interval endpoints behave sensibly on bounded scales, and whether results change materially when influential observations or sampling clusters are handled appropriately. For risk intervals, inspect calibration and the target population’s representation; an interval around a precisely estimated but poorly calibrated model can still be inaccurate. Very small samples, few cases, extreme predicted risks, and weakly estimated calibration parameters can make normal approximations particularly fragile.
Common Mistakes and Reporting Choices
| Incorrect reasoning | Correction |
|---|---|
| “The score is 0.50, so the person has a 50% disease risk.” | A score is defined on its own scale. Absolute risk requires a calibrated risk model and a specified population and horizon. |
| “The AUC interval describes uncertainty in this person’s risk.” | An AUC interval concerns a population ranking-performance quantity, not an individual probability. |
| “A case-control sample’s case fraction is the target prevalence.” | When case and control counts were selected by design, that fraction generally does not estimate target-population prevalence. |
| “The interval covers every source of uncertainty.” | State which estimation steps and sampling sources were included; model misspecification and population shift are not automatically covered. |
Analytic intervals are transparent and efficient when their assumptions are credible, but can understate uncertainty if they omit fitted stages or dependencies. Bootstrap intervals can accommodate complicated estimation steps, but only if the resampling unit and scheme reflect the design. Neither method addresses transportability by itself. Reporting should identify the estimand, scale, population, horizon, interval method, and what was held fixed.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- If a person’s genotype is treated as fixed, what source of uncertainty remains in a PRS calculated using estimated weights?
- Why can separately bootstrapping cases and controls fail to quantify uncertainty in target disease prevalence?
- A logit-scale risk estimate is \(-1.5\) with standard error \(0.10\). Calculate an approximate 95% interval on the logit scale and explain how to obtain the risk-scale interval.
- What does a confidence interval for a validation AUC describe, and what does it not describe?
- Name two reasons a bootstrap of the entire analytic pipeline might still give misleading uncertainty.
Solutions. (1) Sampling uncertainty in the estimated weights remains; conditional on the observed genotype, genotype itself is not resampled. (2) The resampling fixes the case and control counts, so it conditions on those counts rather than representing prevalence sampling. (3) The logit interval is \(-1.5\pm1.96(0.10)=[-1.696,-1.304]\); apply the expit function to both endpoints to obtain the probability-scale interval. The point estimate is \(\operatorname{expit}(-1.5)\approx0.1824\). (4) It describes sampling uncertainty about the population AUC under the validation design, not the risk or outcome of any one person. (5) Examples include a nonrepresentative sample, an incorrect resampling unit for clustered data, omitted estimation steps, or model misspecification; resampling does not automatically fix these problems.