Uncertainty In Standardized Risks

Why Standardized Risks Have Two Sources of Uncertainty

A standardized cumulative incidence averages predicted risks across a target population rather than predicting for one covariate profile. Its estimate can therefore vary for two reasons: the fitted competing-risks model would change in another model-fitting sample, and a different sample from the target population would contain a different mix of covariate profiles. An interval that accounts for only one source can understate uncertainty for a population-level prediction.

Let $F_k(t\mid x;\theta)$ denote the fitted-model form of the cumulative incidence for cause $k$ by horizon $t$, at covariate profile $x$, with model parameter vector $\theta$. This notation includes the fitted quantities needed for prediction, such as regression parameters and baseline-hazard quantities. For a target covariate distribution $G$, the standardized risk is the average of those profile-specific risks over $G$.

Definition: The standardized cumulative incidence for cause $k$, horizon $t$, and target distribution $G$ is $$ \psi_k(t;G,\theta)=\int F_k(t\mid x;\theta)\,dG(x). $$ For a finite target population with profiles $x_1,\ldots,x_m$, it is the average $m^{-1}\sum_{i=1}^{m}F_k(t\mid x_i;\theta)$. An estimate replaces the model parameter and, when the target distribution is sampled, the target distribution by their estimates.

This tutorial concerns uncertainty in that average. The target is assumed to be clearly specified: for example, a finite cohort whose profiles are known, or a larger population represented by a random sample of profiles. These are different inferential settings. If the target is the observed finite cohort itself, its profiles are fixed by definition and there is no target-sampling component. If the target is a wider population represented by sampled individuals, the empirical distribution of their profiles is also uncertain.

Variance When the Model and Target Samples Are Independent

Suppose the model is fitted using a sample of size $n_{\mathrm{fit}}$, producing $\hat\theta$, and the target distribution is represented by an independent sample of size $m$, with profiles $X_1,\ldots,X_m$. The plug-in standardized estimate is

\hat\psi_k(t)=\frac{1}{m}\sum_{i=1}^{m}F_k(t\mid X_i;\hat\theta).

The first uncertainty component is the variation in $\hat\psi_k(t)$ as the fitted model changes. The second is the variation in the average predicted risk because the target sample changes. A first-order approximation makes this separation explicit. Write $a$ for the gradient of the population-standardized risk with respect to the model parameter, evaluated at the true parameter:

a=\frac{\partial}{\partial\theta}\psi_k(t;G,\theta).

If the model estimator has covariance matrix approximately $V_\theta$, the model-fitting contribution is $a^\mathsf{T}V_\theta a$. The target-sampling contribution is the variance of the profile-specific predicted risk divided by the target sample size. When the two samples are independent, their first-order covariance is zero, so the contributions add.

Result: Under regularity conditions for the model estimator, a sufficiently smooth risk function, independent model-fitting and target samples, and finite variances, $$ \operatorname{Var}(\hat\psi_k(t))\ \approx\ a^\mathsf{T}V_\theta a+\frac{\operatorname{Var}_G\{F_k(t\mid X;\theta)\}}{m}. $$ In practice, estimate the first term using the fitted-model covariance and a gradient or model bootstrap; estimate the second using the sample variance of the predicted risks divided by $m$.

This is a first-order approximation, not an exact finite-sample identity. It requires that the model estimator behaves regularly enough for a local linear approximation to be useful, and that the target sample represents the distribution to which the standardized result is intended to apply. The model covariance must include the fitted quantities that affect the predicted cumulative incidence. Omitting uncertainty in a fitted baseline component, for example, does not become harmless merely because the predictions are averaged.

Justification: Linearizing the Standardized Estimate

The variance decomposition follows by expanding the estimate around the true model parameter and target distribution. Define $q(X;\theta)=F_k(t\mid X;\theta)$, and let $\psi=E_G\{q(X;\theta)\}$. Add and subtract the average risk evaluated at the true parameter:

\hat\psi-\psi = \left[\frac{1}{m}\sum_{i=1}^{m}q(X_i;\hat\theta) -\frac{1}{m}\sum_{i=1}^{m}q(X_i;\theta)\right] + \left[\frac{1}{m}\sum_{i=1}^{m}q(X_i;\theta)-E_G\{q(X;\theta)\}\right].

The second bracket is a sample average of centered target-population risks. Its variance is $\operatorname{Var}_G\{q(X;\theta)\}/m$. For the first bracket, a first-order Taylor expansion in $\theta$ gives approximately $\bar g^\mathsf{T}(\hat\theta-\theta)$, where $\bar g=m^{-1}\sum_i\partial q(X_i;\theta)/\partial\theta$. As $m$ grows, $\bar g$ estimates $a$. Thus the model component has approximate variance $a^\mathsf{T}V_\theta a$. Independence of the two samples makes the covariance between these first-order components zero, yielding the stated sum.

When the model-fitting data and target sample overlap, this argument does not justify simply adding the two estimated variances: the components can be correlated. A covariance term is then needed, and the full sampling design should guide the calculation. In many applications the most direct approach is to resample the relevant observational units and repeat the entire fitting-and-standardizing procedure.

Worked Example: Combining Two Estimated Variance Components

Suppose a hypothetical fitted model gives predicted cause-specific cumulative incidences of 0.08, 0.12, 0.20, and 0.10 for four independently sampled target profiles. The estimated model-fitting variance of the standardized risk, obtained from the fitted-model uncertainty calculation, is 0.000400. Calculate the combined standard error and a 95% Wald interval.

The standardized estimate is the mean of the four predictions: $$ \hat\psi=\frac{0.08+0.12+0.20+0.10}{4}=0.125. $$

The sample variance of the four profile-specific predictions is $$ s_q^2=\frac{(0.08-0.125)^2+(0.12-0.125)^2+(0.20-0.125)^2+(0.10-0.125)^2}{4-1} =\frac{0.008300}{3}=0.002767. $$ Thus the estimated target-sampling variance of their mean is $s_q^2/m=0.002767/4=0.0006917$. Since the model-fitting and target samples are independent, add the components: $$ \widehat{\operatorname{Var}}(\hat\psi)=0.000400+0.0006917=0.001092, \qquad SE(\hat\psi)=\sqrt{0.001092}=0.03304. $$

The 95% Wald interval is $0.125\pm1.96(0.03304)$, or approximately $(0.0602,0.1898)$. Under the stated sampling and model assumptions, this interval describes uncertainty in the standardized risk for the target population represented by the sample. It is not an interval for the risk of each individual profile.

Delta-Method Calculation for Model-Fitting Uncertainty

When the fitted model has a parameter covariance estimate, the delta method translates that uncertainty to the standardized risk. Let $\hat a$ be the gradient of the standardized risk with respect to $\theta$, evaluated at the fitted model and target distribution. Then the model component is estimated by $\hat a^\mathsf{T}\widehat V_\theta\hat a$. The target component can be estimated from the variability of the fitted risks across target profiles. This method is convenient when the risk is differentiable and the model covariance estimate is reliable.

The gradient must correspond to the actual standardized quantity: average the profile-specific derivatives over the target distribution. It is generally not correct to calculate uncertainty for one “average profile” and treat that as uncertainty for the average predicted risk. Standardization averages predictions, and its derivative is the corresponding average of derivatives.

Worked Example: A Two-Parameter Delta-Method Calculation

Consider a hypothetical model with parameter vector $\theta=(\theta_1,\theta_2)^\mathsf{T}$. At the fitted values, the gradient of the target-standardized risk is $\hat a=(0.20,0.10)^\mathsf{T}$, and the estimated parameter covariance matrix is $$ \widehat V_\theta= \begin{pmatrix} 0.040 & 0.006\\ 0.006 & 0.010 \end{pmatrix}. $$ The target sample contains $m=100$ profiles; the sample variance of their fitted risks is $0.0016$. The estimated standardized risk is 0.220. Find the combined standard error and Wald interval.

First calculate the model component, retaining the covariance between the two fitted parameters: $$ \hat a^\mathsf{T}\widehat V_\theta\hat a =(0.20)^2(0.040)+2(0.20)(0.10)(0.006)+(0.10)^2(0.010) =0.001600+0.000240+0.000100=0.001940. $$ The estimated target-sampling component is $0.0016/100=0.000016$. For independent samples, the total estimated variance is $0.001940+0.000016=0.001956$, giving $SE=\sqrt{0.001956}=0.04423$.

The 95% Wald interval is $0.220\pm1.96(0.04423)$, which is approximately $(0.1333,0.3067)$. The parameter covariance term matters: dropping it would give a different model component. The result also shows that, in this example, model-fitting uncertainty is much larger than target-sampling uncertainty; the relative sizes should be checked rather than assumed.

Bootstrap the Sampling Process That Matches the Target

A bootstrap can account for uncertainty in both the model and the target profiles, including nonlinearities that make a first-order approximation less persuasive. Each replicate should reproduce the relevant sampling process, refit the model, and recompute the standardized risk. If the model-fitting sample and target sample are independent, resample each independently within a replicate. If they overlap because one cohort supplies both the fitted model and the target profiles, resample cohort units according to the design, refit the model, and standardize to the resampled target.

The bootstrap standard error is the sample standard deviation of the replicate standardized risks. A percentile interval uses quantiles of the replicate estimates; a normal-approximation interval uses the bootstrap standard error with an appropriate critical value. These approaches rely on the bootstrap being a reasonable approximation to the sampling distribution. Few replicates can make quantile estimates unstable, and a bootstrap cannot repair a misspecified model or an unrepresentative target sample.

Worked Example: A Full-Pipeline Bootstrap Summary

In a hypothetical analysis, the model-fitting sample and target sample are independent. In each bootstrap replicate, the analyst resamples both samples separately, refits the competing-risks model, and recomputes the target-standardized risk. Eight illustrative replicate estimates are 0.17, 0.19, 0.20, 0.21, 0.22, 0.23, 0.24, and 0.20. Calculate their bootstrap standard error and a normal-approximation 95% interval.

The mean of these illustrative replicates is $$ \bar\psi^*=\frac{0.17+0.19+0.20+0.21+0.22+0.23+0.24+0.20}{8} =\frac{1.66}{8}=0.2075. $$ Their squared deviations from 0.2075 sum to 0.003550. Therefore the sample variance is $0.003550/(8-1)=0.0005071$, and the bootstrap standard error is $\sqrt{0.0005071}=0.02252$.

Using the original estimate 0.2075 for this illustration, the normal-approximation interval is $0.2075\pm1.96(0.02252)$, or approximately $(0.1634,0.2516)$. Eight replicates are far too few for a dependable applied interval; the calculation only demonstrates how the bootstrap dispersion is summarized. In an analysis, use enough replicates for stable uncertainty estimates and inspect the replicate distribution, not just its standard deviation.

Diagnostics, Alternatives, and Limits

The delta method and bootstrap answer the same uncertainty question through different approximations. The delta method is efficient and transparent when the model is smooth, its covariance estimate is trustworthy, and the gradient is available. A full-pipeline bootstrap is more flexible and can reflect the joint fitting-and-standardizing operation, but it may be computationally demanding and can fail when fits are unstable or some resamples contain too little information. A model-only bootstrap, with target profiles held fixed, estimates model-fitting uncertainty but deliberately excludes target-sampling variation.

Before reporting an interval, check that the uncertainty calculation matches the estimand and data design:

Fixed or sampled target? A known finite target cohort does not acquire sampling variation just because it has many profiles. A sampled target intended to represent a larger population does.
Independent or overlapping data? Separate variance addition is justified by independence. Shared individuals can induce covariance; resample the shared units and repeat the complete analysis, or otherwise account for the dependence.
Did each replicate complete the analysis? Record failed fits, extreme estimates, or changing model specifications. Silently discarding failed replicates can distort the uncertainty distribution.
Is the approximation plausible? Strong skewness, sparse causes, influential profiles, or a small effective sample can undermine a symmetric Wald interval or a routine bootstrap approximation.
Does the interval suit the scale? A Wald interval on the risk scale can extend outside the probability range. That signals a poor approximation for this interval; consider a suitable transformed-scale or bootstrap interval and report how it was constructed.

These calculations quantify sampling uncertainty conditional on the specified model and data collection process. They do not measure uncertainty from choosing among different model structures, unmeasured covariates, misspecification, or a target population that differs from the one represented by the data. Nor should the spread of profile-specific predictions be mistaken for the standard error of their average: it contributes to the latter through division by the target sample size, not as the full uncertainty by itself.

Common pitfall: “The model was fitted precisely, so the standardized risk has little uncertainty.” This ignores the possibility that a different target sample would have a different covariate mix. For a sampled target, include target-sampling variation as well as model-fitting uncertainty; for overlapping samples, account for their dependence rather than adding independent-sample components mechanically.

Check Your Understanding

Use the definitions and assumptions above to answer each question. Solutions follow the questions.

A finite target cohort has known profiles, and a model is fitted in an independent sample. Which uncertainty component is absent when the estimand is the average risk in that finite cohort?
For independent model-fitting and target samples, suppose the estimated model variance contribution is 0.0009 and the target contribution is 0.0004. What is the combined standard error?
Why is it generally incorrect to add model and target variance estimates as if independent when the same cohort is used for fitting and standardization?
A model-only bootstrap holds the target profiles fixed while resampling and refitting the model. Which component of uncertainty does it omit?

Solutions. (1) Target-population sampling variation is absent because the finite cohort’s profiles are fixed for this estimand; model-fitting uncertainty remains. (2) The total variance is $0.0009+0.0004=0.0013$, so the combined standard error is $\sqrt{0.0013}=0.03606$. (3) The fitted model and empirical target distribution may be statistically dependent, so their contributions can have a nonzero covariance; a full-design calculation or bootstrap should retain that dependence. (4) It omits variation from drawing a different target sample, while retaining model-fitting variation.