Why Check a Validation Procedure by Simulation?
A validation metric can be computed exactly as specified and still behave poorly in the sample sizes, event patterns, and censoring conditions where it will be used. Simulation provides a controlled check: specify a multi-state data-generating process with known transition probabilities, generate many validation cohorts, apply the proposed procedure to each cohort, and compare its estimates and intervals with known targets.
The key distinction is between the metric’s target and the procedure used to estimate it. For example, the target may be the population Brier score at a fixed horizon, while the procedure is an inverse-probability-of-censoring weighted estimator with a confidence interval. Repeated simulation can reveal bias in the estimator, poor interval coverage, or sensitivity to censoring. It does not establish that the simulated process represents every real validation setting; it tells us how the procedure performs under the specified conditions.
Specify a Multi-State Process and Its Target
A simulation needs a complete rule for generating event histories. One simple choice is a discrete-time, time-homogeneous Markov model. At each interval, the next state is drawn from a row of a transition matrix. Rows must contain nonnegative probabilities summing to one, and an absorbing state has probability one of remaining in that state. More complex designs can allow transition probabilities to depend on time or covariates.
Two useful targets are calibration-in-the-large for state \(r\), \(E\{Y_{ir}(\tau)-p_{ir}(\tau)\}\), and the state-occupancy Brier score, \(E\{[Y_{ir}(\tau)-p_{ir}(\tau)]^2\}\). The first is zero when average predicted occupancy matches average observed occupancy. The second is average squared prediction error, with smaller values indicating better predictions for that target population and horizon. Neither target alone describes every aspect of predictive performance.
For a simple censoring setup, let \(C_i\) be the time at which subject \(i\)'s state history becomes unobserved, and define \(G_i(\tau)=P(C_i\geq\tau\mid X_i)\), where \(X_i\) denotes observed predictors used in the censoring model. If the state at \(\tau\) is observed only for subjects uncensored through \(\tau\), the inverse-probability weighted estimators are
These expressions require a censoring assumption: conditional on the variables in the censoring model, being observed through \(\tau\) must not convey additional information about the endpoint outcome or prediction error. They also require positivity: \(G_i(\tau)>0\) in the target population. In a simulation, censoring times and predictors are generated by explicit rules, so one can check whether the estimator uses the correct conditioning variables and whether weights become extreme. If \(G_i(\tau)\) is known in the data-generating process, it can be used to isolate the metric estimator’s behavior. If it is estimated, the estimation step must be repeated in every replicate.
Worked Example: Deriving a Known Two-Interval Prediction
Consider three states: 0 (initial), 1 (intermediate), and 2 (absorbing). At each interval, use the transition matrix below. The rows represent the current state and columns the next state.
A subject beginning in state 0 has probability \(0.25\) of occupying state 2 after two intervals. To see this, add the probabilities of the three possible state paths ending in 2: \(0\to0\to2\), \(0\to1\to2\), and \(0\to2\to2\). The calculation is
The other two probabilities are \(P(Z(2)=0\mid Z(0)=0)=(0.70)(0.70)=0.49\) and \(P(Z(2)=1\mid Z(0)=0)=(0.70)(0.20)+(0.20)(0.60)=0.26\). They sum to one: \(0.49+0.26+0.25=1.00\). This check should be built into simulation code. A generated path must also follow an allowed transition; for example, a subject cannot leave absorbing state 2 under this model.
Measure Bias and Coverage Across Replicates
Generate \(R\) independent validation cohorts, each with sample size \(n\). In replicate \(b\), compute the estimate \(\widehat{\theta}_b\) of a prespecified target \(\theta\), and construct its nominal \(100(1-\alpha)\%\) confidence interval \([L_b,U_b]\). The Monte Carlo bias estimate and empirical coverage are
The bias estimate is itself noisy. Its Monte Carlo standard error is the sample standard deviation of the replicate estimates divided by \(\sqrt{R}\). If the true coverage is \(c\), the approximate Monte Carlo standard error of the estimated coverage is \(\sqrt{c(1-c)/R}\); in practice, replacing \(c\) with the observed coverage gives an estimate. Thus, a coverage result slightly different from the nominal level may reflect simulation noise, especially when \(R\) is small.
For a Wald interval based on a replicate-specific standard error \(SE_b\), the interval is \(\widehat{\theta}_b\pm z_{1-\alpha/2}SE_b\). In a simulation, check not only whether these intervals cover at the intended rate, but also whether the standard errors estimate the actual replicate-to-replicate variability. A correct point estimator with an underestimated standard error can have poor coverage. When the censoring model is estimated, uncertainty from estimating \(\widehat{G}\) should be included, for example by refitting that model within each bootstrap resample. Treating estimated weights as fixed can make intervals too narrow.
Worked Example: Checking Bias and Coverage for a Brier Score
Suppose 60% of the target population starts in state 0 and 40% in state 1. Use the transition matrix in the previous example and predict occupancy of absorbing state 2 after two intervals. The true risks are \(0.25\) for the first group and \(0.64\) for the second: from state 1, the probability is \((0.60)(0.40)+(0.40)(1)=0.64\). The predictions equal these true risks.
For a binary outcome with true probability \(q\) and prediction \(p=q\), the expected squared error is \(q(1-q)\): the outcome is one with probability \(q\), giving squared error \((1-q)^2\), and zero with probability \(1-q\), giving squared error \(q^2\). Therefore the population Brier target is
Now suppose a hypothetical simulation uses \(R=2{,}000\) cohorts of \(n=300\), with independent censoring through the horizon for 70% of subjects. Using the known censoring probabilities, assume the mean estimated Brier score across replicates is \(0.2051\), and 1,896 of the 2,000 nominal 95% intervals contain \(0.20466\). The estimated bias is \(0.2051-0.20466=0.00044\). The empirical coverage is \(1{,}896/2{,}000=0.948\), or 94.8%. Its approximate Monte Carlo standard error is \(\sqrt{0.948(0.052)/2{,}000}=0.00496\), about 0.50 percentage points. These hypothetical results are consistent with little detectable bias and coverage close to 95%; they do not prove the procedure is unbiased in every setting.
Check What Censoring Does to the Metric
A direct comparison of complete cases with the full target population is often misleading when censoring depends on predictors, state history, or other factors related to prediction error. The following calculation illustrates how a seemingly modest change in case mix can alter a metric. Suppose two equally sized groups have true state-2 risks \(0.20\) and \(0.80\), predictions \(0.10\) and \(0.60\), and probabilities of remaining observed through the horizon \(0.90\) and \(0.50\), respectively. Assume censoring is independent of the endpoint within each group.
Worked Example: Complete-Case and Weighted Brier Scores
For the first group, the expected Brier score is \(0.20(1-0.10)^2+0.80(0.10)^2=0.17\). For the second, it is \(0.80(1-0.60)^2+0.20(0.60)^2=0.20\). Because the groups are equally common, the target is \((0.17+0.20)/2=0.185\).
Among complete cases, the expected group counts are proportional to \(0.90\) and \(0.50\). The expected complete-case Brier score is therefore
It differs from the target because the less frequently observed group receives less weight. Inverse-probability weighting restores the equal target-group contributions in expectation: each group’s observed contribution is multiplied by \(1/0.90\) or \(1/0.50\), respectively. The weighted target is \(\{(0.17)+(0.20)\}/2=0.185\). This correction is valid only under the stated within-group independent-censoring assumption and positive observation probabilities.
Design, Diagnostics, and Alternatives
A useful simulation varies conditions that can change performance, rather than reporting one favorable scenario. At minimum, consider several sample sizes, censoring rates, transition frequencies, and degrees of censoring heterogeneity. Include settings with few observations in a transition risk set, since large-sample approximations may be unreliable there. If the intended procedure estimates the censoring distribution, simulate that estimation as well. If predictions are obtained by fitting a model, reproduce the full development and validation workflow when that is part of the target procedure; otherwise, hold predictions fixed to isolate the validation estimator.
Before interpreting results, check the simulation itself. Transition probabilities must be valid and sum to one; state paths must obey the allowed transitions; censoring must occur according to the specified rule; and the target must use the same population and horizon as the analysis. Then inspect estimates, standard errors, interval widths, bias, and coverage together. Large weights, frequent empty transition groups, unstable estimates, or coverage that changes sharply across nearby scenarios are warnings, not details to hide in an average.
| Approach | Useful feature | Main limitation |
|---|---|---|
| Complete-case metric | Simple to calculate and may be reasonable when observation through the horizon is independent of prediction error without conditioning. | Can target the observed case mix rather than the intended population under informative censoring. |
| Inverse-probability weighting | Can recover a population target under conditional independent censoring and positivity. | Estimated or small censoring probabilities produce unstable weights and require appropriate uncertainty accounting. |
| Model-based outcome adjustment | Uses a model for the endpoint and can be efficient when that model is adequate. | Results depend on outcome-model specification; it is not automatically protected against misspecification. |
| Simulation-based assessment | Directly checks finite-sample bias and coverage under specified event and censoring patterns. | Conclusions are limited to the data-generating scenarios that were simulated. |
The purpose is not to declare one censoring adjustment universally best. Simulation can expose weaknesses, compare reasonable procedures under controlled conditions, and show which assumptions matter. It cannot compensate for a misspecified data-generating process, and a method performing well under independent censoring may fail if censoring depends on unobserved prognosis.
Common Mistakes
- Calling a near-zero simulated bias proof of validity. The result is conditional on the chosen transition, prediction, censoring, and sample-size scenarios. Add plausible alternatives and report Monte Carlo uncertainty.
- Treating 95% coverage as a result from one cohort. Coverage is a repeated-sampling proportion across simulated cohorts; one interval either contains the target or does not.
- Using complete cases without checking the target. Complete cases may represent a different covariate and risk distribution. State the target population and justify the censoring condition before interpreting the result.
- Ignoring the censoring model’s estimation. If the weights are estimated, replicate that estimation and account for its uncertainty rather than treating the fitted weights as known.
- Reporting only the mean metric estimate. Bias, interval coverage, Monte Carlo standard errors, and instability under sparse transitions provide different information and should be examined together.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- In the three-state model above, why must the row probabilities of a transition matrix sum to one?
- What does an empirical 92% coverage rate mean when the procedure is intended to produce 95% intervals?
- Why can complete-case Brier scores differ from the target-population Brier score under covariate-dependent censoring?
- Name two conditions needed for inverse-probability-of-censoring weighting to identify the target metric.
- With \(R=1{,}000\) replicates and observed coverage \(0.95\), approximately what is the Monte Carlo standard error of coverage?
Solutions. (1) Each row specifies all possible next states conditional on the current state, so their probabilities must total one. (2) In the simulated setting, 92% of repeated intervals contained the fixed target; the result is below nominal coverage, though its simulation uncertainty should be considered. (3) Censoring can change the proportions of risk groups represented among observed subjects, changing the average loss. (4) Conditional independent censoring given the variables in the censoring model, and positivity of remaining observed through the horizon. (5) \(\sqrt{0.95(0.05)/1{,}000}=0.00689\), or about 0.69 percentage points.