Validating a Simulation of Delayed Outcomes

Why Validate a Delayed-Outcome Simulation?

A simulation can produce plausible-looking trial results while implementing the wrong time scale, censoring rule, or intermediate-outcome assessment schedule. Validation asks a narrower question than whether the simulated trial is clinically realistic: does the code generate the outcomes implied by its stated assumptions? For delayed outcomes, this means checking each stage of the data-generating process before trusting downstream arm-selection frequencies.

The central strategy is to create benchmarks whose answers are known independently of the full simulation. Some are exact probabilities, such as the chance of observing an event before censoring. Others are limiting cases: for example, if every participant has enough follow-up, all scheduled intermediate outcomes should be mature. Agreement within expected Monte Carlo variation is reassuring, but it does not prove that the model is appropriate for a real trial.

Check Accrual and Outcome Maturity First

Suppose a fixed cohort of \(n\) participants has entry times independently drawn from a uniform distribution on \([0,A]\), where \(A\) is the accrual duration. Then each entry time \(E_i\) has mean \(A/2\), variance \(A^2/12\), and probability \(t/A\) of occurring by time \(t\), for \(0\le t\le A\). Thus, the number enrolled by \(t\) follows a binomial distribution with parameters \(n\) and \(t/A\). These benchmarks apply to this uniform-entry model, not automatically to a Poisson accrual process or to fixed scheduled enrollment dates.

If an intermediate outcome is assessed after a fixed lag \(L\) from enrollment, it is mature by the end of accrual only when \(E_i+L\le A\). For \(0\le L\le A\), the maturity probability is \((A-L)/A\). The maturity count is binomial under independent uniform entry times. At an earlier calendar-time look \(t\), apply the same reasoning to the participants actually enrolled by \(t\): an outcome is mature if \(E_i\le t-L\). Checking the number enrolled and the number mature separately helps locate errors in the accrual clock versus the assessment clock.

Validate Event Times and Censoring Together

A useful analytic check assumes an event time \(T\sim\operatorname{Exp}(\lambda)\) and an independent censoring time \(C\sim\operatorname{Exp}(\mu)\), with positive rates \(\lambda\) and \(\mu\). The observed follow-up is \(U=\min(T,C,\tau)\), where \(\tau\) is an administrative follow-up limit, and the event indicator is \(\delta=1\) if \(T\le C\) and \(T\le\tau\), and 0 otherwise. Independence and the exponential distributions are substantive assumptions of this benchmark; if the simulation uses other distributions or dependent event and censoring times, these formulas do not apply unchanged.

The density that the event occurs at time \(t\) before censoring is \(\lambda e^{-\lambda t}e^{-\mu t}\). Integrating this density from 0 to \(\tau\) gives the probability of an observed event. By symmetry, replacing \(\lambda\) with \(\mu\) gives the probability that random censoring occurs first by \(\tau\). If neither happens by \(\tau\), both times exceed the limit.

These three mutually exclusive categories sum to one. In a usual record with administrative follow-up, however, the observed censoring indicator \(\delta=0\) combines random censoring before the event with participants who reach \(\tau\) without either time occurring. The simulation should be checked against the correct category definition, not just a loosely labelled “censoring rate.”

Check Intermediate Outcomes and Arm Selection

A maturity check should be made before checking the intermediate-outcome values or selection decision. For example, a binary response that is generated only after an assessment lag must not be counted as observed before that lag. Among mature participants, compare response proportions with the model’s specified probabilities, accounting for the mature sample size. When the outcome is a binary variable with response probability \(p\) among \(m\) independently generated mature participants, its response count has mean \(mp\) and standard deviation \(\sqrt{mp(1-p)}\).

Selection frequencies can also have exact benchmarks in small, deliberately simplified cases. Enumerating every possible outcome is valuable because it tests the selection rule, including ties, without relying on a noisy large simulation. The following hypothetical example assumes independent arm outcomes and a rule that selects the arm with the larger number of responses, breaking ties by a fair random choice.

A Practical Validation Sequence

Validation is more informative when the data-generating process is checked in stages rather than only through the final selection result. A mismatch in selection can originate in accrual, outcome maturity, event generation, or the selection code itself. The sequence below localizes the source.

Diagnostics, Alternatives, and Limitations

A standardized discrepancy such as \((\hat p-p)/\sqrt{p(1-p)/n}\) helps describe the size of a difference for an independently simulated binary outcome. It is a diagnostic, not a universal pass-or-fail rule. A discrepancy of a few standard errors can occur by chance, especially when many checks are performed; a very large discrepancy, such as the one in the event example, signals a likely implementation or specification problem. For counts or continuous summaries, use the appropriate benchmark standard error instead.

Plots add information that a single average can hide. Compare an empirical event-time distribution with its theoretical cumulative distribution, and inspect accrual and maturity counts by time rather than only at one endpoint. A time-unit error may preserve a rank ordering while distorting the distribution. Also compare a small hand-calculated case with code output; this is often more decisive than repeating a large simulation.

Analytic benchmarks are generally the clearest checks when available, while an independently written reference implementation can check more complicated logic. The latter is not a mathematical guarantee: two implementations may share the same mistaken assumption or coding convention. Limiting cases are useful when exact formulas are unavailable, such as checking that the event proportion approaches zero as \(\lambda\) approaches zero, or that all outcomes are mature when every entry precedes the look by at least \(L\). These checks should complement, not replace, direct benchmarks.

The formulas in this tutorial rely on stated assumptions. Uniform accrual does not represent every enrollment process; independent exponential censoring does not cover informative censoring or arbitrary hazards; and an exact selection calculation for two small arms does not establish the correctness of a more complex rule. Further, a simulation may pass its code checks while still using implausible model inputs. Validation supports confidence that the programmed model matches its specification; it cannot establish that the specification describes the clinical setting well.