Simulation Checks for Spending Designs

Why Check a Calibrated Design by Simulation?

Group sequential boundaries are calibrated so that, under the null hypothesis, the probability of first crossing at each look matches the intended alpha spending. Numerical calibration can be accurate and still be implemented incorrectly in software: a simulation check provides an independent way to examine the resulting boundary vector and the code that applies it. It is a check, not a replacement for calibration or a guarantee that the study will have its planned operating characteristics under every possible data-generating process.

This tutorial focuses on a null simulation for an efficacy design. It checks whether simulated first-crossing probabilities agree with the theoretical targets and whether their cumulative total agrees with the intended type I error. The simulation must reproduce the design’s information times, null distribution, boundary rule, and stopping rule. A mismatch in any of these can make an apparently precise simulation answer the wrong question.

Model, Targets, and Simulation Events

Consider \(K\) planned looks at increasing information times \(0<t_1<\cdots<t_K=1\), with an upper efficacy boundary \(b_j\) at look \(j\). Under the canonical large-sample null model, the standardized test statistics have mean zero, variance one, and correlation \(\operatorname{Corr}(Z_i,Z_j)=\sqrt{t_i/t_j}\) for \(i<j\). Equivalently, the underlying information-scaled process has independent increments. This model presumes the information times and analysis rule in the simulation are the ones used to calibrate the design. A one-sided upper-boundary rule stops for efficacy at the first look with \(Z_j\ge b_j\); no futility rule is included here.

The increment \(\delta_j\) is an unconditional probability under the null, not the probability of crossing at look \(j\) given survival to that look. The first-crossing events are mutually exclusive: one simulated trial can contribute to at most one look-specific event. A path that has crossed stops contributing at later looks, even if the simulated statistic would subsequently fall below a boundary.

One convenient simulation generates independent standard normal increments. With \(Z_1=\varepsilon_1\), where \(\varepsilon_1\sim N(0,1)\), generate for \(j\ge2\)

Each \(Z_j\) is standard normal under \(H_0\), and the recursion gives the canonical correlation structure. In \(N\) independent simulated paths, define \(I_{rj}=1\) if path \(r\) first crosses at look \(j\), and \(I_{rj}=0\) otherwise. The estimated first-crossing probability is \(\hat{\pi}_j=N^{-1}\sum_{r=1}^N I_{rj}\). Its Monte Carlo standard error is estimated by \(\sqrt{\hat{\pi}_j(1-\hat{\pi}_j)/N}\). This quantifies simulation noise in the estimated probability; it does not measure uncertainty in the design’s clinical assumptions or in the theoretical model.

Worked Example: Checking a Three-Look Design

Why the Monte Carlo Standard Error Applies

For a fixed look \(j\), each simulated path either belongs to \(E_j\) or does not. Under independent simulation, \(I_{rj}\) is a Bernoulli random variable with probability \(\pi_j=P(E_j\mid H_0)\). Therefore \(E(\hat{\pi}_j)=\pi_j\), and

Replacing the unknown \(\pi_j\) by \(\hat{\pi}_j\) gives the estimated standard error used above. The same argument applies to a cumulative event, such as “crossed by look \(j\),” by using one indicator per path. In contrast, look-specific first-crossing indicators from the same path are not independent: they are mutually exclusive and their estimates are negatively related. Do not treat the separate look-specific estimates as independent when interpreting their joint pattern.

Worked Example: Interpreting a Small First-Look Difference

Worked Example: Checking the Dependence Between Looks

Designing a Useful Simulation Check

The number of paths should be chosen in light of the probability being checked and the precision needed. For a target probability near \(p\), the approximate standard error is \(\sqrt{p(1-p)/N}\). For example, to target an estimated standard error of at most \(0.0005\) for an overall crossing probability near \(0.05\), use

This is a planning approximation; the realized standard error can be reported using the simulated probability. Very small incremental targets may need many more paths for a comparably precise estimate. If no paths cross at a look, the plug-in standard error is zero, but that does not establish that the true probability is zero. Increase \(N\), or use an appropriate interval for a binomial probability, rather than interpreting zero observed crossings as certainty.

For a sound check, use the same boundary direction, exact stopping rule, information times, and null model as in calibration. Simulate independent trial paths, record the first crossing only, and retain the no-crossing category as well. The category counts, including no crossing, should sum to \(N\). Report both incremental and cumulative results: incremental checks can reveal that error is allocated to the wrong look even when the final total happens to be close to \(\alpha\).

Theoretical numerical integration and simulation answer related but distinct checking needs. Numerical integration evaluates crossing probabilities under the specified joint distribution without Monte Carlo sampling error, subject to integration accuracy. Simulation is flexible and can independently test the implementation of a path generator and stopping rule, but its estimates have Monte Carlo error and can miss small discrepancies if \(N\) is too low. Agreement between the two is reassuring; disagreement calls for checking the model inputs, event definitions, calibration tolerance, and simulation code before drawing a conclusion.

Common Mistakes and Limitations

• Comparing the wrong quantities. Compare first-crossing probability at look \(j\) with the incremental target \(\delta_j\), and compare probability of crossing by look \(j\) with cumulative spending \(A(t_j)\). Do not compare both quantities with the same target.
• Simulating independent test statistics at different looks. The statistics are correlated under the canonical model. Use independent increments to generate the path, not independent \(Z_j\) values.
• Ignoring Monte Carlo uncertainty. A small difference from target is expected in a finite run. Report standard errors or intervals and assess the difference against the simulation precision.
• Counting repeated crossings. The design stops at the first crossing. Counting a path again at later looks changes the estimand and can inflate the apparent total.
• Using the wrong null model or analysis schedule. A check under a canonical fixed-information model does not automatically verify a design with different information-time behavior, a different test statistic, or additional stopping rules.

Simulation is therefore a diagnostic, not a universal proof of error control. It checks the operating characteristics of the simulated design under the chosen null model; it cannot establish that the model is appropriate, that a clinically relevant alternative has adequate power, or that unmodeled operational features have no effect.