Why Simulate Both Interpretations?
A futility boundary identifies paths on which continuing may be unlikely to lead to the desired result. Whether crossing that boundary changes the efficacy calculation depends on how the rule is interpreted. Under a binding interpretation, a futility crossing ends monitoring, so a later efficacy crossing cannot count. Under a nonbinding interpretation, the lower boundary does not remove a path from the efficacy calculation: a later upper-boundary crossing still counts. The introductory material on nonbinding futility and type I error protection establishes why this distinction matters; simulation makes its operating consequences concrete.
A simulation is useful for checking a proposed set of boundaries, comparing the two interpretations, and showing the consequences in a form that can be communicated to a design team. It is not enough to simulate unrelated test statistics at each look. The statistics have a joint distribution, and the simulated paths must preserve it.
Model, Boundaries, and Outcomes
Consider a one-sided group-sequential design with \(K\) analyses at information levels \(I_1 < \cdots < I_K\). Write \(t_i=I_i/I_K\), so that \(0<t_1<\cdots<t_K=1\). Let \(Z_i\) be the standardized test statistic at look \(i\), and let \(\theta\) be the standardized effect per square root of final information. Under the canonical joint model,
Thus, under \(H_0\), \(\theta=0\); under a specified alternative, such as \(\theta=\theta_A>0\), the interim statistics have positive means. The covariance structure is the same under either value of \(\theta\). This model assumes the planned information fractions are correct, the standardized statistics follow the canonical joint normal model, and the boundaries are fixed before the simulated paths are evaluated. If the analysis uses a different joint model, that model—not this canonical one—must be simulated.
Assume \(l_i<u_i\), so the lower and upper crossing regions at a given look do not overlap. There is no lower futility boundary at the final analysis in this setup. Efficacy takes precedence as an outcome: once an upper boundary has been crossed, a later lower-boundary crossing is irrelevant to whether the path rejected \(H_0\). This convention makes the reported futility rate the probability of reaching futility before efficacy.
For either interpretation, estimate type I error by the efficacy-crossing probability under \(H_0\), and power by that probability under the chosen alternative. Report the futility-trigger probability separately. Under binding futility, this is also the probability of an actual futility stop. Under a nonbinding interpretation, it is a hypothetical stop rate—the proportion of paths that would have stopped if the lower rule had been binding. If the nonbinding rule is truly not enforced, its actual stop rate due to that rule is zero. Keeping these quantities distinct avoids implying that a nonbinding boundary terminates monitoring.
Generating Valid Paths
A convenient simulation uses independent normal increments. Let \(W(t)\) have independent increments with \(W(t_i)-W(t_{i-1})\sim N(0,t_i-t_{i-1})\), where \(t_0=0\) and \(W(0)=0\). Generate each path by
The mean is \(\theta\sqrt{t_i}\). For \(i\leq j\), the covariance of the Brownian-motion terms after standardization is \(t_i/(\sqrt{t_i}\sqrt{t_j})=\sqrt{t_i/t_j}\), giving the required canonical model. Each simulated replicate must use a fresh set of independent increments; values within the same replicate must retain their dependence.
Record \(t_i\), \(u_i\), and \(l_i\), and select \(\theta=0\) for type I error and a stated \(\theta_A\) for power. Confirm the boundaries and information fractions are the ones intended for evaluation.
For each replicate, simulate the independent normal increments and calculate every \(Z_i\). Use the same generated paths to evaluate both interpretations.
For binding futility, stop at the first futility trigger before efficacy. For nonbinding futility, continue to check all later efficacy boundaries. Record efficacy, first futility trigger, and any later efficacy after that trigger.
Divide each event count by the number of simulated paths. Report Monte Carlo standard errors and inspect whether the two efficacy outcomes differ only on paths that crossed futility first and efficacy later.
Worked Examples
Worked Example: A Path With Futility Followed by Efficacy
Suppose a hypothetical three-look design has \(t=(0.4,0.7,1)\), upper boundaries \(u=(2.8,2.4,2.0)\), and interim lower boundaries \(l=(-0.5,0.0)\). Consider one simulated path \(Z_1=-0.8\), \(Z_2=0.1\), and \(Z_3=2.2\). Assume the path was generated under the canonical model; the realized values are used to illustrate classification, not as evidence about an actual trial.
At look 1, \(-0.8\leq-0.5\), so the path triggers futility before any efficacy crossing. Under the binding interpretation, monitoring stops, and this path is a futility stop, not an efficacy success. Under the nonbinding interpretation, monitoring continues. At look 2, \(0.1\) is between \(l_2=0.0\) and \(u_2=2.4\), so neither boundary is crossed. At look 3, \(2.2\geq2.0\), so the path crosses the final efficacy boundary. It counts as efficacy under the nonbinding interpretation, but not under the binding interpretation.
This is the key discordant outcome in a paired simulation: futility occurs first and a later efficacy crossing follows. It contributes to nonbinding efficacy but not binding efficacy.
Worked Example: Estimating Error, Power, and Futility Rates
Suppose a hypothetical simulation uses \(M=10{,}000\) paths under each scenario and applies the same boundaries to both interpretations. Under \(H_0\), 472 paths cross an efficacy boundary under binding rules and 516 do so under nonbinding rules. In 2,130 paths, futility is reached before any efficacy crossing; 44 of those paths later cross an efficacy boundary. Under \(\theta_A\), the corresponding efficacy counts are 6,810 and 7,240, with 2,640 futility-first paths, of which 430 later cross efficacy.
Under \(H_0\), the estimated binding type I error is
The nonbinding estimate is \(516/10{,}000=0.0516\). The difference is \(0.0516-0.0472=0.0044\), which equals \(44/10{,}000\): precisely the estimated proportion of paths rescued by a later efficacy crossing after futility. Under the alternative, estimated power is \(6{,}810/10{,}000=0.6810\) with binding futility and \(7{,}240/10{,}000=0.7240\) with nonbinding futility. The power difference, \(0.0430\), likewise equals \(430/10{,}000\).
The futility-trigger rate under \(H_0\) is \(2{,}130/10{,}000=0.2130\), and under \(\theta_A\) it is \(2{,}640/10{,}000=0.2640\). These are actual futility-stop rates under binding rules. Under nonbinding rules they are hypothetical stop rates; the rule itself does not stop those paths. The 44 and 430 later efficacy crossings illustrate why a trigger-rate estimate alone does not describe how often nonbinding paths would later succeed.
Worked Example: Quantifying Monte Carlo Error
Use the hypothetical null results above. For an estimated probability \(\hat{p}\) based on \(M\) independent simulated paths, the Monte Carlo standard error is estimated by \(\sqrt{\hat{p}(1-\hat{p})/M}\). For binding type I error this gives
For nonbinding type I error, the estimate is \(\sqrt{0.0516(1-0.0516)/10{,}000}\approx0.00221\). These standard errors describe simulation noise, not uncertainty about the treatment effect or about whether the model assumptions are appropriate.
Because both interpretations were evaluated on the same paths, their estimates are paired. Here nonbinding efficacy includes all binding efficacy paths plus the 44 later crossings, so the estimated difference is the mean of a binary indicator with probability \(0.0044\). Its Monte Carlo standard error is approximately \(\sqrt{0.0044(1-0.0044)/10{,}000}=0.000662\). Treating the two estimates as if they came from independent simulations would discard this pairing and give an inappropriate precision calculation for their difference.
Justification and Diagnostics
For any fixed simulated path, binding efficacy implies nonbinding efficacy: if the path reaches an upper boundary before any futility stop, nonbinding monitoring also observes that crossing. Conversely, nonbinding efficacy not counted under binding rules must occur after a prior futility trigger. Therefore, when both rules are applied to the same complete paths with the definitions above,
The equality follows path by path: an outcome counted only under nonbinding rules is exactly a later efficacy crossing following an earlier futility trigger. Averaging the indicator of that event over the same \(M\) paths gives the difference in the two estimated probabilities. This is a useful implementation check.
Several diagnostics can reveal errors. Nonbinding efficacy should never be less frequent than binding efficacy when the same complete paths and boundaries are used. The difference should match the count of futility-first paths that later cross efficacy. Every efficacy crossing must occur at a look where \(Z_i\geq u_i\), and futility must be recorded only at an interim look before efficacy. Check that the simulated means and correlations are consistent with the specified canonical model; for example, under \(H_0\), each \(Z_i\) should have mean near zero and variance near one. Large deviations can indicate an incorrect increment variance, incorrect standardization, or an indexing error.
Monte Carlo estimates also vary across runs. If an estimate changes materially when the number of paths is increased, the simulation may be too small for the precision required. A fixed random-number seed aids reproducibility but does not remove Monte Carlo error. If the path-level ordering or subset checks fail, first investigate coding and event definitions rather than interpreting the difference as a substantive design result.
What Simulation Adds—and What It Does Not
Analytic joint-probability calculations or recursive multivariate-normal integration can evaluate crossing probabilities without Monte Carlo sampling error, and are often preferable when the design and model permit reliable calculation. Simulation is especially useful as an independent check, for illustrating individual path behavior, and for evaluating more complicated rules that are awkward to express as a small number of joint probabilities. Using common simulated paths for the binding and nonbinding comparison generally makes that comparison more precise.
Simulation does not repair an incorrect model or justify unplanned boundaries. Its estimates are conditional on the assumed information fractions, joint distribution, effect value, and decision rules. It can also be computationally inefficient for very rare events: observing few or no events does not establish that their true probability is zero. In that setting, increase the simulation effort or use a suitable analytic calculation, and report the remaining uncertainty.
Check Your Understanding
Answer each question, then check the concise solutions below.
- Why is it invalid to simulate independent \(Z_i\) values at successive looks under the canonical model?
- In a simulation of 5,000 paths, 180 paths trigger futility before efficacy and 12 of those later cross an efficacy boundary. What is the estimated futility-trigger rate, and how much larger is nonbinding efficacy than binding efficacy?
- Under a nonbinding interpretation, does a lower-boundary crossing itself count as an actual stop? Distinguish the trigger rate from the actual stop rate.
- What path-level relationship should hold between binding and nonbinding efficacy outcomes when the same simulated paths are used?
Solutions. (1) Successive statistics are positively correlated under the canonical model, with \(\operatorname{Cov}(Z_i,Z_j)=\sqrt{t_i/t_j}\) for \(i\leq j\); independent simulation would give the wrong path distribution. (2) The futility-trigger rate is \(180/5{,}000=0.0360\). The nonbinding efficacy estimate exceeds the binding estimate by \(12/5{,}000=0.00240\), because those 12 paths cross efficacy after futility. (3) No. The lower crossing is recorded as a trigger; if the rule is not enforced, its actual stop rate is zero, while its hypothetical binding stop rate is the trigger rate before efficacy. (4) Every binding efficacy path is also a nonbinding efficacy path, and the nonbinding-only efficacy paths are exactly those with futility first and efficacy later.