Why Simulate the Complete Decision Rule?
An adaptive trial is not defined only by its planned sample size or its final statistical test. Interim data can change whether the trial stops or continues, so the operating characteristics belong to the entire prespecified decision rule. A simulation must reproduce each possible path: data generation, interim analysis, stopping or continuation, and any final decision. This tutorial constructs a two-stage example and evaluates its error rate, power, expected sample size, Monte Carlo precision, and implementation risks.
The example uses a continuous endpoint for which larger values are better. It is deliberately simple enough to check by hand; real applications may need richer data-generating models to represent the design and population adequately.
Specify the Trial Before Simulating
Suppose treatment and control outcomes are independent normal observations with a common, known standard deviation of 10 endpoint units. Each arm has 30 participants at the interim analysis. If the trial continues, a further 30 participants per arm are enrolled, for a maximum of 60 per arm. Let \(\delta\) denote the treatment-minus-control difference in population means. The hypotheses for the one-sided efficacy decision are \(H_0:\delta\leq 0\) and \(H_a:\delta>0\).
The interim analysis uses the first 30 participants in each arm. Stop for efficacy if its one-sided Z statistic is at least 2.2415. Stop for futility if it is below 0.5. Otherwise, continue to the planned maximum. At the final analysis, declare efficacy if the Z statistic based on all 60 participants per arm is at least 2.2415. These cutoffs are prespecified; the simulation does not choose them after seeing simulated results.
The assumptions for these statistics are that observations are independent within and between arms, the endpoint follows the stated normal model, the standard deviation is known and equal in both arms, and allocation and sample sizes follow the design. Under \(H_0\) at \(\delta=0\), each statistic calculated at its planned sample size has a standard normal distribution. For \(\delta<0\), its mean is shifted below zero. This example has no missing outcomes, no changes to the endpoint or analysis, and no sample-size re-estimation.
The efficacy cutoff is set conservatively. The exact 98.75th percentile of the standard normal distribution is approximately 2.2414027; the operational cutoff 2.2415 is rounded upward, so its one-sided null tail probability is no greater than 0.0125. The futility rule is a stopping rule, not evidence of no treatment effect. Its cutoff of 0.5 is a design choice and does not establish a clinically meaningful effect threshold.
Worked Examples: Following Trial Paths
Worked Example: An Interim Efficacy Stop
In a hypothetical trial, the interim mean is 56 endpoint units in the treatment arm and 50 in the control arm. The difference is \(56-50=6\) units. The interim standard error is \(10\sqrt{1/30+1/30}=10\sqrt{2/30}\approx 2.582\) units, so \(Z_I=6/2.582\approx 2.324\), using the unrounded standard error for the division. Since \(2.324\geq 2.2415\), the prespecified rule stops the trial for efficacy at the interim. No final analysis is performed on this trial path.
This is a decision under the stated hypothetical rule, not a claim that the treatment causes a six-unit improvement or that this result would establish clinical benefit. The simulation’s efficacy definition is statistical and one-sided.
Worked Example: Futility and Continuation Decisions
For a second hypothetical interim, suppose the treatment mean is 52 and the control mean is 51 endpoint units. Then the difference is 1 unit and the same interim standard error is \(2.582\) units. Thus \(Z_I=1/2.582\approx 0.387\). Because \(0.387<0.5\), the trial stops for futility.
For a third hypothetical interim with a mean difference of 2 units, \(Z_I=2/2.582\approx 0.775\). This value is at least 0.5 but below 2.2415, so the trial continues. Suppose its eventual means using all 60 participants per arm are 53 in treatment and 50 in control. The final standard error is \(10\sqrt{2/60}\approx1.826\) units, and \(Z_F=3/1.826\approx1.643\). Since \(1.643<2.2415\), efficacy is not declared. The final value is evaluated only because this path passed the interim continuation rule.
Constructing the Simulation
For each replicate, simulate the full planned data set, apply the interim rule to the first 30 observations per arm, and calculate the final statistic only for trials that continue. Retaining the potential final data in the simulation is useful for verifying the final statistic, although a simulated trial that stops would not actually enroll those additional participants. The following sequence makes the decision logic explicit.
Choose a treatment difference \(\delta\) and generate independent treatment and control outcomes under the specified normal model.
Calculate \(Z_I\). Record an efficacy stop if \(Z_I\geq2.2415\), a futility stop if \(Z_I<0.5\), and continuation otherwise.
For continuing trials, calculate \(Z_F\) from all 60 participants per arm and declare efficacy exactly when \(Z_F\geq2.2415\).
Estimate the efficacy-declaration rate, interim stop rates, and mean number of participants. Repeat under the null and under relevant nonzero treatment effects.
The efficacy-declaration rate under \(\delta=0\) estimates the design’s type I error at the boundary of the composite null. The boundary is the most adverse null value for this one-sided test: shifting \(\delta\) below zero shifts both statistics downward, so it cannot increase the probability of crossing either positive efficacy cutoff. The efficacy rate under a specified positive \(\delta\) estimates power for that scenario; it is not a universal property of the design.
Illustrative Operating Characteristics and Monte Carlo Precision
The table gives hypothetical illustrative simulation output from 200,000 replicates per scenario. The rates are examples for interpreting simulation summaries, not results from a clinical study. “Final-stage contribution” counts final efficacy declarations as a proportion of all replicates, not just the trials that continued; total efficacy is the sum of the disjoint interim and final declaration paths.
| True difference \(\delta\) (units) | Interim efficacy stop | Interim futility stop | Final-stage contribution | Total efficacy declarations | Mean participants |
|---|---|---|---|---|---|
| 0 (null) | 0.0124 | 0.6918 | 0.0112 | 0.0236 | 77.75 |
| 4 | 0.245 | 0.147 | 0.404 | 0.649 | 96.48 |
Under the null scenario, the estimated efficacy rate is 0.0236, an estimate of type I error that is below the prespecified 0.025 upper bound. Under the 4-unit scenario, the estimated power is 0.649, or about 64.9% for that assumed effect and model. The simulation also shows the sample-size trade-off: the mean is lower when more trials stop at interim, but early stopping for futility may also reduce power for effects the design is intended to detect.
For \(R\) independent simulated trials and an estimated event probability \(\hat{p}\), the Monte Carlo standard error is \(SE=\sqrt{\hat{p}(1-\hat{p})/R}\). It measures simulation noise in the estimated operating characteristic, not uncertainty about the true treatment effect or the adequacy of the assumed model.
Worked Example: Quantifying Monte Carlo Error
For the null result, use \(\hat{p}=0.0236\) and \(R=200{,}000\): \(SE=\sqrt{0.0236(1-0.0236)/200{,}000}\approx0.000339\). An approximate 95% Monte Carlo interval is \(0.0236\pm1.96(0.000339)\), or approximately \([0.0229,\,0.0243]\). It describes the precision of the estimated null rejection probability across repeated simulation runs of this size. It does not give a 95% probability that the design’s true type I error lies in that interval.
For power \(\hat{p}=0.649\), \(SE=\sqrt{0.649(1-0.649)/200{,}000}\approx0.00107\). The corresponding approximate 95% Monte Carlo interval is \(0.649\pm1.96(0.00107)\), or \([0.6469,\,0.6511]\). More replicates narrow these Monte Carlo intervals, but they cannot correct misspecified assumptions or inaccurate input values.
Diagnostics, Alternatives, and Limitations
A simulation result is credible only if the simulated process matches the protocol and analysis. Check the following before interpreting a large run:
- Verify the branches. Test known values around each cutoff: \(Z_I=0.49\) must stop for futility, \(Z_I=0.50\) must continue, and \(Z_I=2.2415\) must meet the efficacy boundary. Check that efficacy is evaluated before futility if any rule could otherwise overlap.
- Check sample sizes and standard errors. Confirm the interim statistic uses 30 per arm and the final statistic uses 60 per arm, with the intended units and treatment-minus-control direction.
- Run null and positive-control scenarios. A null rejection estimate substantially above 0.025 warrants investigation; a power estimate that fails to rise as \(\delta\) increases may indicate a coding or sign error.
- Audit stopping and sample-size counts. Each simulated trial should have one terminal status, and its sample size should be 60 if it stops at interim or 120 if it continues. The mean sample size should agree with the observed continuation proportion.
- Separate model error from Monte Carlo error. A narrow Monte Carlo interval only shows that the simulation average is stable under the coded assumptions. It does not show that normality, the standard deviation, adherence, missingness, or the endpoint model is realistic.
The Bonferroni-style bound used here is simple and conservative: dependence between interim and final statistics is allowed, but the actual type I error may be smaller than 0.025. An alternative is to calibrate stage-specific efficacy boundaries using a formal group-sequential error-spending approach, which can use the relationship between interim and final information more efficiently. Other alternatives may be appropriate when rules include response-adaptive allocation or Bayesian predictive criteria, but they require their own prespecified decision definitions and operating-characteristic evaluation. Comparisons should use the same scenarios and outcomes rather than a single headline power value.
This illustration assumes a known standard deviation, a normally distributed endpoint, complete data, fixed allocation, and no operational delay or protocol deviation. If the real design uses an estimated variance, covariate adjustment, unequal allocation, delayed outcomes, missing data, or changes in enrollment, those features can alter both the statistics and the distribution of decisions. The simulation should represent them when they are material. It should also distinguish the scientific target from the operational rule: a statistical declaration, a futility stop, and a clinically relevant benefit are not interchangeable conclusions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- At an interim analysis, \(Z_I=0.30\). What does the rule do, and what conclusion should not be drawn from that decision?
- Why does the type I error bound not require independence between \(Z_I\) and \(Z_F\)?
- With 100,000 replicates and an estimated power of 0.50, calculate the Monte Carlo standard error.
- If 40% of simulated trials stop at interim and the remaining 60% complete the trial, what is the mean sample size in participants?
Solutions. (1) Since \(0.30<0.5\), the trial stops for futility; this does not prove that the treatment has no effect. (2) The probability of either crossing is bounded by the sum of the two marginal probabilities, regardless of dependence. (3) \(SE=\sqrt{0.50(1-0.50)/100{,}000}\approx0.00158\). (4) The mean is \(0.40(60)+0.60(120)=96\) participants.