Why Evaluate the Rule by Simulation?
An unblinded re-estimation rule uses interim treatment-effect information to decide whether a trial stops or continues with a changed sample size. Its consequences are not captured by the maximum sample size alone. Across repeated trials, the rule can produce a distribution of final sample sizes, different stopping frequencies, and a power that varies with the true treatment effect. Simulation estimates these quantities by repeatedly generating trial data and applying the rule exactly as specified.
The key is to simulate the whole adaptive procedure, not just its final sample size. Each replicate must generate interim data, make the rule’s decision from those data, generate any required later data, and apply the specified final decision. The resulting estimates describe the performance of that complete procedure under the chosen data-generating assumptions.
Define the Simulation Before Running It
First specify the outcome model, allocation, interim timing, re-estimation rule, and final decision rule. The following simple model will be used for illustration. Outcomes are continuous, allocation is equal, and observations are independent across participants and treatment groups. For a participant in the control group, \(X\sim N(\mu_C,\sigma^2)\); for a participant in the treatment group, \(X\sim N(\mu_C+\delta,\sigma^2)\). The common standard deviation \(\sigma\) is known, there are no missing outcomes, and the treatment effect is the difference in population means, \(\delta\). These assumptions make the examples transparent; an evaluation for a real design should simulate its planned endpoint and analysis assumptions instead.
Suppose the trial starts with \(n_1=50\) participants per group. At the interim analysis, calculate \(Z_1=(\bar{x}_T-\bar{x}_C)/\{\sigma\sqrt{2/n_1}\}\). The prespecified rule is: stop for futility if \(Z_1<0.5\); continue to 100 per group if \(0.5\leq Z_1<1.5\); and increase to 150 per group if \(Z_1\geq1.5\). This is an illustrative rule, not a recommendation for a particular trial. At the final analysis, use the one-sided known-variance statistic \(Z_F=(\bar{x}_T-\bar{x}_C)/\{\sigma\sqrt{1/n_T+1/n_C}\}\), rejecting \(H_0:\delta\leq0\) when \(Z_F>1.96\). If the trial stops for futility, it does not reject.
Use independent simulated trials as replicates. In replicate \(b\), record the final rejection indicator \(R_b\), total sample size \(N_b\), whether the maximum sample size was selected, and the trial’s stopping outcome. Total sample size means participants in both groups combined; state clearly if a different convention is used. For \(B\) replicates, the natural estimates are averages of those recorded quantities:
Here \(I_b=1\) if replicate \(b\) reaches the increased sample size and \(I_b=0\) otherwise. A stopping probability is estimated in the same way: average an indicator for the specified stopping event. These formulas follow because each indicator is 1 precisely when its event occurs, so its expectation is that event’s probability. Averaging the simulated indicators therefore estimates the probability, while averaging \(N_b\) estimates the expected total sample size under the model.
Worked Example: Apply the Rule to One Hypothetical Trial
Suppose \(\sigma=10\) outcome units. In a hypothetical trial, the interim means are 53 units in the treatment group and 50 units in the control group, with 50 participants per group.
The interim standard error is \(10\sqrt{2/50}=2\) units, so \(Z_1=(53-50)/2=1.50\). The rule assigns this trial to the increase category because \(Z_1\geq1.5\). The trial therefore enrolls 100 additional participants per group, reaching 150 per group.
Suppose the later-stage means, based on those additional participants, are 55.4 units for treatment and 50.0 units for control. The combined treatment-minus-control difference is the sample-size-weighted average of the stage differences: \( \{50(3.0)+100(5.4)\}/150=4.60 \) units. Thus \(Z_F=4.60/\{10\sqrt{2/150}\}=3.984\). Since \(3.984>1.96\), this replicate rejects under the stated final decision rule. It contributes 1 to the rejection count and 300 to the total-sample-size sum. This is a hypothetical illustration of the algorithm, not evidence about a real trial.
Translate the Algorithm into Replicate-Level Records
A simulation should make each rule branch explicit. One implementation may generate all outcomes at the start and reveal only the interim data to the decision rule; another may generate the later-stage outcomes only after the trial continues. Under the stated independent-outcomes model, both approaches produce the same distribution if the later outcomes are generated independently and the rule uses only interim information. In either case, a replicate should not use outcomes that would not yet be available when making its interim decision.
Choose \(\delta\), \(\sigma\), allocation, interim sample size, maximum sample size, and all rule and final-test cutoffs. Keep these fixed across replicates within a scenario.
Generate the control and treatment outcomes under the specified model, and calculate \(Z_1\) using only the interim observations.
Record a futility stop, a planned continuation, or a sample-size increase according to the stated cutoffs. Do not revise the rule after seeing simulation results.
For a continuing trial, generate the additional observations, calculate the final statistic using the specified analysis, and record rejection and total sample size. For a stopped trial, record its stopping outcome and sample size.
Average the rejection indicator, sample size, increase indicator, and indicators for each stopping outcome over all replicates in that scenario.
Worked Example: Calculate Operating-Characteristic Estimates
Consider 10,000 hypothetical simulated trials at a specified treatment effect. The resulting counts are 3,100 futility stops at 50 participants per group, 3,800 continuations to 100 per group, and 3,100 increases to 150 per group. Of the continuations, 700 reject at the final analysis; of the increased trials, 900 reject. No futility-stopped trial rejects.
The estimated power is the total number of rejections divided by the number of replicates: \((700+900)/10{,}000=0.1600\), or 16.00% under this treatment-effect scenario and the specified model. This is a rejection probability estimate under the alternative, not the probability that \(H_0\) is true.
The estimated increase frequency is \(3{,}100/10{,}000=0.3100\), or 31.00%. The estimated futility-stopping probability is also 31.00%, and the planned-continuation frequency is \(3{,}800/10{,}000=38.00\%\). These three outcomes sum to 100%.
The estimated mean sample size per group is \(\{3{,}100(50)+3{,}800(100)+3{,}100(150)\}/10{,}000=100\) participants. Since allocation is equal, estimated mean total sample size is \(2(100)=200\) participants. This arithmetic also checks that the sample-size average uses every replicate, including those that stopped early.
Check the Simulation Against the Design
A useful diagnostic is to check branch frequencies against calculations based on the interim model. Under the normal model above, \(Z_1\sim N(\delta\sqrt{n_1/2}/\sigma,1)\). This follows because the interim difference in sample means has expectation \(\delta\) and standard error \(\sigma\sqrt{2/n_1}\). The interim branch probabilities can therefore be calculated directly from a standard normal distribution, providing a check on the simulated rule decisions.
Worked Example: Benchmark the Interim Branch Frequencies
Let \(n_1=50\), \(\sigma=10\), and \(\delta=2\) outcome units. Then the mean of \(Z_1\) is \(2\sqrt{50/2}/10=1\), and its standard deviation is 1. The probability of stopping for futility is \(P(Z_1<0.5)=\Phi(0.5-1)=\Phi(-0.5)=0.3085\).
The probability of increasing is \(P(Z_1\geq1.5)=1-\Phi(1.5-1)=1-\Phi(0.5)=0.3085\). The remaining probability is a planned continuation: \(1-0.3085-0.3085=0.3830\). A simulation with many replicates should produce branch frequencies near these benchmarks, though the simulated proportions will not generally equal them exactly. This check tests the interim data generation and branch logic; it does not calculate final power.
Diagnostics should also include checks that total sample size takes only the values allowed by the rule, every replicate enters exactly one decision branch, and the recorded rejection outcome follows the final cutoff. Inspect results across several treatment-effect scenarios, including the null and effects below, near, and above the planning effect. A branch that never occurs may be plausible for a particular scenario, but should prompt a check that the cutoff, scale, and \(Z_1\) calculation use consistent units.
For example, if the interim difference is measured in outcome units but is compared directly with a standardized \(Z\) cutoff, the rule is implemented incorrectly. Likewise, calculating the increase frequency only among trials that did not stop for futility gives a conditional frequency, not the unconditional probability of increase among all randomized trials. Label conditional summaries explicitly when they are intended.
What Simulation Can and Cannot Establish
Simulation is especially useful when a rule combines several data-dependent branches, a maximum sample size, and a final analysis whose behavior is cumbersome to calculate directly. It can estimate several consequences from the same simulated trials, making trade-offs among rejection probability, participant burden, and stopping outcomes visible. The results are conditional on the data-generating scenarios and on faithful implementation of the rule.
For a simple normal model, direct calculations can provide benchmarks for interim branch probabilities and sometimes for final operating characteristics. Those calculations can be faster and more precise for that model. Simulation is more flexible when assumptions or rules become complicated, but flexibility does not guarantee accuracy: a coding error or an unrealistic data-generating model can give precise-looking but misleading results. A sound evaluation checks the code against available calculations and repeats the analysis over scenarios that reflect plausible uncertainty.
Finally, an estimated rejection probability does not by itself establish that the procedure has the intended inferential properties. In particular, evaluating power and sample-size behavior is not a substitute for justifying the final test’s error-rate behavior under the adaptive rule. That question requires its own analysis. Keep the conclusion appropriately scoped: report what the simulation estimated, under which assumptions and rule, and do not treat the simulation as proof that those assumptions or inferential guarantees hold in practice.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a simulation with 5,000 replicates, 1,250 trials reach the maximum sample size. What is the estimated increase frequency?
- Under the rule in this tutorial, why must a futility-stopped trial still be included when estimating expected sample size?
- With \(n_1=50\), \(\sigma=10\), and \(\delta=0\), what is the distribution of \(Z_1\), and what is the theoretical probability of an increase?
- A report gives a 24% rejection rate but does not state the true treatment effect used to generate the data. What essential context is missing?
Solutions. (1) The estimated increase frequency is \(1{,}250/5{,}000=0.2500\), or 25.00%. (2) Expected sample size averages the sample size over all trials generated by the procedure. A stopped trial has enrolled 50 participants per group, so omitting it would condition on continuation and would not estimate the unconditional mean. (3) The mean of \(Z_1\) is \(0\sqrt{50/2}/10=0\), with variance 1, so \(Z_1\sim N(0,1)\). The increase probability is \(P(Z_1\geq1.5)=1-\Phi(1.5)=0.0668\). (4) The report must state the treatment effect, along with the relevant model and procedure, because power is the rejection probability under a specified alternative and assumptions.