Simulating Type I Error for a Seamless Design

Why Simulate the Complete Decision Path?

In a seamless phase 2/3 trial, the confirmatory test is reached only if earlier decisions allow the trial to continue. Participants are randomized, phase 2 outcomes are observed, a treatment may be selected, and stopping rules determine whether phase 3 proceeds. The null rejection rate is the probability that this complete process ends in a false confirmatory rejection when the null hypothesis is true. It is therefore a property of the design’s full sequence of rules, not just of the final test considered in isolation.

Simulation estimates this probability by generating many hypothetical trials under a specified null model and applying the design exactly as written to each one. A simulation is useful only if it reproduces the design’s randomization, selection, stagewise testing, and stopping. For example, simulating the final test but ignoring the chance that phase 2 stops early does not estimate the operating characteristic of the actual design.

Specify the Design and Null Model First

Consider a hypothetical design with three candidate treatments, A, B, and C, and a control. The endpoint is a continuous outcome measured so that larger values are better. The example uses a simple model to make the simulation transparent; a real simulation should use the endpoint model and assumptions specified for its own design.

In phase 2, 120 participants are randomized in equal numbers to control, A, B, and C: 30 per group. Let \(\bar{x}_j\) denote the phase 2 sample mean for group \(j\). The design selects the treatment with the largest observed mean difference from control, provided that difference is at least 0.15 outcome units. Otherwise, the trial stops without a confirmatory claim. If a treatment is selected, phase 3 enrolls 100 new participants to that treatment and 100 new participants to control, with equal allocation.

After 50 phase 3 participants per group have outcomes, an interim futility rule stops the trial if the selected-treatment-versus-control \(Z\)-statistic is below 0. If the trial continues, the remaining participants are enrolled. At the end, the design rejects the selected treatment’s null hypothesis only if the one-sided final \(Z\)-statistic exceeds 1.96. The phase 2 data select the treatment but are not included in this confirmatory test.

For this example, assume each outcome is independent and normally distributed with variance 1, and under the global null all four group means are 0. The phase 2 and phase 3 participants are distinct. Randomization fixes the stated group sizes; phase 3 allocation is randomized equally between the selected treatment and control. Outcomes are complete, and the variance used in the test is known to be 1. These assumptions make the test statistic’s null distribution explicit. They are assumptions of this hypothetical model, not facts about a real clinical trial.

The null hypothesis for simulation is the global null: none of A, B, or C differs from control. Each simulated trial must still follow the same allocation and decision rules it would follow if the outcomes happened to favor a treatment. It would be incorrect to force the selected treatment to be a particular arm, or to bypass selection because the simulation is being run under the null.

Implement Each Replicate in Order

The simulation should follow the protocol’s temporal sequence. The flow below gives the steps for the hypothetical design; the group sizes and thresholds should be replaced by those of the design being assessed.

In phase 3, with \(n\) participants per group, the test statistic is the observed difference in means divided by its standard error. Under the stated model, each group mean has variance \(1/n\); the independent difference has variance \(1/n+1/n=2/n\). Thus \(SE=\sqrt{2/n}\). At the final analysis \(n=100\), and at the interim analysis \(n=50\). The one-sided threshold 1.96 corresponds approximately to a significance level \(\alpha=0.025\) for a standard normal statistic.

Estimate the Rate and Quantify Simulation Error

Across \(M\) simulated trials, each \(I_b\) is a Bernoulli random variable: it equals 1 with probability \(p_{\text{null}}\) and 0 otherwise. If the replicates are independently generated, their average estimates that probability. Its approximate Monte Carlo standard error is \(\sqrt{\hat{p}_{\text{null}}(1-\hat{p}_{\text{null}})/M}\). This standard error measures simulation noise, not uncertainty about the model assumptions or about how well the model represents a future trial.

Why the Final-Test Argument Is a Useful Check

A simple probability argument provides a check on the simulation. Let \(C\) denote the event that the design selects a treatment and continues far enough to make the final test. Let \(R\) denote final rejection. For this design, \(R\) requires the final statistic to exceed 1.96. Under the global null, conditional on phase 2 selection, the new phase 3 data have no treatment effect and are independent of phase 2. The selected arm is therefore fixed with respect to the phase 3 outcomes, and the final \(Z\)-statistic has a standard normal distribution under the stated model.

The interim futility rule can only remove possible rejections; it cannot create one. The rejection event is contained in the event that the final statistic exceeds 1.96. Therefore:

This is a justification for expecting a rate no greater than approximately 0.025 under the stated assumptions. It does not supply the exact rate, because phase 2 selection and the interim rule affect whether the final test is reached. The simulation estimates the rate for the complete set of rules. If the design instead used phase 2 outcomes in its confirmatory statistic, or allowed an early efficacy rejection, the argument above would not establish control; those rules would need to be included and justified in the evaluation.

Diagnostics, Alternatives, and Limitations

Before trusting the estimate, verify that the implementation records the entire decision path. Useful checks include confirming that every replicate has the intended randomized group sizes, that selection follows the prespecified tie and threshold rules, that no phase 3 outcomes are generated or analyzed after a stop, and that stopped trials are counted as no rejection. Check a few replicates by hand, as in the worked example, and verify that the final statistic uses the intended sample size and standard error.

A common mistake is to simulate a fresh, fixed phase 3 comparison for a treatment chosen in advance, then treat its rejection rate as the seamless design’s null rejection rate. That omits randomized phase 2 selection and the stopping path. In this particular example, independent phase 3 data mean selection does not invalidate the phase 3-only test, but ignoring the probability of stopping still fails to reproduce the design’s overall rejection event. In a design that reuses phase 2 evidence, independence cannot be assumed and the simulated joint data and testing rule must reflect that reuse.

Simulation is flexible: it can follow a complicated sequence of allocation, selection, and stopping rules that may be awkward to analyze in closed form. A direct probability argument, as above, is a valuable independent check when its assumptions apply. For simpler designs, exact calculations or analytical calibration may give a more precise answer without Monte Carlo error. Simulation does not automatically establish validity: its estimate applies to the specified model and the implemented rules. It can also be imprecise when \(M\) is too small, and errors in code can make a precise estimate wrong.

The global-null simulation answers a specific question: how often does the design reject when all candidate treatments have no effect? It does not by itself establish error control for every configuration in which some treatments work and others do not. Nor does it establish that the assumed outcome model is appropriate. The estimated rate should be reported together with the null model, the decision rules, the number of replicates, and the Monte Carlo standard error so that its scope is clear.