End-to-End Simulation of a Multi-Arm Multi-Stage Design

From a Design Specification to Replicate-Level Results

A MAMS simulation is useful only if the whole sequence of decisions is represented in the right order. Generating plausible outcomes is not enough: the simulation must apply the interim rule, carry forward only eligible arms, test them as specified, and record the resulting decisions. Repeating that complete process gives estimates of operating characteristics such as family-wise error and power.

This tutorial works through a hypothetical design with two experimental arms and a shared control. It is deliberately specific: a clear design makes it possible to identify exactly which data are used at each stage and why the resulting tests have their stated error properties. The same workflow can be adapted to other prespecified designs, but changing a rule or reusing data can change the operating characteristics.

The stage-1 rule is a screening rule, not an efficacy test: no arm can be declared successful at the interim. The final comparisons use only new stage-2 observations, so their test statistics are independent of the stage-1 selection decisions under the model. The control group is shared within each stage, meaning that comparisons with A and B are not independent of each other. The per-arm final threshold is Bonferroni's allocation of a family-wise level \(0.025\) across two possible comparisons.

Model, Statistics, and Error Control

Write \(\mu_0,\mu_A,\mu_B\) for the mean outcomes in control, A, and B, respectively. In a scenario, the simulation specifies these means. Each participant's outcome in group \(g\) follows \(N(\mu_g,1)\), and outcomes are independent across participants. Assignment counts are fixed as stated above; there are no missing outcomes, dropouts, or changes to the allocation in this worked design. These are assumptions of the example, not universal properties of MAMS trials.

For arm \(j\in\{A,B\}\), let \(\bar{x}_{j,1}\) and \(\bar{x}_{0,1}\) be the stage-1 sample means. Since each stage-1 group has 40 participants and the variance is known to be 1, the standard error of the difference is \(\sqrt{1/40+1/40}\). The interim statistic and continuation indicator are therefore

Here \(I(\cdot)\) is 1 when its condition is true and 0 otherwise. At stage 2, the comparison uses new data, with 60 participants in each contributing group:

A retained arm succeeds if \(Z_{j,2}\geq z_{0.9875}\), or equivalently if its one-sided p-value is at most \(0.0125\). The exact quantile defines the boundary; its approximate numerical value is about 2.241. An arm that stops at stage 1 is not tested at stage 2 and cannot be successful in that replicate.

The final tests are valid after selection because the stage-2 observations are independent of all stage-1 outcomes under the model. Conditional on an arm being retained, its stage-2 test still has its null distribution when \(\mu_j=\mu_0\). Thus its conditional probability of rejection under that null is \(0.0125\). For the two comparisons, the probability of at least one false rejection is at most the sum of the two marginal probabilities, hence at most \(2(0.0125)=0.025\). This Bonferroni bound does not require the two statistics to be independent; the shared stage-2 control generally makes them correlated.

A Complete Replicate: Generate, Select, Test, Record

The simulation unit is one complete trial replicate. Within a replicate, generate all stage-1 observations, apply the continuation rule, generate stage-2 observations for the retained arms and their shared control, perform the final tests, and save the decisions. It is important to generate stage-2 data only for retained arms, while generating a single stage-2 control group if at least one arm continues.

A useful record has one row per replicate and fields that distinguish selection from rejection. For example, save both continuation indicators, both final success indicators, the number of arms retained, and whether any final rejection occurred. A final success indicator must be zero for an arm not retained; otherwise summaries can accidentally count an unavailable test as a negative or positive result inconsistently.

From Replicates to Operating Characteristics

For a specified scenario, the estimated power for arm A is the fraction of replicates in which A is declared successful. This definition incorporates both its chance of passing interim selection and its chance of passing the final test. Under the global null, the estimated family-wise error rate is the fraction of replicates with at least one false final success. If only one arm is null, its false-positive rate is calculated from that arm's success indicator. These quantities have different denominators and meanings; the scenario's true means must be recorded alongside the summaries.

Why the Workflow Matters: Assumptions and Failure Modes

The order of operations is part of the statistical design. In this example, using stage-1 data again in the final test would make the final statistic dependent on selection and invalidate the simple conditional error argument above. A design that reuses interim data can be simulated, but its testing rule and error control must explicitly account for that reuse; it is not a harmless coding variation.

Likewise, a simulation must represent the shared control correctly. Generating a separate control group for each comparison would remove the correlation induced by the common control and could distort the distribution of “at least one success.” Conversely, using one control group but treating the two final tests as independent when summarizing family-wise error would ignore their dependence. Bonferroni's bound remains valid without independence.

Check at least the following before interpreting results:

• Design consistency: verify group sizes, stage timing, allocation, and whether stage-2 control is generated once or more than once.
• Decision consistency: confirm the interim boundary includes equality as specified, and that the final rule uses \(Z\geq z_{0.9875}\) or \(p\leq0.0125\). At the exact boundary, these two rules agree.
• Record consistency: ensure no arm can be marked successful if it was not retained, and ensure “any success” is computed from the arm-level indicators.
• Scenario consistency: verify that null scenarios truly set the relevant treatment and control means equal, and that power scenarios use the intended effects.
• Numerical checks: flag missing values, non-finite statistics, or unexpected sample counts rather than silently treating them as non-rejections.

The common shortcut “generate a large matrix of outcomes, then test whatever arms look promising” is not itself a design specification. It can work computationally only if the stored data and indices preserve the required stage boundaries, shared-control structure, and eligibility rules. Modular code for generation, decision logic, and summaries makes these checks easier; the earlier material on simulation verification and coding errors gives complementary tools for testing those components.

Simulation does not repair a poorly specified design, guarantee control under unexamined distributions, or remove Monte Carlo error. Results apply to the scenarios and assumptions actually represented. In particular, the normal model with known variance here is a transparent teaching example; other outcome types, nuisance-parameter estimation, missingness, or data reuse require their own generation and analysis rules.