Why Simulate a Null Configuration?
A MAMS design can produce a complicated path: some treatment arms stop early, others continue, and the final rejection set depends on the sequence of interim statistics and the prespecified decision rules. A boundary may appear to control error in a calculation while a coding choice, dependence assumption, or dropped-arm rule makes the implemented design behave differently. Simulation provides a direct diagnostic: generate trials under a specified pattern of true and false null hypotheses, apply the complete design, and count how often at least one true null is rejected.
The target here is the familywise error rate (FWER) for one specified configuration. It is not enough to simulate only the global null if the design’s behavior can depend on which arms are null, which are active, or how stopping and rejection rules operate. Conversely, a result for one configuration alone does not establish strong control for every configuration.
Specify the Configuration and the Trial Generator
Before drawing simulated outcomes, specify what is held fixed. This includes the set \(I_0\), treatment effects for the non-null arms, nuisance parameters, allocation, stage sizes, outcome distribution, and any features such as delayed outcomes or missing observations that affect the actual design. Also specify the complete decision rule: the stagewise boundaries, which arms remain eligible after each look, stopping rules, and how rejections are declared.
The simulated data must preserve the dependence present in the trial. In a shared-control design, treatment-versus-control statistics are generally correlated because they use the same control observations. Statistics from repeated looks are also dependent because earlier observations contribute to later analyses. Simulate the trial’s outcomes and calculate its statistics from those outcomes, or generate a correctly specified joint distribution for all the statistics. Drawing every arm-by-stage statistic independently is generally not a valid substitute.
A useful simulation record contains, for each replicate, the stagewise statistics, active-arm set at each look, stopping stage, and rejection set. It should also include the reason an arm left the design: for example, it crossed a futility rule, was dropped under a prespecified rule, or the trial stopped. These details make the estimate auditable and can reveal a coding error that a final FWER count conceals.
Estimate FWER and Its Monte Carlo Uncertainty
For \(B\) independent simulated trials, define \(E_b=1\) if replicate \(b\) falsely rejects at least one true null, and \(E_b=0\) otherwise. The estimated FWER is the proportion of replicates with a familywise error. Since the indicators are Bernoulli, the estimate is unbiased and its Monte Carlo standard error (the simulation-to-simulation uncertainty, conditional on the generator and design) is estimated using the sample proportion.
When the number of errors and the number of non-errors are both reasonably large, an approximate 95% Monte Carlo interval is \(\widehat{\operatorname{FWER}}\pm1.96\widehat{SE}_{MC}\). Near zero or one, that interval can behave poorly; a Wilson interval or an exact binomial interval is preferable. These intervals quantify Monte Carlo uncertainty, not uncertainty about whether the chosen model or nuisance parameters are correct.
Worked Example: Counting Errors Along the Rejection Path
Consider a hypothetical two-stage trial with treatment arms A and B and a shared control. In the specified configuration, A is a true null and B is non-null. The complete design is applied to 1,000 simulated trials. The stored path records show that A is rejected at the first look in 9 trials, is not rejected at the first look but is rejected at the second look in 17 trials, and is never rejected in the other 974 trials.
The familywise-error event is rejection of A at either look. Whether B is also rejected does not change this event, because B is non-null in this configuration. Thus $$ \widehat{\operatorname{FWER}}=\frac{9+17}{1000}=\frac{26}{1000}=0.0260. $$
The estimated Monte Carlo standard error is $$ \widehat{SE}_{MC}=\sqrt{\frac{0.0260(1-0.0260)}{1000}} =\sqrt{0.00002532}=0.00503. $$ The approximate 95% interval is \(0.0260\pm1.96(0.00503)\), or approximately \(0.0161\) to \(0.0359\). The point estimate is 2.60%, but this simulation is too imprecise to distinguish it clearly from a 2.5% target. The path counts also show when the false rejections occurred, which can help locate a stage-specific implementation problem.
A Short Justification and a Calibration Example
For a fixed configuration, each \(E_b\) has mean \(p=\operatorname{FWER}(I_0)\) and variance \(p(1-p)\). Independence across replicates therefore gives $$ E(\widehat{\operatorname{FWER}})=p,\qquad \operatorname{Var}(\widehat{\operatorname{FWER}})= \operatorname{Var}\left(\frac{1}{B}\sum_{b=1}^{B}E_b\right) =\frac{p(1-p)}{B}. $$ Replacing the unknown \(p\) by its estimate gives the reported Monte Carlo standard error. This derivation explains why increasing \(B\) reduces simulation noise at the rate \(1/\sqrt{B}\); multiplying the number of replicates by four approximately halves the standard error.
For boundary calibration, simulate a candidate schedule, compare its estimated FWER with the target, and use additional simulation or a prespecified search procedure to refine the schedule. A small point-estimate excess is not, on its own, persuasive evidence of inflation. Conversely, a point estimate close to the target with a very wide interval does not establish that the boundary is well calibrated.
Worked Example: Interpreting Monte Carlo Precision Near the Target
Suppose a hypothetical candidate boundary schedule is evaluated in \(B=10{,}000\) independent trials under a configuration with one true-null arm. That arm is falsely rejected in 276 trials. The estimate is $$ \widehat{\operatorname{FWER}}=\frac{276}{10{,}000}=0.0276. $$
Its estimated Monte Carlo standard error is $$ \widehat{SE}_{MC}=\sqrt{\frac{0.0276(1-0.0276)}{10{,}000}} =\sqrt{0.000002684}=0.00164. $$ The approximate 95% interval is \(0.0276\pm1.96(0.00164)\), or \(0.0244\) to \(0.0308\). If the target is 0.025, the estimate is above target, but the interval includes 0.025. These results do not give clear Monte Carlo evidence that the true FWER under this generator exceeds 0.025. They also do not prove control; the interval is compatible with values above the target.
For a rough planning calculation, if the anticipated FWER is 0.025 and the desired Monte Carlo standard error is at most 0.001, solve \(p(1-p)/B\leq0.001^2\): $$ B\geq\frac{0.025(1-0.025)}{0.001^2}=24{,}375. $$ This is a precision calculation, not a guarantee that the model is correct or that a selected boundary will meet the target.
Validation, Diagnostics, and Limits
A simulation used to choose a boundary is also a source of selection: a schedule may look favorable partly because of random variation in the same simulation runs used to select it. A practical safeguard is to use one simulation set for calibration, then evaluate the chosen schedule in an independent validation set with a fixed random seed policy and enough replicates for the required precision. Report the validation estimate, interval, number of replicates, configuration, and decision algorithm. If validation is inconclusive, increase simulation precision rather than treating a point estimate as a pass.
Inspect more than the overall proportion. Check that simulated null statistics have the intended means, variances, and correlations; that the number of observations and active arms at each stage match the design; and that recorded stopping and rejection paths obey the specified rules. For example, a path in which an arm is rejected after it was dropped, or in which a final rejection occurs without satisfying the required test, signals a design-implementation discrepancy. Compare empirical stagewise crossing frequencies with expected behavior where that check is available.
A common mistake is to count a trial as an error whenever any treatment hypothesis is rejected. If some arms are non-null, that incorrectly counts correct rejections as familywise errors. The correction is to mark the true-null set before simulation and count only replicates in which at least one member of that set is rejected. Another mistake is to simulate independent test statistics despite shared controls or repeated observations; the correction is to preserve their joint dependence.
Simulation is especially useful when arm dropping, adaptive rules, or complex joint distributions make direct calculation difficult. If the joint distribution and stopping rules are simple enough, numerical integration or an analytic calculation can be more efficient and can avoid Monte Carlo noise. Simulation remains conditional on the generator: it cannot diagnose model misspecification merely by producing a narrow interval. Finally, one specified configuration diagnoses FWER only for that configuration; a claim of strong control requires justification across all configurations with at least one true null, not one reassuring simulation.
State the true-null set, effects, nuisance parameters, and trial features to be represented.
Preserve shared-control and repeated-look dependence, then apply the full prespecified decision algorithm.
Store active arms, stopping stage, rejection set, and whether any true null was rejected.
Report the error proportion and Monte Carlo uncertainty; use independent validation after boundary selection.
Worked Example: Comparing Candidate Boundaries on Validation Runs
Suppose two hypothetical boundary schedules are checked under the same specified configuration, with one true-null arm, using 20,000 independent validation trials per schedule. The schedules differ only in their prespecified boundary stringency. The less stringent schedule produces 624 false rejections; the more stringent one produces 482.
The respective estimates are \(624/20{,}000=0.0312\) and \(482/20{,}000=0.0241\). For the more stringent schedule, $$ \widehat{SE}_{MC} =\sqrt{\frac{0.0241(1-0.0241)}{20{,}000}} =0.001085. $$ Its approximate 95% interval is \(0.0241\pm1.96(0.001085)\), or \(0.0220\) to \(0.0262\). The less stringent schedule’s estimate is \(0.0312\), with standard error \(\sqrt{0.0312(0.9688)/20{,}000}=0.00123\), and approximate interval \(0.0288\) to \(0.0336\).
For a 0.025 target, the first schedule’s interval lies above target, while the second schedule’s interval includes it. The second schedule is a more promising candidate in this configuration, but its simulation does not demonstrate that its FWER is at most 0.025. A larger independent validation run can narrow Monte Carlo uncertainty; neither more replicates nor a narrower interval resolves uncertainty about the assumed data-generating model.
Check Your Understanding
Use the definitions and simulation logic above. Brief solutions follow the questions.
- In a configuration with three arms, arms 1 and 3 are true nulls. A replicate rejects arms 2 and 3 only. Is this a familywise error?
- For \(B=5{,}000\) simulated trials, 125 have at least one true-null rejection. Calculate the estimated FWER.
- Why can simulating each stagewise treatment-versus-control statistic independently give a misleading FWER estimate?
- A validation estimate is 0.024 with a 95% Monte Carlo interval from 0.018 to 0.030, against a target of 0.025. What can be concluded?
Solutions. (1) Yes: arm 3 is a true null and is rejected, even though arm 2 is non-null. (2) \(125/5{,}000=0.0250\), or 2.50%. (3) It removes dependence induced by shared controls and repeated observations, changing the probability of joint boundary crossings and hence the FWER. (4) The point estimate is below target, but the interval includes values above target; the simulation is inconclusive about whether the FWER is at most 0.025 under that configuration and generator.