Stress-Testing Strong Control Across Null Patterns

Why the Global Null Is Not Enough

A sequential multi-arm procedure can appear well calibrated when every treatment is ineffective and still reject a true null too often when some treatments are effective. The reason is that the probability of a false rejection can depend on the full configuration: which hypotheses are true, how strongly the other arms work, and how the interim statistics are related. This tutorial uses simulation to stress-test those possibilities and identify the configurations that deserve the closest scrutiny.

The central comparison is among the global null, where every arm-specific null hypothesis is true; partial-null configurations, where some but not all are true; and single-true-null configurations, where exactly one is true. The last two are essential when assessing strong control. A low error rate under the global null alone does not guarantee that the procedure controls error under every configuration.

Definition: Let $I_0(\theta)$ be the set of indices of the hypotheses that are true under parameter configuration $\theta$, and let $R_i$ indicate whether the sequential procedure rejects $H_i$ by the end of the trial. The familywise error rate at $\theta$ is $$ \operatorname{FWER}(\theta)=P_\theta\left(\bigcup_{i\in I_0(\theta)}\{R_i=1\}\right). $$ Strong control at level $\alpha$ means this probability is at most $\alpha$ for every permitted configuration with at least one true null hypothesis.

For example, with three one-sided arm hypotheses $H_i:\delta_i\leq 0$, the global-null configuration has $\delta_1=\delta_2=\delta_3=0$. A partial-null configuration might have $\delta_1=\delta_2=0$ and $\delta_3>0$; a single-true-null configuration might have $\delta_1=0$ and $\delta_2,\delta_3>0$. These names describe which hypotheses are true, not the particular effect sizes or nuisance parameters. Those values must also be specified when simulating.

What to Simulate and Count

A replicate must represent one complete trial under the configuration being assessed. Generate outcomes under the chosen effects, allocation, variance and dependence structure; calculate the interim statistics; apply the proposed stopping, dropping and boundary rules; and record the final rejection decisions. In each replicate, the FWER event is at least one rejection among the true nulls. Rejections of false nulls do not themselves count as familywise error.

This means the simulation must reproduce the actual sequential procedure, including its look times and decisions after an arm is dropped. Applying only the final-stage test, or simulating separate arms as independent when the design uses a shared control, can change the error event being estimated. The data-generating assumptions should match the intended use of the procedure; when variance, correlation or other nuisance quantities are uncertain, assess plausible values rather than relying on one convenient setting.

For a fixed configuration $\theta$, let $Y_b=1$ if replicate $b$ has at least one false rejection and $Y_b=0$ otherwise. With $B$ independent replicates, the estimated FWER is the proportion of replicates with that event. Since $Y_b$ is Bernoulli with success probability $\operatorname{FWER}(\theta)$, its sample mean has the following Monte Carlo standard error estimate:

\widehat{\operatorname{FWER}}(\theta)=\frac{1}{B}\sum_{b=1}^B Y_b, \qquad \widehat{SE}_{MC}= \sqrt{\frac{\widehat{\operatorname{FWER}}(\theta) \left\{1-\widehat{\operatorname{FWER}}(\theta)\right\}}{B}}.

This standard error describes simulation noise, not uncertainty about whether the model or procedure represents the trial correctly. A rough 95% Monte Carlo interval is $\hat p\pm1.96\widehat{SE}_{MC}$, where $\hat p$ denotes the estimated FWER. The approximation is useful when the number of simulated error events is not very small; for rare events or small counts, use a binomial interval designed for those conditions. Increasing $B$ reduces Monte Carlo error, but cannot correct a misspecified simulation.

Worked Example: Comparing Three Null Patterns

Hypothetical data. A three-arm procedure is simulated in independent batches of $B=100{,}000$ trials for each configuration. The observed numbers of trials with at least one false rejection are 4,800 under the global null, 6,350 under a partial null with two true nulls, and 7,200 under a single-true-null configuration.

The estimated FWERs are, respectively, $$ \frac{4{,}800}{100{,}000}=0.0480,\qquad \frac{6{,}350}{100{,}000}=0.0635,\qquad \frac{7{,}200}{100{,}000}=0.0720. $$

For the single-true-null estimate, the estimated Monte Carlo standard error is $$ \sqrt{\frac{0.0720(1-0.0720)}{100{,}000}} =\sqrt{0.00000066816}\approx0.000818. $$ The rough 95% Monte Carlo interval is $0.0720\pm1.96(0.000818)$, or approximately $0.0704$ to $0.0736$. The corresponding intervals for the global and partial configurations are approximately $0.0467$ to $0.0493$ and $0.0620$ to $0.0650$.

The single-true-null configuration has the largest estimated FWER, and its estimate is well above the design level $\alpha=0.05$. The simulation therefore flags a serious calibration concern for that configuration; it does not establish the exact true FWER or prove which configuration is worst over every possible parameter value. The global-null estimate alone would have concealed the problem.

Compare Error Events, Not Just Armwise Rates

When a configuration has multiple true nulls, its FWER is the probability of the union of their rejection events. It is not generally the sum of the individual false-rejection probabilities, because two or more true nulls may be rejected in the same replicate. Conversely, reporting only a pooled count of all false rejections can overcount trials with multiple errors. Retain replicate-level decisions so that the union event can be counted directly.

Worked Example: Counting the Union for Two True Nulls

Hypothetical data. In 20,000 simulated trials, hypotheses $H_1$ and $H_2$ are true and $H_3$ is false. The procedure rejects $H_1$ in 680 trials and rejects $H_2$ in 620 trials. It rejects both true nulls in 190 trials.

The number of trials with at least one false rejection is the union count: $$ 680+620-190=1{,}110. $$ Thus the estimated FWER is $1{,}110/20{,}000=0.0555$. The armwise estimated false-rejection rates are $680/20{,}000=0.0340$ and $620/20{,}000=0.0310$. Adding those rates gives $0.0650$, which is too large for the FWER because it counts the 190 trials with both false rejections twice.

The estimated Monte Carlo standard error for the union estimate is $$ \sqrt{\frac{0.0555(1-0.0555)}{20{,}000}}\approx0.00162. $$ A rough 95% Monte Carlo interval is $0.0555\pm1.96(0.00162)$, approximately $0.0523$ to $0.0587$. This interval is above $0.05$, so this hypothetical result warrants investigation. It is the union event, not either armwise rate alone, that answers the familywise error question.

Stress-Test Effects and Other Parameters

Finding the most concerning null pattern requires more than listing which hypotheses are true. For each pattern, specify values for effects on the false nulls and for relevant nuisance parameters. A single-true-null configuration with large effects on all other arms may behave differently from the same pattern with weak effects. Such differences can arise when the procedure uses correlated statistics, interim selection, or data-dependent dropping. In particular, earlier material on subset pivotality explains why the joint distribution relevant to true-null tests need not remain unchanged when effects on other arms change.

A practical comparison therefore uses a structured grid: enumerate the nonempty sets of true null hypotheses, and within each set examine plausible effect sizes for the remaining arms and plausible nuisance values. The simulation estimate is conditional on each point in this grid. The highest estimate is the least-favourable pattern among those tested, not automatically the least-favourable configuration in the full parameter space.

Worked Example: A Least-Favourable Effect Setting

Hypothetical data. In a three-arm design, only $H_1$ is true. Two independent batches of $B=50{,}000$ trials are simulated, holding the null effect and nuisance settings fixed. In the first batch, both false-null arms have effect $0.2$ in the study’s prespecified effect units; there are 2,810 trials with a false rejection. In the second, both effects are $0.8$; there are 2,390 such trials.

The estimated FWERs are $2{,}810/50{,}000=0.0562$ and $2{,}390/50{,}000=0.0478$. Their estimated Monte Carlo standard errors are $$ \sqrt{\frac{0.0562(1-0.0562)}{50{,}000}}\approx0.00103, \qquad \sqrt{\frac{0.0478(1-0.0478)}{50{,}000}}\approx0.000954. $$

The estimated difference is $0.0562-0.0478=0.0084$. Because the batches are independent, the estimated standard error of this difference is $$ \sqrt{0.00103^2+0.000954^2}\approx0.00140. $$ The difference is about six standard errors, so ordinary Monte Carlo noise is an unlikely explanation for the contrast. Of these two effect settings, the weaker alternatives give the larger estimated FWER. The finding does not imply that weaker alternatives always maximize FWER; it identifies a parameter dependence that should be examined across the relevant range.

How to Interpret a Simulated Maximum

The comparison is a diagnostic for a proposed procedure. It can reveal that global-null calibration is insufficient, expose sensitivity to a particular effect or nuisance setting, and direct attention to a vulnerable subset of hypotheses. It does not replace a mathematical argument for strong control when such an argument is available. Closed testing, valid intersection tests, or a valid error-spending construction can supply structural guarantees under their stated assumptions; simulation can check implementation and explore settings not resolved by a simple analytic argument.

A sound stress test checks both the estimated values and the simulation itself. Inspect the implemented rejection rules, verify that only true-null rejections enter the FWER count, and confirm that simulated correlations and stopping decisions match the design. Repeat concerning configurations with larger $B$, and report Monte Carlo uncertainty. When comparing many configurations, the largest observed estimate is subject to selection: even if all true FWERs were equal, one estimate could be largest by chance. Treat a small apparent excess over $\alpha$ cautiously and investigate it with additional simulation rather than selecting a winner from noisy estimates.

There are also limits to what a finite grid can establish. It may miss an adverse effect size between grid points, an unexamined nuisance value, or a configuration created by adaptive features of the design. A simulation over selected scenarios cannot prove that the supremum of FWER over all permitted parameters is below $\alpha$. If the estimated maximum exceeds $\alpha$, the proposed procedure has a warning signal requiring investigation; if all estimates are below $\alpha$, that is reassuring evidence over the tested scenarios, not proof of strong control.

Enumerate null patterns.
Include the global null, partial-null sets, and single-true-null sets relevant to the procedure.

Set parameter values.
For each pattern, specify effects on false nulls and plausible nuisance parameters, including values that could alter dependence or interim decisions.

Simulate full trials.
Apply the proposed sequential rules at every look and record the final decisions for each replicate.

Estimate and compare FWER.
Count replicates with any rejection among true nulls, report Monte Carlo uncertainty, and identify the highest estimate among tested configurations.

Investigate and qualify.
Recheck implementation and concerning scenarios; describe the result as a simulation finding, not a proof over untested configurations.

Common Mistakes

Checking only the global null. A well-calibrated global-null result does not establish strong control. Include partial and single-true-null patterns.
Counting any rejection as an error. In a partial-null configuration, rejecting a false null is not a familywise error. Count only trials that reject at least one true null.
Adding armwise error rates. The armwise events can overlap. Estimate the union directly from replicate-level outcomes.
Calling the largest simulated value the true worst case. It is only the largest among tested settings and is itself subject to Monte Carlo variation.
Treating simulation as a proof. A finite number of simulated patterns cannot verify the error bound for every parameter value.

Key takeaway: To stress-test strong control, compare the probability of at least one false rejection across global, partial, and single-true-null configurations, while varying relevant effects and nuisance parameters. The largest credible estimate identifies where to investigate; it does not, by itself, prove or disprove a universal guarantee.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a four-arm trial, three null hypotheses are true and one is false. Which rejections count toward FWER?
In $B=10{,}000$ simulated trials, 540 have at least one true-null rejection. Calculate the estimated FWER and its approximate Monte Carlo standard error.
Two true nulls are rejected in 120 trials, and each is rejected in 400 trials overall. How many trials have at least one false rejection?
A procedure has estimated FWER $0.048$ under the global null and $0.061$ under a single-true-null configuration. What does this comparison support, and what does it not prove?

Solutions. (1) Only rejections of the three true nulls count; rejection of the false null is not a familywise error. (2) The estimate is $540/10{,}000=0.0540$, with estimated standard error $\sqrt{0.0540(0.9460)/10{,}000}\approx0.00226$. (3) By inclusion-exclusion, the union count is $400+400-120=680$ trials. (4) The results flag the single-true-null configuration as more concerning than the global null and suggest global-null calibration alone is inadequate. They do not prove the exact FWER or identify the maximum over untested configurations and parameter values.