Why Simulate a Selection Rule?
A treatment-selection procedure is a complete sequence of decisions: trial data are generated, one or more arms may be selected or dropped, and a prespecified testing rule is applied to the claims that remain. Its operating characteristics depend on that whole sequence. Simulating only the final test, or treating selected arms as though they had been chosen in advance, can therefore give misleading estimates of error and power.
Simulation makes it possible to examine the procedure under specified treatment-effect scenarios. It does not replace a proof of error control. Instead, it provides numerical evidence about how the implemented procedure behaves, exposes coding or design problems, and helps assess whether estimated operating characteristics are precise enough to be informative. The earlier material on strong control across null configurations establishes the importance of examining different configurations of true and false null hypotheses; here the emphasis is on designing and interpreting those simulations.
Specify the Procedure and the Scenarios
Let \(K\) be the number of experimental treatments and let \(\delta_j\) denote the treatment effect for treatment \(j\), for \(j=1,\ldots,K\), on the endpoint’s prespecified scale. A scenario is one chosen vector of effects, together with the other data-generating assumptions. For example, \((\delta_1,\delta_2,\delta_3)=(0,0,0)\) represents a global-null scenario, while \((0,0.3,0.3)\) represents a partial-null scenario if zero is the null effect.
Before simulating, state the planned procedure precisely. It must include the allocation, outcome model, any interim adaptations, the selection or dropping rule, the testing family, and the rule for declaring a rejection. Every simulated replicate must run that same procedure from start to finish. In particular, selection must be performed using the replicate’s simulated data, and the multiplicity rule must be applied as it would be in the trial.
The data-generating model should represent the design being evaluated. A model might specify independent patient outcomes conditional on assigned treatment, with a prespecified distribution and variance in each arm. If treatment contrasts share a control group, their statistics are generally correlated; a simulation must preserve that dependence rather than generating independent contrast statistics. Interim adaptations, if part of the procedure, must use the information available at that interim look, not future outcomes.
An illustrative model used below has three treatment-versus-control statistics \(Z_1,Z_2,Z_3\) that are jointly normal with unit variances, means determined by the scenario, and pairwise correlation \(0.5\). This correlation could arise from equal-sized independent treatment and control groups with a shared control, under a normal model with known variance: each contrast contains the same control mean. The examples suppose the selection rule chooses the arm with the largest statistic and the testing rule uses a prespecified maxT adjustment. The simulation must implement that adjustment as defined for the design; the numerical outputs below are hypothetical illustrations, not results from an actual trial or a run of software.
Selection frequency and power answer different questions. For treatment \(j\), its selection frequency is the probability it is selected. Its marginal power is the probability its null hypothesis is rejected when it is false, counting a treatment that was not selected as not rejected. Disjunctive power is the probability of rejecting at least one false null in the family. State which power measure is being reported: a single number called “power” can otherwise conceal an important difference.
Estimate Operating Characteristics From Replicates
Generate \(M\) independent replicate trials under a single scenario. For replicate \(b\), record whether any true null was rejected, whether each treatment was selected, and whether each false null was rejected. These are indicator outcomes: each is 1 when the event occurs and 0 otherwise. The estimated probability is the average of its \(M\) indicators.
For marginal power, replace the selection indicator with \(I(H_{0j}\text{ is rejected})\), in a scenario where \(H_{0j}\) is false. For disjunctive power, use the indicator that at least one false null is rejected. If only one treatment is selected, the selection frequencies across all treatments should sum to 1, provided the rule always selects an arm. If the rule can select none or several, report that explicitly; frequencies need not sum to 1 in those cases.
Monte Carlo Precision: A Short Justification
For any one event, let \(Y_b\) be its indicator in replicate \(b\), and let its true probability under the scenario be \(\theta\). If replicates are independent and identically generated, then \(Y_b\) follows a Bernoulli distribution with variance \(\theta(1-\theta)\). The simulation estimate is the sample mean of these indicators. Its variance is therefore the variance of one indicator divided by \(M\):
Replacing the unknown \(\theta\) by the estimate \(\hat{\theta}\) gives the estimated Monte Carlo standard error, \(SE=\sqrt{\hat{\theta}(1-\hat{\theta})/M}\). This standard error describes simulation noise, not uncertainty from estimating the trial’s treatment effects from real data. When the estimate is based on enough events, a rough 95% Monte Carlo interval is \(\hat{\theta}\pm1.96SE\). Near zero or one, that approximation can be poor; use a binomial interval such as a Wilson or exact interval, and report the number of simulated events as well as the estimate.
Worked Example: Global-Null FWER
Suppose all three null hypotheses are true. In each replicate, three correlated statistics are generated under the specified global-null model, the largest statistic determines the selected arm, and the prespecified maxT testing rule is applied. In \(M=20{,}000\) hypothetical replicates, at least one true null is rejected in 472 trials. Because every null is true, every rejection in this scenario is false.
Estimate. The estimated FWER is \(472/20{,}000=0.02360\), or 2.36%.
Monte Carlo uncertainty. The estimated standard error is \(SE=\sqrt{0.02360(1-0.02360)/20{,}000}=\sqrt{0.000001152}=0.001073\). The approximate 95% Monte Carlo interval is \(0.02360\pm1.96(0.001073)\), or approximately \(0.02150\) to \(0.02570\).
The estimate is close to a nominal \(\alpha=0.025\), and the approximate interval includes 0.025. This simulation alone does not establish that FWER is controlled at 0.025: it estimates performance under this scenario with finite Monte Carlo precision. Other relevant null configurations and checks of the implementation are still needed.
Worked Example: Partial-Null FWER and Selection Frequency
Now suppose treatment 1 has zero effect and treatments 2 and 3 have positive effects. Thus, only \(H_{01}\) is true. The procedure selects exactly one arm and applies its prespecified multiplicity rule. Across 12,000 hypothetical replicates, treatment 1 is selected in 2,160, treatment 2 in 5,640, and treatment 3 in 4,200. A false rejection occurs in 150 replicates, each involving the true null \(H_{01}\).
Selection frequencies. The estimated frequencies are \(2{,}160/12{,}000=0.1800\) for treatment 1, \(5{,}640/12{,}000=0.4700\) for treatment 2, and \(4{,}200/12{,}000=0.3500\) for treatment 3. They sum to 1 because one arm is always selected.
FWER. There is only one true null in this scenario, so FWER is \(150/12{,}000=0.01250\). Its estimated standard error is \(\sqrt{0.01250(1-0.01250)/12{,}000}=0.001014\), giving an approximate 95% Monte Carlo interval of \(0.01051\) to \(0.01449\).
The selection frequency of treatment 1 is not its false-rejection probability: selection alone is not a rejection. The FWER calculation counts only replicates in which the true null is actually rejected. This distinction is essential when a selection rule often chooses an ineffective arm but the testing rule rarely makes a false claim about it.
Worked Example: Marginal Power and Conditional Rejection
Consider a scenario in which treatment 2 is the only treatment with a nonzero beneficial effect. Suppose 10,000 hypothetical replicates are generated under the specified model and run through the complete selection-and-testing procedure. Treatment 2 is selected in 7,100 replicates, and its null hypothesis is rejected in 6,200.
Marginal power. Counting nonselection as no rejection, estimated marginal power for treatment 2 is \(6{,}200/10{,}000=0.6200\), or 62.0%.
Conditional rejection among selections. Since treatment 2 cannot be rejected unless it is selected, its estimated rejection frequency conditional on selection is \(6{,}200/7{,}100=0.8732\), or 87.32%. This conditional quantity answers a different question and should not be presented as marginal power. The result also shows why reporting selection frequency alongside power is useful: treatment 2 is often selected, and rejection is frequent when it is selected, but the overall probability of rejection is lower.
Diagnostics, Comparisons, and Limitations
A simulation is only as credible as its correspondence to the planned trial and the correctness of its implementation. Useful diagnostics include checking that simulated sample sizes and allocations match the design, that outcome summaries resemble the specified model, and that adaptation uses only available information. Check that the selection and test rules are applied in every replicate, including ties, dropped arms, and replicates with no eligible claim. Review code paths where the number of selected arms or tested hypotheses changes.
Also check the replicate-level bookkeeping. Under a global null, every rejection is false. Under a partial null, only rejections of true null hypotheses count toward FWER. Confirm that selection counts agree with the selection rule and that power indicators count a nonselected treatment as not rejected. Very few events can make a probability estimate imprecise even when \(M\) seems large; the Monte Carlo standard error or interval reveals this. Repeating the simulation with an independent random-number stream can flag instability, though agreement across runs does not prove the code is correct.
Simulation complements, rather than displaces, analytical arguments for multiplicity control. An analytical result can establish control under stated assumptions, while simulation can check numerical performance under selected scenarios and test a particular implementation. Where a procedure’s operating characteristics can be computed exactly, exact calculation may avoid Monte Carlo error. Simulation is more flexible for complex adaptation, but it only answers the scenarios and model assumptions actually simulated; it cannot establish performance for every possible configuration merely by examining a few cases.
The main limitation is model dependence. If the generated outcomes, dependence among contrasts, missingness, or adaptation differ materially from the trial conditions, the estimated operating characteristics may not describe the intended design. Prespecify scenarios that are scientifically and operationally relevant, including configurations with true and false nulls. Keep the simulation question narrow enough to interpret: a single average across scenarios can hide unacceptable behaviour in one important configuration.
Write down each scenario, the data-generating assumptions, and the complete selection-and-testing procedure.
Generate independent replicate trials while preserving the dependencies and adaptation specified by the design.
For each replicate, store false rejections, selected treatments, and rejections of false nulls.
Estimate each operating characteristic, quantify Monte Carlo precision, and verify that the procedure and bookkeeping match the design.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In a scenario with two true null hypotheses, 84 of 8,000 replicates have at least one of those nulls rejected. Estimate FWER.
- In a simulation, treatment 1 is selected in 3,000 of 10,000 replicates and its false null is rejected in 2,400. What are its marginal power and its rejection frequency conditional on selection?
- Why must a simulation preserve the dependence between treatment-control contrasts that share a control group?
- What does an approximate 95% Monte Carlo interval describe, and what does it not describe?
Solutions. 1. FWER is \(84/8{,}000=0.01050\), or 1.05%, because the event counted is at least one false rejection among the true nulls. 2. Marginal power is \(2{,}400/10{,}000=0.2400\), or 24%; conditional rejection given selection is \(2{,}400/3{,}000=0.8000\), or 80%. 3. Shared control data induce correlated contrasts; generating independent statistics would simulate a different procedure and could misstate selection and testing behaviour. 4. The interval describes Monte Carlo sampling uncertainty in the estimated probability from a finite number of simulated replicates. It does not quantify uncertainty about the real trial’s effects or protect against an incorrect data-generating model or coding error.