Assessing Multiplicity in Multi-Arm Trials

Why multiplicity matters in a multi-arm trial

A multi-arm trial can compare several doses with one control group. Each planned comparison addresses a scientifically relevant question, but making several tests creates several opportunities for a false positive. The relevant error rate is often the familywise error rate (FWER): the probability of making at least one false rejection among a specified family of hypotheses.

The comparisons are not independent when they reuse the same control participants. Consequently, an adjustment designed for independent tests may be unnecessarily conservative, while treating each comparison as if it were the only test can inflate the chance of a false positive. Simulation lets the trial team evaluate the exact planned contrasts, analysis, and decision rule under explicit assumptions.

For a dose-ranging trial, the family might consist of the planned comparisons of each active dose with control. Defining the family requires specifying which doses, endpoints, and decision opportunities count as part of it. Here the focus is a fixed set of dose-versus-control contrasts on one endpoint, evaluated at one planned analysis. The simulated FWER is meaningful only for the family and decision rule actually specified.

Shared controls induce correlated contrasts

Consider a continuous endpoint for which larger values are better. Let \(\bar{Y}_j\) denote the sample mean in dose group \(j\), and let \(\bar{Y}_0\) denote the control mean. The planned contrast for dose \(j\) is \(\hat{\Delta}_j=\bar{Y}_j-\bar{Y}_0\). Suppose there are two dose groups, equal sample size \(n\) in each group, and independent normally distributed observations with common variance \(\sigma^2\). The contrast estimates share \(\bar{Y}_0\).

The covariance follows because the two dose-group means are independent of one another and of the control mean, while both contrasts contain \(-\bar{Y}_0\). Their shared control contribution to covariance is therefore \(\operatorname{Var}(\bar{Y}_0)=\sigma^2/n\). The resulting positive correlation changes the distribution of the largest test statistic across the family.

With unequal group sizes, the corresponding covariance is \(\sigma^2/n_0\), where \(n_0\) is the control sample size, and the contrast variances are \(\sigma^2(1/n_j+1/n_0)\). With more doses, each pair of dose-versus-control contrasts still shares the control mean. Other planned contrasts may have a different covariance structure, so their actual definitions should be carried into the simulation.

A simulation workflow for error rates and selection

Before running simulations, specify the design, endpoint model, planned contrasts, analysis method, multiplicity procedure, and trial decision rule. Under each scenario, generate trial data, apply the same analysis that would be used in the trial, and record the decisions. A replicate is one simulated trial; the number of replicates is not the trial sample size.

If \(R\) independent replicates yield \(K\) occurrences of an event, its estimated probability is \(\hat{q}=K/R\). The approximate Monte Carlo standard error is \(\sqrt{\hat{q}(1-\hat{q})/R}\). This describes simulation precision, not uncertainty about the clinical treatment effect.

Adjustment choices and their interpretation

A simple Bonferroni procedure tests each of \(m\) hypotheses at \(\alpha/m\), or equivalently compares each valid p-value with \(\alpha/m\). It controls the FWER at no more than \(\alpha\) without requiring independence. Holm’s step-down procedure is another general method: order the p-values from smallest to largest and compare them sequentially with \(\alpha/m,\alpha/(m-1),\ldots,\alpha\), stopping at the first failure to reject. These procedures are convenient when a general dependence-robust rule is desired.

For several comparisons with a common control, a Dunnett procedure uses the joint distribution of the dose-versus-control statistics to obtain a familywise critical value or adjusted p-values. Under a normal, equal-variance model with a pooled variance estimate, this is a correlated multivariate \(t\) distribution with the relevant degrees of freedom. A simulation can estimate the critical value from the maximum test statistic under the null. When the analysis has features not covered by the standard Dunnett assumptions, simulate the actual planned analysis rather than borrowing a critical value from a different model.

A calibration run can estimate the critical value \(c\) such that the probability that the largest test statistic exceeds \(c\) is \(\alpha\) under the global null. Use a separate validation run to estimate FWER at that value. Separating calibration from validation avoids reporting performance evaluated on the same simulated replicates used to tune the cutoff.

Estimating the chance of selecting an effective dose

Error control and dose-selection probability answer different questions. FWER is evaluated by asking how often the procedure rejects at least one true null. The probability of selecting an effective dose depends on the assumed treatment effects, the definition of an effective dose, and the selection rule. A trial can control FWER while having a low probability of selecting an effective dose if the sample size is small or effects are modest.

Define the rule before simulating. For example, a rule might select the dose with the largest estimated benefit among doses that pass a Dunnett-adjusted significance threshold, and select no dose if none passes. Define “effective” in advance, such as a true mean improvement greater than or equal to a specified clinically meaningful threshold. Then estimate the probability that the rule selects a dose meeting that definition. Also consider recording the probability of selecting no dose and the probability of selecting an ineffective dose; these outcomes help explain the overall selection performance.

Diagnostics, limitations, and common mistakes

Check that the implementation reproduces the allocation, shared-control structure, endpoint model, missing-data assumptions, contrast direction, and analysis specified for the trial. Under a global-null scenario, verify that each individual test has approximately its intended marginal type I error before interpreting familywise results. Compare empirical contrast variances and correlations with their theoretical values when those values are available. Monte Carlo standard errors help distinguish an apparent deviation from \(\alpha\) from simulation noise.

A global-null simulation alone does not establish strong FWER control. To assess strong control, simulate partial-null configurations too—for example, one dose truly has no effect while another has an effect—and record rejection of any true null. The test procedure must also be applied exactly as planned in every replicate. Changing the family after seeing which comparisons look promising, or selecting a dose by a rule that was not included in the simulation, makes the reported error and selection probabilities inapplicable.

• Pitfall: treating shared-control tests as independent. This gives the wrong distribution for the maximum statistic. Simulate the common control observations once per replicate and use them in every planned contrast.
• Pitfall: reporting single-test error as familywise error. A nominal 2.5% test does not imply a 2.5% chance of any false rejection across several tests. Count trial-level events in which at least one true null is rejected.
• Pitfall: claiming strong control from the global null alone. The global-null scenario evaluates only the case where every hypothesis is true. Include partial-null scenarios when the claim concerns all configurations.
• Pitfall: confusing FWER with the chance of a correct selection. FWER concerns false rejections; selection probability depends on the assumed effects and the rule for choosing a dose. Report them separately.

Simulation estimates depend on the assumed data-generating model. If normality, common variance, or the planned analysis is unrealistic, results from a simple normal model may not represent the trial. More realistic scenarios can vary these assumptions, but no finite collection of scenarios proves performance under every possible data-generating process. The purpose is to expose the operating characteristics and vulnerabilities of the specified design and rule under clearly stated conditions.