Why Selection Creates Multiplicity
A multiarm trial may compare several treatments with a control, select an arm using interim or final data, and then test that arm. The central error is to treat the selected arm as though it had been named before the data were observed. Selection makes the chance of a false rejection depend on the entire set of comparisons and on the rule used to choose among them. A test that is valid for one fixed comparison does not, by itself, validate a procedure that searches across comparisons and reports only the winner.
Multiplicity is not automatically a problem whenever a trial has several arms. It becomes an error-control problem when the trial can make one or more confirmatory claims from a family of hypotheses and the probability of making any false claim is not controlled for that family. The relevant family includes every hypothesis that could lead to a reported claim under the actual selection and testing rules—not just the hypothesis attached to the arm that ultimately wins.
Selection procedures often focus on one selected treatment, but strong control still asks whether any true null could be rejected. For instance, a procedure might select an effective-looking arm when another arm is truly ineffective. Protecting only against the case in which every treatment is ineffective is weak control, not necessarily strong control. The distinction matters whenever some treatments may work and others may not.
Testing Only the Winner at Nominal Alpha
Suppose \(m\) treatment-control hypotheses are tested, and the rule selects the largest observed test statistic. If each comparison is tested at the same nominal level \(\alpha\), the probability of at least one false rejection is the probability that at least one statistic crosses its rejection threshold when its null is true. Under the complete null, this event is the union of the individual rejection events.
If the tests are independent and each has Type I error probability exactly \(\alpha\), then under the complete null this probability is \(1-(1-\alpha)^m\). Independence is not needed to conclude that the risk can exceed \(\alpha\); it is needed for this particular product formula. Under arbitrary dependence, the union bound gives an upper bound of \(m\alpha\), which can exceed \(\alpha\).
Worked Example: Three Independent Arms at Nominal Alpha
Consider a hypothetical trial with three treatment-control comparisons. Assume the complete null is true, and suppose the three one-sided test statistics are independent standard normal random variables. Each comparison uses a critical value of \(1.960\), corresponding to a one-sided nominal level of approximately \(0.025\). The selection rule chooses the arm with the largest statistic and rejects its null if that statistic exceeds \(1.960\).
Each statistic has probability \(0.025\) of exceeding the threshold under its null. Because the statistics are independent, the probability that none exceeds the threshold is
Therefore, the probability of at least one rejection is
The FWER is about \(7.31\%\), not \(2.5\%\). The winner-only report does not erase the two comparisons that could also have produced a false positive. This calculation assumes independent statistics and a complete null; it illustrates weak-control risk and does not establish strong control for other null configurations.
The same logic applies if the selected arm is chosen using interim data and then tested using those same data: the selection event and the final test are linked. Earlier material on selection and confirmatory testing on reused data distinguishes that setting from a genuinely independent confirmation sample. Here the diagnostic point is that a selected comparison is not automatically a single, fixed test merely because only one arm appears in the final report.
Shared Control: Correlation Is Part of the Multiplicity Calculation
Treatment-control contrasts that reuse the same control observations are generally correlated. A multiplicity calculation that assumes independence without checking the design can therefore use the wrong joint rejection probability. The direction and size of the error depend on the joint distribution; correlation cannot simply be ignored or replaced by an intuition about whether it is “helpful.”
For a transparent derivation, suppose there are two treatment groups and one control group, with independent normally distributed observations in the three groups and common known variance \(\sigma^2\). Let each group have \(n\) participants. Write \(D_i=\bar{X}_i-\bar{X}_0\) for the estimated treatment-minus-control difference for arm \(i\). The treatment means are independent of one another and of the control mean. Thus,
The shared control contributes the positive covariance: the covariance of \(-\bar{X}_0\) with itself is \(\operatorname{Var}(\bar{X}_0)\). Standardizing each contrast does not change their correlation, so
With unequal group sizes \(n_i,n_j,n_0\), under the same independent-observation and common-variance assumptions, the corresponding correlation is
This formula describes the correlation of the normal contrasts under the stated model. In practice, estimated variance, stratification, or other design features can affect the joint distribution, so the assumptions behind a proposed correlation calculation must be checked.
Worked Example: Two Arms Sharing a Control
Suppose two hypothetical treatment groups and a control group each have \(n=100\) participants. Outcomes are independent and normally distributed with common known standard deviation \(\sigma=10\) units. Under the complete null, compare each treatment with control using a one-sided threshold of \(1.960\), which has marginal Type I error probability \(0.025\). The contrast correlation is \(0.5\), as derived above.
If the standardized statistics \(Z_1,Z_2\) were independent, the probability of at least one exceedance would be
With the actual correlation \(0.5\), the bivariate standard normal probability that both statistics are at or below \(1.960\) is approximately \(0.95463\). Equivalently, the probability that both exceed \(1.960\) is approximately \(0.00463\), so
Thus the FWER is approximately \(0.0454\), compared with \(0.049375\) under the independence assumption. In this example, treating positively correlated contrasts as independent overstates the union probability. That direction is not a general license to ignore dependence: a different covariance structure or rejection region can lead to a different discrepancy. The correct calculation must reflect the actual design and the full set of opportunities to reject.
Changing the Selection Rule After Interim Review
A prespecified rule defines which data are used for selection, which arms can be selected, and what conditions permit a confirmatory rejection. Its error calibration is tied to that rule. If investigators inspect interim results and then alter the rule—for example, by adding an arm, endpoint, subgroup, or alternative ranking criterion—the original calibration does not automatically cover the newly available paths to a claim.
This does not mean that every adaptation is invalid. A rule may be adaptive and still have controlled error if the adaptation is specified and its possible outcomes are included in the procedure’s calibration. The error is to treat a data-dependent amendment as though it had been fixed in advance, without accounting for the additional opportunities to select favorable results.
Worked Example: Expanding the Candidate Set After Review
Suppose a hypothetical trial initially plans to choose the best of two treatment arms using a prespecified endpoint, then test the selected comparison at a one-sided level of \(0.025\). After reviewing interim data, the team adds two other arm-endpoint combinations to the candidate set and chooses the most favorable of all four. To isolate the multiplicity arithmetic, assume under a simplified complete-null model that the four candidate statistics are independent and each has rejection probability \(0.025\).
If all four are eligible to produce the final claim, the chance that none crosses the nominal threshold is
The chance of at least one false rejection is therefore
The original two-candidate calculation would have been \(1-(0.975)^2=0.049375\) under the same independence assumption. The expanded search nearly doubles that risk in this illustration. Real candidate statistics may be correlated, and an actual analysis must use their joint structure; the simplified calculation shows why the original two-arm calibration cannot simply be carried over after the family changes.
Diagnostics and Safer Comparisons
Before interpreting a treatment-selection result, check the decision process as well as the final test output. Identify every null hypothesis that could lead to a claim, list every data-driven branch that could select one, and verify the dependence assumptions used in the error calculation. The earlier material on verifying strong control across null configurations and simulation diagnostics for selection and multiplicity rules provides the relevant perspective: checking only the complete-null case is not enough to establish strong control.
- Look for a hidden family. Include all arms, endpoints, interim looks, or other candidates that could have generated the reported claim under the actual rule.
- Check the covariance model. Shared observations, especially a common control group, make comparisons dependent. Compare the assumed covariance with the one implied by the randomization and outcome model.
- Check the rule against the protocol. Compare the implemented selection and testing decisions with the rule specified before results were reviewed. A deviation requires fresh justification and calibration.
- Check more than the global null. Strong control concerns every configuration of true and false nulls; a favorable global-null result alone is insufficient.
The main alternatives differ in how they handle the family. A Bonferroni allocation tests each of \(m\) hypotheses at level \(\alpha/m\); by the union bound, its FWER is at most \(\alpha\) under arbitrary dependence, provided each true-null test is valid at its assigned level. A joint calibration can use the known or justified correlation structure and may be less conservative, but depends on correctly specifying that structure. A prespecified hierarchy can limit which hypotheses are eligible for testing at each stage, but its ordering and rules must be fixed and its claims interpreted accordingly. Independent confirmation on data not used for selection avoids reusing the selection data for the confirmatory test, but the confirmatory family and testing plan still need to be defined.
| Approach or error | What it assumes or does | Main caution |
|---|---|---|
| Test only the selected winner at nominal \(\alpha\) | Uses the ordinary single-comparison threshold after searching among arms | Does not generally control FWER for the full family |
| Bonferroni allocation | Uses level \(\alpha/m\) for each of \(m\) candidate hypotheses | Simple and dependence-robust, but can be conservative |
| Joint correlation-based calibration | Uses the joint distribution of the candidate statistics | Incorrect covariance assumptions can misstate error |
| Change the rule after reviewing data | Adds or changes routes by which a claim can be selected | Requires calibration that includes the amended rule |
Common Mistakes
- “Only one arm is tested, so there is no multiplicity.” The correction is to count every arm that could have won and been rejected. Selecting one result after examining several does not make the search disappear.
- “Each test is at level \(\alpha\), so the FWER is \(\alpha\).” Each marginal Type I error probability is at most \(\alpha\); the probability of at least one rejection can be larger. The exact risk depends on the joint distribution and selection rule.
- “The treatment comparisons are independent.” When the same control observations enter multiple contrasts, their covariance is generally nonzero. Derive or justify the joint distribution rather than assuming independence.
- “The change was only operational, so the old threshold still applies.” A change that affects which result can be selected or claimed changes the procedure’s possible rejection events. It must be covered by the error calibration.
- “Passing the global-null check proves strong control.” Strong control requires the FWER bound in every configuration of true and false null hypotheses.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Four independent one-sided tests each have Type I error probability \(0.01\) under the complete null. What is the probability of at least one false rejection if all four can produce a claim?
- In a shared-control design, why can the contrasts \(D_1=\bar{X}_1-\bar{X}_0\) and \(D_2=\bar{X}_2-\bar{X}_0\) have positive covariance?
- A nominal test is conservative and has actual Type I error probability \(0.018\) at a stated level of \(0.025\). Is it correct to say its Type I error probability is exactly \(0.025\)?
- After interim review, a team adds a treatment arm to the set eligible for selection but keeps the original winner-only threshold. What must be checked before claiming error control?
Solutions. (1) By independence, the probability that all four tests avoid rejection is \((1-0.01)^4=0.99^4=0.96059601\). Thus the probability of at least one false rejection is \(1-0.96059601=0.03940399\), or about \(3.94\%\). (2) Both contrasts contain the term \(-\bar{X}_0\). Under independent groups, their covariance includes \(\operatorname{Var}(\bar{X}_0)>0\), so the shared control induces positive covariance. (3) No. Its Type I error probability is \(0.018\), which is at most the nominal level \(0.025\); equality is not required. (4) Check that the full amended candidate family, its selection rule, the dependence among its statistics, and all configurations of true and false nulls are covered by the error calibration. The original threshold is not justified merely because it was specified for the smaller candidate set.