Why adjust for all candidates?
In a multiarm trial, investigators may use interim results to retain promising treatments and drop others. The final confirmatory question can then concern only the retained arms. But if the same data helped choose which arms to test, the final set is data-dependent: a treatment may be tested precisely because its interim estimate looked favorable. A multiplicity adjustment can protect against false discoveries across the candidate treatments, provided the p-values being adjusted are valid for the design.
Bonferroni adjustment is useful here because its basic guarantee does not require independent test statistics and does not require the selected set to be fixed in advance. It does require a clearly defined candidate family and valid p-values for the candidate hypotheses. The adjustment protects the probability of one or more false rejections over the full trial, not necessarily that probability conditional on a particular selection outcome.
Candidate hypotheses and the Bonferroni rule
Suppose \(m\) candidate treatments are compared with a common control. Let \(\theta_i\) denote the treatment effect for candidate \(i\), defined so that a positive value favors treatment. For a superiority claim, one possible one-sided hypothesis is \(H_i:\theta_i\leq 0\) against \(H_{a,i}:\theta_i>0\). The same multiplicity logic applies to other prespecified null hypotheses and valid tests.
For each candidate \(i\), let \(p_i\) be a p-value that is valid under its null: for every \(t\) between 0 and 1, \(P(p_i\leq t)\leq t\) when \(H_i\) is true. This is the superuniformity property. The probability is taken under the trial’s actual design, including any adaptation that affects the data used in the test. The Bonferroni rule rejects a candidate null only when its p-value is no greater than \(\alpha/m\). Equivalently, one may report the Bonferroni-adjusted p-value \(\min(1,mp_i)\), and reject when that adjusted value is no greater than \(\alpha\).
Why the bound holds after selection
Let \(I_0\) denote the set of indices for which the null hypothesis is true. Let \(S\) denote the set of candidates selected for final testing; \(S\) may depend on interim results, final results, or both. A false rejection can occur only if at least one true null in the selected set has a sufficiently small p-value. Therefore the event of any false rejection is contained in the union of the events \(p_i\leq\alpha/m\) over all true null candidates. The union bound gives a complete justification:
The first inequality holds because the procedure can falsely reject only a true null in the candidate family. The second is the union bound, which does not assume independence. The third uses validity of each true-null p-value. The final inequality follows because there are at most \(m\) true nulls. Selection does not appear in the bound: even if selection favors unusually small p-values, a false rejection still requires at least one true-null candidate p-value to cross the fixed \(\alpha/m\) threshold.
Worked Example: Selecting Two of Four Candidate Treatments
A hypothetical trial prespecifies four treatment-versus-control hypotheses and uses a one-sided familywise level of \(\alpha=0.05\). An interim rule selects two arms using their interim estimates. At the end of the trial, the selected arms have valid candidate-level p-values of 0.009 and 0.018. Apply Bonferroni control.
The adjustment uses the four candidates, not just the two selected arms:
For the first selected arm, \(0.009\leq0.0125\), so its null is rejected. Its adjusted p-value is \(\min(1,4(0.009))=0.036\). For the second arm, \(0.018>0.0125\), so its null is not rejected; its adjusted p-value is \(4(0.018)=0.072\). Thus the trial makes a familywise-controlled superiority claim for the first arm only. This conclusion assumes that both p-values are valid under their respective nulls for the actual adaptive design.
What “valid after selection” does and does not mean
The result above is an unconditional FWER guarantee: across repetitions of the entire trial, the probability of at least one false rejection is at most \(\alpha\). It does not promise that the error probability is at most \(\alpha\) after conditioning on a particular selection event. For example, among trials in which a particular arm was selected because its interim estimate was high, its conditional distribution may be less favorable than its unconditional null distribution. The earlier material on selection events and their sampling distributions helps explain why these are different probability statements.
Bonferroni adjustment also cannot repair an invalid p-value. A candidate-level p-value must be calibrated for the analysis actually conducted. If an interim decision changes which observations contribute to a test statistic, changes the analysis population, or leads to unaccounted reuse of data, ordinary fixed-design p-values may no longer be superuniform under the null. In that case, the union-bound proof’s validity step does not hold. The design needs a testing procedure whose p-values or test statistics account for the adaptation.
Strong control matters when some treatments truly work and others do not. Validity only under the global null—that all candidate null hypotheses are true—does not by itself establish strong FWER control. The needed condition is p-value validity for each true null under every configuration of effects among the other candidates. Correlation from a shared control does not invalidate Bonferroni: the union bound accommodates any dependence, although strong correlation may make the bound conservative.
Worked Example: Strong Control With a Mixture of True and False Nulls
Suppose a hypothetical trial has five candidate hypotheses and uses \(\alpha=0.05\). Three treatment effects are nonpositive, so their null hypotheses are true; the other two are positive. The selection rule may choose any subset after observing interim data. Assume each true-null candidate p-value is valid under this mixed configuration.
The Bonferroni threshold remains \(0.05/5=0.01\). Regardless of which arms are selected, the probability of a false rejection is bounded by the sum over the three true nulls:
The two false nulls do not contribute to the false-rejection event. The bound is therefore at most 0.03 for this configuration, and at most 0.05 in the worst case when all five nulls are true. This illustrates why the argument gives strong rather than merely global-null control.
Why the selected number is not generally the denominator
A tempting shortcut is to select \(k\) arms and use \(\alpha/k\), treating the selected set as if it had been fixed before looking at the data. If the selection rule depends on the same evidence used for testing, this shortcut can fail: the selected candidates tend to have unusually favorable results. The fixed-family threshold \(\alpha/m\) avoids this problem without modeling the selection distribution.
Worked Example: The Risk of Dividing by the Number Selected
Consider four candidate hypotheses that are all true. For a simple illustration, suppose their valid p-values are independent uniform random variables under the global null. The rule selects the candidate with the smallest p-value and tests that one at level 0.05, using the number selected, \(k=1\), as the denominator.
A false rejection occurs if the smallest of the four p-values is at most 0.05. By independence, the probability that all four exceed 0.05 is \(0.95^4\). Therefore:
The resulting FWER is about 18.55%, not 5%. This example uses independence for transparent arithmetic; the general warning is that adjusting only for the selected count has no universal guarantee when selection is data-dependent. With the prespecified denominator \(m=4\), the threshold would instead be \(0.05/4=0.0125\), and the union bound would guarantee FWER no greater than 0.05.
Practical choices, diagnostics, and limitations
Before applying the adjustment, verify the inferential family and the p-value inputs. The candidate family should include every treatment hypothesis that could become eligible for the confirmatory claim. If the protocol allows selection from a larger pool than the set used in the calculation, the smaller denominator is not justified merely because fewer arms were ultimately tested. If the number of eligible candidates varies, a safe approach is to use a prespecified upper bound \(M\) and threshold \(\alpha/M\), provided every possible tested candidate belongs to that bounded family.
Useful checks include confirming that the selection rule is documented, identifying the exact candidate-level hypotheses, and checking that the test for each candidate respects the interim design and any data reuse. A large discrepancy between nominal and adjusted p-values is not a calculation error: Bonferroni deliberately pays for the full candidate family. Conversely, a result that appears significant only after dividing by the selected count is a warning to revisit the family definition.
Bonferroni is simple, transparent, and robust to dependence, but can be conservative, especially when many hypotheses are strongly correlated. A prespecified single primary hypothesis avoids a multi-hypothesis family only when the scientific question and selection plan genuinely support that interpretation. Procedures that use the joint distribution of correlated test statistics may gain power, but require additional assumptions and calibration. The key comparison is not “conservative versus powerful” alone: any alternative must still provide the error guarantee needed for the actual selection and analysis plan.
Check Your Understanding
Answer each question using the candidate family and error-control target stated. Short solutions follow the questions.
- A trial has six candidate hypotheses, and three are selected using interim outcomes. At \(\alpha=0.05\), what Bonferroni threshold applies to each selected candidate?
- In one sentence, explain why Bonferroni control does not require independent candidate p-values.
- If the selection rule uses interim data, does a fixed-family Bonferroni guarantee imply FWER of at most \(\alpha\) conditional on the selected set? Explain.
- A valid p-value calculation assumes a fixed design, but the trial adapts data collection in a way not accounted for by the test. Does dividing that p-value by the number of candidates restore the Bonferroni guarantee?
- There are four candidates, two true nulls, and a valid Bonferroni threshold of 0.01 per candidate. Give the union-bound upper limit on FWER for this configuration.
Solutions. (1) Use the full candidate count: \(0.05/6\approx0.00833\). (2) The union bound controls the probability of a union of false-rejection events without requiring independence. (3) No; the guarantee is unconditional over repetitions of the whole trial, not necessarily conditional on selection. (4) No; Bonferroni requires valid candidate p-values, and adjustment cannot correct invalid p-values. (5) The bound is \(2(0.01)=0.02\), which is no greater than the familywise target 0.05.