Why efficacy-based dropping needs a separate error calculation
An interim efficacy statistic can be used to decide which experimental arms continue. That decision may look like a promising sign: arms with stronger interim evidence survive, while others are dropped. But an interim screen is not the same as a final efficacy conclusion. If the surviving arms are then tested at an ordinary per-arm significance level, the chance of at least one false final rejection depends on how many true-null arms are tested. It can be substantial even when the selection rule is sensible.
The key distinction is between conditional risk, given the set of arms that survived, and unconditional risk, averaged over all possible interim selection outcomes. Efficacy screening changes how often different sets are tested. It does not, by itself, make multiple nominal-level final tests a familywise-error procedure.
For a concrete setup, suppose there are \(K\) experimental arms. Arm \(j\) has treatment effect \(\theta_j\), measured as the mean outcome on arm \(j\) minus the mean outcome on control. The one-sided efficacy hypotheses are \(H_{0j}:\theta_j\leq 0\) and \(H_{aj}:\theta_j>0\). The family is the collection of these \(K\) hypotheses, and strong FWER control means keeping the FWER at or below the target \(\alpha\) for every configuration of true and false null hypotheses.
Assumptions that make the calculations interpretable
The calculations below use a two-stage design. Interim efficacy statistics are calculated from interim data. Only the selected arms proceed, and final p-values are calculated from a new, independent confirmatory data set. Under each true null, each final p-value is valid conditional on the interim selection: for every \(t\) between 0 and 1, \(P(p_j\leq t\mid S)\leq t\). This conditional validity follows, for example, when the final data are independent of the interim data and the final test is correctly specified.
When an exact probability is calculated, we make an additional assumption that the relevant final tests are mutually independent. This simplifies the illustration; in a trial with shared control participants, treatment-control statistics are generally correlated. The union bound used for Bonferroni control does not require independence between final tests. It does require valid final p-values after the selection rule.
An interim p-value threshold is a selection criterion in this setup, not a final rejection rule. The analysis would be different if the interim observations were reused in the final test: then final p-values need not be conditionally valid given selection. The independent-final-data assumption is therefore substantive, not a technical convenience that can be silently dropped.
Deriving the risk for nominal-level tests
Given a selected set \(S\), suppose there are \(m\) true nulls among the survivors. Write \(A_j\) for the event that the final test falsely rejects true null \(j\). Under independent tests with exact null rejection probability \(\alpha\), each event has probability \(\alpha\), and the probability that none occurs is the product of their non-rejection probabilities. Therefore,
If the tests have rejection probabilities \(r_j\leq\alpha\), independence instead gives \(1-\prod_j(1-r_j)\), which is at most \(1-(1-\alpha)^m\). For arbitrary dependence, adding the probabilities of the false-rejection events gives the union bound \(P(\bigcup_j A_j\mid S)\leq m\alpha\). These distinctions matter: a test calibrated at level at most \(\alpha\) need not reject with probability exactly \(\alpha\).
Worked Example: Three Arms With Nominal Final Tests
Suppose three true-null arms survive interim, and each receives an independent final test at level \(\alpha=0.05\). Under the exact-null assumptions, calculate the conditional FWER.
There are \(m=3\) true-null tests. Their individual probabilities of not falsely rejecting are \(1-0.05=0.95\). Thus
The conditional probability of at least one false rejection is about 0.1426, or 14.26%, not 5%. This is an exact result under the stated independence and exact-size assumptions. If the tests are merely known to have rejection probabilities no greater than 0.05, 0.142625 is an upper bound under independence. Without independence, the union bound is \(3(0.05)=0.15\).
How efficacy screening changes unconditional risk
To see how screening changes the unconditional calculation, consider three arms and the following rule: retain arm \(j\) if its interim one-sided p-value is below 0.20. Assume all three null hypotheses are true, the interim p-values are independent and uniform under the null, and the final tests are independent of the interim data and of one another. Each arm is therefore selected with probability 0.20. A selected arm receives a final one-sided test at level 0.05; a dropped arm is not tested.
Worked Example: Unconditional Risk Under an Interim Screen
Let \(K\) be the number of selected arms. Under these assumptions, \(K\) has a binomial distribution with three trials and selection probability 0.20. Calculate the unconditional probability of at least one false final rejection.
For a selected set of size \(k\), the conditional FWER is \(1-(0.95)^k\). The probabilities of selecting \(k=0,1,2,3\) arms are respectively \(0.8^3=0.512\), \(3(0.2)(0.8)^2=0.384\), \(3(0.2)^2(0.8)=0.096\), and \(0.2^3=0.008\). Averaging the conditional risks gives
The unconditional FWER is about 2.970%. It is lower than the 14.26% risk when all three arms are always tested at 0.05 because screening means fewer null arms reach final testing. Yet, conditional on all three surviving, the risk remains 14.26%. Conditional on at least one arm surviving, the risk is \(0.029701/(1-0.512)=0.06086\), or about 6.086%.
The same unconditional calculation can be checked arm by arm. Under the global null, one arm must both pass the screen and falsely reject at final: the probability is \(0.20(0.05)=0.01\). These events are independent across arms here, so the probability of at least one is \(1-(1-0.01)^3=0.029701\), matching the calculation above.
The smaller unconditional number does not establish strong control at 0.05 for every possible efficacy-screening rule or every null configuration. It is a result for the specific assumptions and thresholds just stated. In particular, a low selection probability can lower unconditional risk while leaving the risk high for a particular selected set.
A final rule that protects the family
A simple way to protect strong FWER when no more than \(K\) arms can be tested is to use a per-arm final level of \(\alpha/K\). This is the Bonferroni rule. For any realized selected set, let \(m\) be the number of true-null arms among its survivors. Conditional validity of each final p-value and the union bound imply
Because the bound holds for every selected set and every true-null configuration, averaging over selection also gives unconditional FWER at most \(\alpha\). Independence among tests is not needed. The bound can be conservative, particularly when fewer than \(K\) true-null arms survive, but it is transparent and robust to dependence among the final tests.
Worked Example: Bonferroni Testing After Efficacy Selection
There are three candidate arms, the target familywise level is \(\alpha=0.05\), and two arms survive the interim screen. Their final p-values are 0.012 and 0.021. Apply Bonferroni using the original maximum family size of three.
The final rejection threshold is the exact fraction \(0.05/3\), which is \(0.016666\ldots\). The first p-value satisfies \(0.012<0.05/3\), so its null hypothesis is rejected. The second does not, because \(0.021>0.05/3\). Thus only the first arm is declared efficacious by this rule.
If both survivors were true nulls, the union bound for false rejection would be
If all three arms had survived and all were true nulls, the corresponding bound would be \(3(0.05/3)=0.05\), exactly the target. Using the exact level \(0.05/3\) avoids a rounding-up error: the rounded value 0.01667 gives \(3(0.01667)=0.05001\), which is slightly above 0.05.
A less conservative multiplicity procedure, such as Holm’s step-down method, can also be applied to the final p-values among the survivors when those p-values remain valid conditional on selection. The choice of procedure changes the rejection thresholds, not the need to account for the family. Correlation-aware procedures may gain efficiency when their assumptions and calibration are appropriate; Bonferroni remains a useful reference because its familywise guarantee does not rely on independence.
Diagnostics, limitations, and a common mistake
Before interpreting a false-positive calculation, check the design and the actual rejection rule. Confirm how many arms can reach final testing, whether the stated error rate is per arm or familywise, which null hypotheses are true in the configuration being assessed, and whether the final p-values are valid conditional on the selection. Then check whether an exact joint probability relies on independence or whether only an upper bound is justified.
- Selection is not rejection. Passing an interim efficacy screen does not itself establish efficacy unless the design explicitly defines it as a formal rejection boundary with appropriate error spending.
- Nominal level is not familywise level. Testing several surviving arms at 0.05 does not generally keep the chance of any false rejection at 0.05.
- Conditional and unconditional risks answer different questions. A low unconditional FWER can coexist with a much larger risk after a particular set of arms survives.
- Reused observations require care. If final tests use the same data that selected the arms, selection and final evidence are dependent. The conditional validity assumption used here may fail, so the calculations cannot simply be carried over.
A common mistake is to argue that “only promising arms are tested, so testing each survivor at 0.05 must be safe.” The correction is to specify the relevant event and calculate it. Given \(m\) independent true-null survivors with exact size 0.05, the conditional risk is \(1-(0.95)^m\); with arbitrary dependence, it is at most \(0.05m\). Screening may reduce the unconditional risk by reducing the number tested, but does not make nominal testing of a realized multiple-arm set a 5% familywise procedure.
The main limitation of the numerical examples is their simplifying independence assumption, especially the independence among arm-level tests. Shared controls commonly induce correlation, so exact joint FWER can differ from the independent formula. The Bonferroni bound remains valid under arbitrary dependence if the conditional p-values are valid. In practice, the error calculation must match the planned selection rule, final analysis, and data reuse structure; it cannot be inferred from the word “adaptive” or from the interim p-value threshold alone.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Four true-null arms survive and receive independent exact level-0.05 tests. What is the conditional FWER?
- Why is \(1-(1-\alpha)^m\) not generally an equality when independent tests have rejection probabilities at most \(\alpha\)?
- Three arms are screened independently, each with null selection probability 0.10. Each selected arm receives an independent exact level-0.05 final test. Under the global null, find the unconditional FWER.
- For five possible arms and familywise target \(\alpha=0.05\), two arms survive. What Bonferroni threshold based on the original family size protects strong FWER?
- What assumption in this tutorial can fail if interim observations are reused in the final test?
Solutions. (1) \(1-(0.95)^4=1-0.81450625=0.185494\), about 18.55%. (2) If a test rejects with probability \(r_j<\alpha\), independence gives \(1-\prod_j(1-r_j)\), which can be strictly less than \(1-(1-\alpha)^m\); the latter is an upper bound. (3) Each null arm is selected and then falsely rejected with probability \(0.10(0.05)=0.005\). Across three independent arms, the risk is \(1-(0.995)^3=0.014925\), about 1.493%. (4) The threshold is \(0.05/5=0.01\) per tested arm; using two survivors does not change the stated original-family Bonferroni threshold. (5) Final p-values may no longer be valid conditional on the selected set, because the same observations influence both selection and final testing.