Why the Family Must Be Defined Before Selection
A multi-stage trial may begin with several treatment arms, retain only some after an interim analysis, and continue testing those arms with additional data. The final report may contain just one or two treatment claims, but that does not make the statistical family just those claims: which arms reach the final stage is itself determined by the trial data. To assess the chance of a false conclusion, specify the family of confirmatory hypotheses and the full decision rule before looking at the results.
For \(K\) candidate treatment arms compared with a control, let \(H_j\) denote the null hypothesis for arm \(j\), such as no benefit over control, for \(j=1,\ldots,K\). The associated alternative \(H_{a,j}\) states the treatment benefit being tested. The family is the set of all \(K\) planned confirmatory hypotheses, including those for arms that may later be dropped. If an unselected arm cannot be rejected by the design, it contributes no rejection on that path, but its hypothesis remains part of the original family.
Control only when every null hypothesis is true is called weak control. It can be insufficient: an adaptive rule might behave acceptably when all arms are ineffective but have an elevated chance of falsely declaring a particular ineffective arm beneficial when another arm truly works. Strong control covers both situations. The relevant probability is over all random data and design decisions, including interim selection and stopping.
Selection Paths and the Overall Error Probability
Represent the possible outcomes of interim decisions by paths \(s\). A path records which arms were selected, which stages were reached, and any stopping decisions. Let \(S=s\) mean that the trial followed path \(s\), and let \(R_j\) mean that the procedure ultimately rejects \(H_j\). The false-rejection event on path \(s\) is that at least one true null among the tested arms on that path is rejected.
The equality partitions the overall error event according to the mutually exclusive and exhaustive paths. It shows why selection cannot simply be ignored: the probability of a false rejection depends both on how often a path occurs and on the false-rejection probability after that path is reached. A conventional test that is valid marginally under \(H_j\) need not remain valid conditional on having selected arm \(j\).
One sufficient route to strong control is to ensure that, on every possible path, the conditional probability of at least one false rejection is at most \(\alpha\). Taking the weighted average over paths then gives overall FWER at most \(\alpha\). A less restrictive route is to bound each path's conditional error by \(q_s\) and verify that \(\sum_s P(S=s)q_s\leq\alpha\) for every configuration of true nulls. In either approach, the conditional bounds must actually hold under the design's data-generating assumptions.
The bound on a path can be established using a valid conditional familywise procedure, such as a suitable Bonferroni allocation. Alternatively, the whole adaptive rule can be calibrated using the joint distribution of its statistics, or tested through an appropriate closed-testing procedure. These approaches answer the same central question—whether the probability of any false claim is controlled—but rely on different distributional information and design requirements.
Justifying the Pathwise Bound
Here is the short argument behind the pathwise result. Write \(E_s\) for the event that path \(s\) occurs and that at least one true null is rejected on that path. Since the paths partition the trial outcomes, the overall false-rejection event is the disjoint union of the \(E_s\). By the conditional probability identity, \(P(E_s)=P(S=s)P(E_s\mid S=s)\). Substituting the pathwise bound and summing gives the result:
For a pathwise Bonferroni argument, suppose \(J(s)\) is the set of arms tested on path \(s\), and the conditional probability of rejecting each true \(H_j\), given \(S=s\), is at most \(a_j(s)\). The union bound gives \(P(\text{any true rejection}\mid S=s)\leq\sum_{j\in I_0\cap J(s)}a_j(s)\). Thus it is sufficient for the sum of the allocations across the tested hypotheses on every path to be at most \(\alpha\). This remains a valid bound even when the tests are dependent. Crucially, the individual conditional probability bounds must be justified; merely assigning nominal levels does not establish them.
Worked Example: Selecting One Arm for Independent Validation
Hypothetical design. An interim sample ranks three treatment arms, and the trial selects the arm with the largest interim score. A new, independent validation sample is then used for one one-sided test at level \(0.05\). Suppose the test is valid for each arm's null hypothesis and its validation data are independent of the interim data.
Conditional on any selected arm, the validation data still follow their planned distribution. Under that arm's null, its validation p-value is uniform under the null (or super-uniform if the test is conservative). Therefore, if the selected arm's null is true, its conditional probability of rejection is at most \(0.05\). If the selected arm's null is false, its rejection does not count as a false rejection.
On every path there is at most one tested arm. The conditional probability of any false rejection is consequently at most \(0.05\), and averaging over the selection paths gives FWER at most \(0.05\) for any configuration of true nulls. When all three nulls are true and the validation test has exact level \(0.05\), the FWER is exactly \(0.05\): whichever arm is selected, its independent validation test rejects with probability \(0.05\). The independence of the validation data—not the act of selecting one arm by itself—is what justifies the conditional calculation.
Worked Example: Retaining Two Arms and Allocating Error
Hypothetical design. An interim analysis selects two of four candidate arms. Each selected arm is tested in an independent validation sample, using a one-sided test at level \(0.025\). Assume each validation test is valid under its corresponding null and the validation outcomes are independent of the interim selection.
For any path, at most two hypotheses are tested. For each true null that is tested, the conditional rejection probability is at most \(0.025\). By the union bound, the conditional probability of at least one false rejection is at most \(0.025+0.025=0.050\). The same bound holds if only one of the selected hypotheses is true, since then there is only one term in the bound. Averaging across paths gives strong FWER control at \(0.05\).
If all four nulls are true and the two validation tests are independent and exact, the conditional probability of at least one false rejection is \(1-(1-0.025)^2=1-0.975^2=0.049375\), or about \(0.0494\). That value is below the Bonferroni upper bound \(0.05\). The calculation applies on each path because, conditional on the interim data, the two validation tests remain independent and each has level \(0.025\). If validation tests are dependent, the union bound still gives the \(0.05\) guarantee, although the exact expression \(1-0.975^2\) may no longer apply.
Continued Testing and Error Allocation Across Stages
A trial may also permit confirmatory rejection at an interim stage and continue testing if no interim rejection occurs. In this case, the error event includes false rejections at either stage. A transparent sufficient strategy is to allocate a total error budget across stages, while also controlling multiplicity across arms within each stage. The later-stage guarantee must account for the interim history; it cannot be assumed just because a test has its advertised marginal level.
Worked Example: Combining Interim and Validation Error Budgets
Hypothetical design. Three arms are tested at an interim stage. A false interim rejection is controlled by testing each arm at level \(0.01/3\), so the union bound makes the probability of any false interim rejection at most \(0.01\). If the trial continues, it selects one arm and tests it in an independent validation sample at level \(0.04\). The validation test is valid conditional on all interim data.
The probability of at least one false rejection anywhere in the trial is no greater than the probability of any false interim rejection plus the probability of a false validation rejection. The first is at most \(0.01\). Conditional on the interim data, the second is at most \(0.04\) if the selected arm's null is true and is zero as a false-rejection event if that null is false. Therefore the validation contribution is at most \(0.04\), and the total is bounded by \(0.01+0.04=0.05\).
This calculation allows the interim data to select the arm because the validation guarantee holds conditional on those data. It would not justify the same \(0.04\) validation contribution if the final test reused interim observations and its nominal level were valid only marginally. When a design uses data-dependent conditional testing or another more intricate continuation rule, the path-specific conditional error must be derived for that rule rather than presumed.
When Nominal Testing Fails After Selection
Selection based on a statistic changes the distribution of the statistic for the chosen arm. A simple calculation shows the size of the issue. Suppose three interim statistics \(Z_1,Z_2,Z_3\) are independent and each follows \(N(0,1)\) under its null. Select the largest statistic and declare a false success if it exceeds the one-sided \(0.05\) critical value, \(1.645\). Each statistic by itself has a \(0.05\) chance of exceeding that value, but the selected maximum is not a single prespecified statistic.
Worked Example: The Largest Interim Statistic Is Not a Single 5% Test
Hypothetical calculation. Assume all three null hypotheses are true and the three interim statistics are independent standard normal variables. The probability that a particular statistic does not exceed \(1.645\) is approximately \(0.95\). Independence implies that the probability none of the three exceeds the threshold is approximately \(0.95^3=0.8574\). Thus the probability that the selected maximum exceeds the threshold is \(1-0.95^3=1-0.8574=0.1426\).
A nominal \(0.05\) threshold applied to each arm has produced an overall false-positive probability of about \(0.1426\) when the largest statistic is selected. This calculation concerns a rule that uses the interim statistics themselves to make a rejection; it is not a valid independent confirmatory test. If those statistics also enter a later-stage test, the full joint rule must be calibrated, or the continuation test must have a justified conditional error guarantee. Treating the selected statistic as if it had been chosen before the data ignores the selection event.
Diagnostics, Alternatives, and Limitations
Before interpreting a multi-stage design's error claim, check that the hypothesis family and the rejection rules are explicit. In particular, list all treatment claims that could be made, identify which stages can reject each claim, and state how selection and stopping depend on the observed data. Then verify that the probability bounds used at each path are valid for the corresponding true nulls.
| Approach | What it controls or uses | Main requirement |
|---|---|---|
| Pathwise conditional control | Bounds false-rejection risk after each selection and stopping path | Conditional tests or bounds must be valid given the path |
| Stagewise alpha allocation | Budgets error for interim and later-stage rejection events | Each stage's contribution must account for prior data and decisions |
| Joint-distribution calibration | Calibrates the complete adaptive rejection rule | The joint distribution and selection rule must be correctly specified |
| Closed testing | Tests intersection hypotheses to support strong control | The required intersection tests and their validity must be established |
Common warning signs include reporting an error rate only under the global null, counting only the arms that survived selection as the family, and using per-arm nominal p-values after data-dependent selection without conditional or joint calibration. Another failure is to add stagewise error budgets when the later-stage bound is not valid given earlier data. Each shortcut can understate the probability of at least one false claim.
Pathwise bounds can be conservative, particularly when the tests are positively correlated or when some paths are rare. Joint-distribution methods can use dependence to gain power, but their guarantees rely on correctly modeling that dependence and the complete adaptive rule. Independent validation can make the conditional argument simpler, but it uses additional data and may be costly. No single strategy eliminates the need to specify the family, the decision paths, and the error guarantee being claimed.
Check Your Understanding
Answer each question using the distinction between the original family and the hypotheses tested on a realized path.
- A trial begins with five candidate treatment hypotheses but selects one for independent validation. What is the confirmatory family, and what condition supports a level-\(0.05\) validation test?
- Two selected null hypotheses are tested at level \(0.03\) each on every path. What FWER bound does the union bound provide on that path?
- Why is control under the global null alone not enough to establish strong FWER control?
- Three independent standard normal statistics are screened under the global null; the largest is called positive if it exceeds \(1.645\). What is the false-positive probability?
Solutions. (1) The family is all five planned confirmatory hypotheses. A level-\(0.05\) validation test supports strong FWER control when it is valid conditional on the interim selection for any selected true null; independent validation data are one way to justify that condition. (2) The pathwise bound is \(0.03+0.03=0.06\), so it does not establish a \(0.05\) guarantee. (3) Under partial-null configurations, some hypotheses are true and others false, and selection may favor the false nulls or alter which true null is tested. Strong control requires the bound for every such configuration. (4) The probability is \(1-0.95^3=0.1426\), approximately, because none exceeds the threshold with probability \(0.95^3\).