Why Arm Selection Changes a Combination Test
In a multi-arm, multi-stage trial, interim results can determine which treatment arms continue. A natural way to use evidence from both stages is to combine the stagewise p-values for each arm. Fisher’s method is attractive because it gives a simple statistic and a known reference distribution when its inputs are independent uniform p-values under the null. But selection creates a difficulty: an arm that continues because its interim result was promising does not have an ordinary, unselected interim p-value. Its interim evidence has been filtered by the selection rule.
This tutorial focuses on that distinction. The Fisher statistic itself is straightforward; the key question is whether the p-values supplied to it have the distribution required by its reference law. Correcting that issue at the arm level is not, by itself, a complete familywise error procedure. Multiplicity across arms must also be addressed, and strong familywise error control requires valid tests for the relevant true-null configurations.
The requirements are substantive. Each stagewise p-value must be valid for the same null hypothesis, and the stagewise evidence must be independent under that null (or its dependence must be included in the calibration). In a group-sequential trial, stagewise p-values are typically based on non-overlapping increments, so later-stage evidence can be independent of interim data under the null when the design and test construction justify independent increments. An arbitrary p-value calculated from cumulative data at each look does not automatically satisfy this condition.
A selection rule based on interim data does not necessarily destroy the validity of a Fisher test for an arm that was specified in advance and tested regardless of selection. If its unselected \(P_1\) and \(P_2\) are independent uniforms under the null, its unconditional Fisher p-value remains valid. The problem arises when the analysis treats the selected arm’s \(P_1\) as if it were still an unselected uniform p-value, or when the final rejection decision only examines selected arms without accounting for how selection changed the relevant distribution or multiplicity.
Worked Example: An Unselected Fisher Combination Test
Suppose a hypothetical trial tests one prespecified arm at two stages. Under its null, the stagewise p-values are independent uniforms. The observed values are \(P_1=0.20\) and \(P_2=0.05\). Calculate the Fisher statistic and its p-value, then compare it with \(\alpha=0.05\).
The statistic is $$ F=-2\{\log(0.20)+\log(0.05)\} =-2\log(0.010)=9.210. $$
For four degrees of freedom, the upper-tail probability is \(e^{-F/2}(1+F/2)\). Thus $$ p_F=e^{-9.210/2}(1+9.210/2) =0.0100(1+4.605)=0.0561. $$
The Fisher p-value is approximately \(0.0561\), which exceeds \(0.05\). This test does not reject the arm’s null at the 5% level. The conclusion relies on the stated independence and uniformity assumptions; it does not establish that the same reference distribution is valid after selecting the arm using these p-values.
Conditional Calibration After Selection
A direct way to account for selection is to calibrate the interim p-value under the null conditional on the event that caused the arm to continue. Let \(S\) denote that event. If \(P_1\) has a continuous null distribution conditional on \(S\), write its conditional cumulative distribution function as \(G_S(u)=P(P_1\leq u\mid S,H_0)\). Then \(Q_1=G_S(P_1)\) is uniform conditional on \(S\) under the null. If the later-stage p-value \(P_2\) is also uniform under the null and independent of the interim data conditional on selection, Fisher’s statistic formed from \(Q_1\) and \(P_2\) again has a \(\chi^2_4\) reference distribution conditional on \(S\).
This is a probability integral transform applied to the selected interim evidence. The conditional distribution must correspond to the actual selection rule, not a convenient approximation. If selection is based on correlated interim statistics across arms, the distribution of \(P_1\) given that a particular arm was selected can depend on the other arms’ results and on which arms continued. In that case, \(G_S\) must reflect the joint null distribution and the precise selection event. For discrete or non-continuous statistics, a conditional p-value must be constructed conservatively or randomized as appropriate; simply applying a continuous transformation is not generally exact.
Worked Example: Adjusting for a Simple Interim Rule
In a hypothetical trial, an arm continues if its interim p-value is at most \(c=0.40\). Under that arm’s null, \(P_1\) is Uniform\((0,1)\), and the later-stage p-value is independent of interim data. The observed values are \(P_1=0.08\) and \(P_2=0.03\). Calculate a selection-adjusted Fisher p-value.
Conditional on continuation, \(P_1\) is Uniform\((0,0.40)\). Its conditional distribution function at \(u\), for \(0\leq u\leq0.40\), is \(G_S(u)=u/0.40\). Therefore the transformed interim value is $$ Q_1=G_S(0.08)=0.08/0.40=0.20. $$
By construction, \(Q_1\) is Uniform\((0,1)\) conditional on continuation under the null. Given the assumed independence of \(P_2\), the adjusted combination statistic is $$ F_S=-2\{\log(0.20)+\log(0.03)\} =-2\log(0.006)=10.232. $$
Its conditional Fisher p-value is $$ p_{F,S}=e^{-10.232/2}(1+10.232/2) \approx 0.00600(6.116)=0.0367. $$
The arm-level conditional test rejects at the 5% level because \(0.0367<0.05\). This calculation is exact for the stated simple rule and assumptions. It should not be carried over unchanged to selection by ranking arms or to a rule involving correlated interim statistics: those rules generally give a different conditional distribution.
Multiplicity Among Continuing Arms
Even if each selected arm has a valid conditional combination-test p-value, testing several continuing arms at level \(\alpha\) each can inflate the probability of one or more false rejections. A simple conditional solution is Bonferroni: if \(m\) arms continue and each true-null arm’s combination p-value is valid conditional on the selected set, reject an arm only when its p-value is at most \(\alpha/m\). Conditional on that set, the probability of rejecting at least one true null is at most the sum of the \(m\) individual rejection probabilities, hence at most \(m(\alpha/m)=\alpha\). Averaging over possible selected sets preserves the bound.
The condition “each true-null arm’s p-value is valid conditional on the selected set” is important. Armwise calibration under the complete null alone does not automatically give strong control when some other arms have non-null effects. The selection event may depend on correlated statistics, so the conditional distribution for a true-null arm may change with the effects in other arms. Strong familywise error control requires valid conditional tests for every relevant true-null configuration, or a suitable local-testing and closed-testing construction. The earlier material on conditional error and combination tests provides related tools for maintaining error control after adaptation.
Worked Example: Conditional Bonferroni Threshold for Selected Arms
Suppose three hypothetical arms are assessed at interim, and two continue. Assume the selection-adjusted Fisher p-values are conditionally valid for each true null given the selected set, including under configurations where other arm nulls are false. The values for the two continuing arms are \(0.0367\) and \(0.0120\). Use conditional Bonferroni control at familywise level \(\alpha=0.05\).
There are \(m=2\) continuing arms, so the per-arm threshold is $$ \frac{\alpha}{m}=\frac{0.05}{2}=0.025. $$
For the first arm, \(0.0367>0.025\), so it is not rejected. For the second arm, \(0.0120<0.025\), so it is rejected. By the union bound, conditional on this selected set, the probability of at least one false rejection among the two arms is at most \(2(0.025)=0.05\). The guarantee depends on the assumed conditional validity for true nulls, not just on the fact that two arms were selected.
A Conservative Alternative and Its Trade-Off
Selection-adjusted conditional distributions can be difficult to derive or calibrate. A conservative alternative is to construct an ordinary valid Fisher p-value for every arm in the original family, whether or not that arm continues, and compare each with \(\alpha/J\), where \(J\) is the number of arms in the original family. Under each true arm null, the ordinary Fisher p-value must be valid unconditionally. Any rejection of a selected arm is then also a rejection among the original \(J\) tests. Bonferroni over all \(J\) arms therefore controls familywise error at no more than \(\alpha\), regardless of how the selection rule uses interim results.
This approach can be conservative because it does not use the reduction in the number of arms continuing. It also does not repair an invalid arm-level Fisher p-value: the stagewise p-values still need the appropriate null distributions and dependence assumptions. The conditional approach can yield a less conservative threshold based on the selected set, but it demands a correctly specified conditional calibration and stronger attention to the selection mechanism.
| Approach | What is calibrated | Main requirement or cost |
|---|---|---|
| Original-family Bonferroni | Unselected Fisher p-values for all \(J\) arms | Simple and selection-robust; can be conservative |
| Conditional Fisher plus selected-set Bonferroni | Interim evidence conditional on the selection event, then multiplicity across \(m\) continuing arms | Needs valid conditional distributions for true nulls under the relevant configurations |
| Conditional-error or other combination-test calibration | Future rejection probability subject to the original design’s error constraint | Can accommodate more complex adaptation, but requires its own valid calibration |
Derivation, Diagnostics, and Limitations
The conditional Fisher result follows from two facts. First, if \(P_1\) has continuous conditional distribution function \(G_S\) under the null, then \(Q_1=G_S(P_1)\) is Uniform\((0,1)\) conditional on \(S\). Second, for a Uniform\((0,1)\) random variable \(U\), the transformation \(-2\log(U)\) has a chi-square distribution with two degrees of freedom. If \(Q_1\) and \(P_2\) are conditionally independent uniforms, their transformed values are independent and their sum has a chi-square distribution with four degrees of freedom. Thus $$ -2\{\log(Q_1)+\log(P_2)\}\sim\chi^2_4\mid S,H_0. $$ The distributional argument justifies the reference law; it does not establish that a proposed \(G_S\) or independence assumption is correct.
Several checks help detect problems before applying the procedure:
- Write down the exact selection event. “The two best arms continue” is not the same event as “each arm continues if its p-value is below a fixed cutoff.”
- Verify that stagewise p-values test the same null and use evidence with the required independent-increment structure.
- Check whether selection-adjusted p-values are valid for each true null when other arms have effects, rather than only under the complete null.
- Use simulation under representative null configurations to check conditional type I error, especially when arms share controls or selection uses ranking. Simulation is a diagnostic, not a substitute for a justified calibration.
- Do not use a \(\chi^2_4\) reference distribution if the inputs are dependent or are not conditionally uniform. A different joint calibration is then needed.
Fisher combination gives equal log-scale contribution to each stage when the stagewise p-values are equally weighted. It does not inherently account for unequal information across stages, changing hypotheses, or a complex selection mechanism. Nor does per-arm combination testing automatically provide strong familywise error control. The choice between a conditional selection adjustment, a conservative original-family threshold, and a broader conditional-error design depends on the selection rule, the dependence structure, and the error-control guarantee required.
Check Your Understanding
Answer each question using the assumptions stated in it. Brief solutions follow.
- Under a fixed arm null, \(P_1=0.10\) and \(P_2=0.20\) are independent uniforms. Calculate the Fisher statistic. Is its Fisher p-value below \(0.05\)?
- An arm continues only if \(P_1\leq0.25\). Under the null \(P_1\) is uniform and the observed value is \(0.05\). What is the conditional-uniform interim value?
- Four arms are in the original family. If ordinary arm-level Fisher p-values are valid under each true null, what original-family Bonferroni threshold controls familywise error at \(0.05\), even if interim selection determines which arms are tested?
- Why is dividing \(0.05\) by the number of selected arms not automatically valid when arms were selected using correlated interim statistics?
Solutions. (1) \(F=-2\log(0.10\times0.20)=-2\log(0.02)=7.824\). The upper tail for \(\chi^2_4\) is \(e^{-7.824/2}(1+7.824/2)\approx0.0982\), so it is not below \(0.05\). (2) Given continuation, \(P_1\) is Uniform\((0,0.25)\), so \(Q_1=0.05/0.25=0.20\). (3) The threshold is \(0.05/4=0.0125\). Testing only selected arms at this threshold is no more liberal than testing all four, provided the individual p-values are unconditionally valid for their true nulls. (4) The selection event changes the conditional distribution of each arm’s interim evidence. A selected-set Bonferroni argument needs each true-null p-value to be valid conditional on that selected set, including the dependence and relevant null configuration; correlation and ranking can violate this condition.