Why Reusing Selection Data Changes a Test
In a multiarm trial, investigators may compare several treatments with control, select the arm with the most favorable interim result, and then test that arm using the same observations. The selected treatment is not statistically equivalent to an arm specified in advance: its observed result helped determine which hypothesis would be tested. A nominal p-value calculated as if that choice had been fixed can therefore understate the chance of a false positive.
The issue is not that the observations become unusable. The issue is that confirmatory inference must account for the rule that selected the treatment. Valid approaches include adjusting for the number and dependence of candidate tests, using a prespecified selection-adjusted procedure, or basing confirmation on an independent validation sample. The adjustment must match the actual selection and testing plan.
For a simple reference model, suppose there are \(m\) candidate treatment-control comparisons with prespecified one-sided hypotheses \(H_{0j}\) of no benefit against \(H_{aj}\) of benefit, for \(j=1,\ldots,m\). Assume each test statistic \(Z_j\) has a standard normal distribution under its null, so the ordinary one-sided p-value is \(1-\Phi(Z_j)\), where \(\Phi\) is the standard normal cumulative distribution function. The selection rule and the direction of testing must be specified; for example, select the largest \(Z_j\) and test for benefit.
This reference model presumes the individual tests are calibrated under their respective null hypotheses. Whether the \(Z_j\) are independent depends on the trial design: comparisons using a shared control are typically correlated. An adjustment that assumes independence is not automatically appropriate for such a design.
How Selection Alters the Null Distribution
Consider two independent candidate statistics under the global null, meaning both treatments have no effect. If the larger statistic is selected, then the selected statistic is \(M=\max(Z_1,Z_2)\). For any threshold \(c\), independence gives
A fixed one-sided test at level \(\alpha\) uses \(c=z_{1-\alpha}\), where \(\Phi(z_{1-\alpha})=1-\alpha\). Applying that same cutoff after choosing the larger of two statistics gives global-null rejection probability \(1-(1-\alpha)^2\), not \(\alpha\). At \(\alpha=0.05\), the probability is \(1-0.95^2=0.0975\). Thus, in this model, the nominal 5% naive test has a 9.75% false-positive probability when the winner is tested.
The Bonferroni argument does not require independence. If each of \(m\) individual p-values is valid under its own null, reject a selected hypothesis only when its p-value is at most \(\alpha/m\). For any configuration of true nulls, the probability of rejecting at least one true null is at most the sum of the rejection probabilities for those true nulls:
where \(\mathcal{H}_0\) is the set of true null hypotheses. The selection rule may choose among the candidates, but it cannot make rejection of a true null more likely than rejection of any member of the full family at the Bonferroni threshold. This is a conservative but broadly applicable guarantee.
Worked Examples
Worked Example: The Global-Null Risk of Choosing the Larger Statistic
Suppose two candidate treatment statistics are independent standard normal variables under the global null. A team selects the larger statistic and rejects if it exceeds the ordinary one-sided 5% critical value, \(z_{0.95}=1.6449\). Find the actual global-null rejection probability.
The selection rule rejects when at least one of the two statistics exceeds 1.6449. By independence,
The global-null false-positive probability is therefore 9.75%, almost twice the nominal 5%. This calculation checks the key assumptions: the tests are independent, each statistic is standard normal under the global null, and selection is the larger statistic. If the trial uses a shared control, independence generally fails and this exact numerical calculation is not justified without a suitable joint distribution.
Worked Example: Testing the Selected Arm With the Same Observations
Consider a hypothetical design with two candidate treatment-control comparisons. For simple arithmetic, suppose each uses a separate control sample, outcomes are normally distributed with known standard deviation \(\sigma=10\) units, and each arm has \(n=100\) participants. These assumptions make the two comparisons independent. The observed treatment-minus-control differences are 3.0 units for Arm A and 1.8 units for Arm B. The prespecified rule selects the larger standardized statistic and tests for benefit.
For either comparison, the standard error under this model is
The statistics are \(Z_A=3.0/(10\sqrt{0.02})\approx2.1213\) and \(Z_B=1.8/(10\sqrt{0.02})\approx1.2728\). Arm A is selected. Its ordinary one-sided p-value is approximately \(1-\Phi(2.1213)=0.0169\). A naive analysis would compare this with 0.05 and report significance, but that comparison ignores the selection among two candidates.
Under the global null and the stated independence assumption, the exact p-value for observing a maximum at least this large is
This global-null adjustment is below 0.05. For strong familywise control, the Bonferroni-adjusted p-value is \(\min(1,2[1-\Phi(3/(10\sqrt{0.02}))])\approx0.0339\), also below 0.05. Thus both the exact global-null calculation and the strong-control Bonferroni procedure reject at the 5% level in this particular example, although their guarantees are not identical. The numerical result is conditional on the hypothetical normal model, known standard deviation, and selection rule stated above; it is not evidence about a real treatment.
Worked Example: Confirming a Selected Arm in Independent Data
Suppose a hypothetical trial uses a screening sample to choose one of three arms. Arm C is selected, and a separately collected validation sample—whose outcomes were not used for selection—produces an estimated treatment-control difference of 4.0 units with a prespecified standard error of 1.5 units. Assume the validation statistic is standard normal under the null and the one-sided direction was specified before validation outcomes were examined.
The validation statistic and p-value are
If exactly one selected arm is tested in this validation sample, and the validation data are independent of all selection data, this p-value is valid for testing the selected arm's null at the prespecified level. Conditional on the screening results and hence on the identity of Arm C, the validation data still follow their null distribution if Arm C truly has no effect. If the validation analysis tests multiple arms, or makes additional data-driven choices using those outcomes, it needs the corresponding multiplicity adjustment. This example assumes the standard error and normal calibration are appropriate; a different outcome model would require its own valid test.
What Makes an Adjustment Valid?
The choice of method depends on what was selected, what data were reused, and the intended error guarantee. The following comparison summarizes common options.
| Approach | What it uses | Key validity condition or limitation |
|---|---|---|
| Naive test of the winner | Ordinary p-value for the selected arm | Generally invalid at its nominal level when the same data selected the arm |
| Bonferroni adjustment | Number of candidate hypotheses and valid individual p-values | Controls familywise error under arbitrary dependence; can be conservative |
| Selection-adjusted or simultaneous procedure | The selection rule and, when needed, the joint distribution of test statistics | Must match the specified rule, covariance structure, and testing family |
| Independent validation sample | Outcomes not used to choose the arm | Requires genuine independence of validation outcomes from selection and a prespecified validation test |
A selection-adjusted procedure can use the joint distribution of the candidate statistics to calibrate the selected result. For example, the maximum-statistic calculation above accounts for the chance that either of two independent null statistics crosses the observed maximum. Other designs may need a different joint distribution or an established multiplicity procedure. The earlier material on adjusted p-values and simultaneous testing provides related tools; the essential requirement here is that the adjustment reflect the actual rule and dependence structure.
A practical analysis should establish these points before interpreting a result:
- Selection rule: State which treatments could be selected and exactly how the winner is identified, including any futility or efficacy criteria that affect the candidate set.
- Test family: Specify which null hypotheses are confirmatory and whether the aim is control of the global-null error or strong familywise error across configurations.
- Dependence: Determine whether comparisons share control observations or otherwise have correlated statistics. Use an adjustment that accounts for this dependence, or a method such as Bonferroni that remains valid without an independence assumption.
- Data separation: If using validation data, verify that its outcomes and analysis choices were not influenced by the screening results beyond the prespecified choice of arm.
- Model calibration: Check that the individual test statistics and p-values have the stated null distributions under the outcome model, sample sizes, and variance assumptions.
Derivation: Why Bonferroni Protects the Selected Test
Suppose there are \(m\) prespecified candidate hypotheses and each individual p-value is valid, meaning \(P(p_j\le t)\le t\) whenever \(H_{0j}\) is true. Only the selected candidate is ultimately reported as tested, but use the rule \(p_{\mathrm{selected}}\le\alpha/m\) for rejection. A false positive can occur only if the selected hypothesis is true and its p-value crosses that threshold. That event is contained within the event that at least one true candidate p-value crosses the threshold. Therefore,
The second inequality is the union bound, which holds whether or not the tests are dependent. The proof also shows why the adjustment must account for all candidates that could have been selected, not merely the number of arms that happened to look promising after selection.
Diagnostics, Failure Modes, and Limitations
The clearest warning sign is a report that gives a nominal p-value for the winning arm but does not explain how that arm was chosen. Other warning signs include applying an independence-based maximum distribution to shared-control statistics without justification, changing the selection rule after seeing results, or describing an exact global-null adjustment as strong control under every null configuration. Each is a mismatch between the calculation and the design.
Naive reuse is not automatically invalid in every imaginable design. If selection is genuinely independent of the test data under consideration, the ordinary test may retain its calibration; independent validation is the standard transparent example. Likewise, some prespecified selection procedures have a valid adjusted reference distribution. But simply using the same observations twice does not establish validity, and a favorable p-value cannot repair a procedure whose error rate was not controlled.
Bonferroni is easy to apply and robust to dependence, but it may sacrifice power, especially with many candidates. More tailored methods can be less conservative, but they require reliable knowledge of the joint distribution and close adherence to the prespecified selection rule. Independent validation avoids reusing screening outcomes for the confirmatory statistic, but needs additional data and can still require multiplicity control if several validation hypotheses are tested.
Common mistake: “The selected arm has an ordinary p-value below 0.05, so its confirmatory test is significant at the 5% level.” This treats the winner as though it had been chosen in advance. The correction is to use a procedure calibrated for the selection process—such as a valid multiplicity adjustment—or to test the selected arm using independent validation data under a prespecified plan.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Two independent one-sided tests are each used at level 0.05, and the larger statistic is selected. What is the global-null probability that the selected statistic exceeds its ordinary 5% critical value?
- There are four valid candidate p-values, and one arm is selected using the same data. What Bonferroni threshold gives strong familywise control at \(\alpha=0.05\)?
- Why does an exact maximum-statistic calculation under the global null not automatically establish strong control under partial-null configurations?
- A selected arm is tested in an independent validation sample, but three arms are all tested in that sample. Is one ordinary 0.05 test automatically sufficient to claim strong familywise control?
Solutions. (1) \(1-0.95^2=0.0975\), or 9.75%. (2) \(0.05/4=0.0125\) for each candidate p-value. (3) The global-null calculation assumes all candidate nulls are true; with some false and some true nulls, the selection and rejection probabilities can differ, so a separate strong-control argument is needed. (4) No. Independence from screening protects against reuse-related selection distortion, but testing three validation hypotheses still requires a multiplicity procedure for strong familywise control, such as Bonferroni at \(0.05/3\) per test.