Why Selection Changes the Error Probability
Suppose a trial compares a control with several candidate treatments. At an interim analysis, the treatment with the largest standardized statistic looks most promising. If that same statistic is then compared with the ordinary cutoff for one prespecified treatment, the analysis ignores that the winning treatment was chosen from several candidates. Even when every treatment has no effect, the largest of several noisy statistics is more likely to exceed the cutoff than any one statistic is.
The calculation below isolates this problem under a simple global null: all candidate treatments are ineffective, and their interim statistics are independent and standard normal. It is a counterexample to treating a data-selected treatment as if it had been specified in advance. It is not a claim that every adaptive trial has inflated error: for example, an appropriately designed test using independent confirmatory data can have a different error behavior.
Deriving the Global-Null Rejection Probability
For a single prespecified treatment, \(P(Z_i>c_\alpha)=\alpha\). After selection, the test rejects precisely when the maximum exceeds the same cutoff. The complement of this event is that every statistic is at or below the cutoff. Independence lets us multiply those probabilities:
The final equality follows from the definition \(c_\alpha=\Phi^{-1}(1-\alpha)\), so \(\Phi(c_\alpha)=1-\alpha\). For \(K>1\) and \(0<\alpha<1\), the result exceeds \(\alpha\): because \(0<1-\alpha<1\), raising it to a power greater than one makes it smaller, and therefore \(1-(1-\alpha)^K>\alpha\). This is the familywise probability of at least one rejection when all the candidate null hypotheses are true. It is not the probability that any particular treatment is effective.
Worked Example: Three Candidate Treatments at a 5% Level
Assume three independent interim statistics, each distributed as \(N(0,1)\) under the global null. Select the largest and reject if it exceeds the one-sided 5% cutoff. Use the exact cutoff \(c_{0.05}=\Phi^{-1}(0.95)\approx1.6448536\), not the rounded value 1.645 in an exact calculation.
For each prespecified candidate, \(P(Z_i\leq c_{0.05})=0.95\) and \(P(Z_i>c_{0.05})=0.05\). The selected statistic exceeds the cutoff if at least one of the three statistics does. Thus,
The global-null rejection probability is therefore \(0.1426\), or about \(14.26\%\), rather than \(5\%\). Each individual statistic still has a 5% upper-tail probability; the increase arises because the rule rejects if any of the three crosses the cutoff. Using \(1.645\) instead gives only an approximation, since its standard-normal upper-tail probability is about \(0.0499849\), not exactly \(0.05\).
How the Inflation Grows with the Number of Candidates
The formula makes the effect of the candidate set explicit. For example, at a nominal one-sided level of \(\alpha=0.05\), testing the selected maximum from ten independent candidates at the ordinary single-test cutoff gives
Thus the global-null rejection probability is about \(40.13\%\). The calculation does not say that the selected treatment has a \(40.13\%\) chance of being effective. It says that, under the stated all-null model, the selection-and-rejection rule rejects at least one candidate about \(40.13\%\) of the time over repeated trials.
Worked Example: Five Candidates and a Nominal 1% Test
Suppose five independent standard-normal interim statistics are screened under the global null. The selected maximum is tested using the ordinary one-sided \(\alpha=0.01\) cutoff, \(c_{0.01}=\Phi^{-1}(0.99)\approx2.32635\). The cutoff gives each candidate a 1% upper-tail probability, so each has probability \(0.99\) of not exceeding it.
The probability that none of the five exceeds the cutoff is \(0.99^5\). Therefore,
Although the nominal level for one prespecified treatment is \(1\%\), the global-null rejection probability after selecting among five is about \(4.901\%\). The relevant unit is the whole selection-and-test rule, not the final test viewed in isolation.
Calibrating a Cutoff for the Selected Maximum
One way to avoid this particular inflation is to choose the cutoff for the maximum rather than reuse a single-test cutoff. Under the independent standard-normal global-null model, a cutoff \(c_K\) has global-null rejection probability \(\alpha\) when
This is an exact calibration for the maximum under the stated assumptions. Another common option is the Bonferroni cutoff \(c_B=\Phi^{-1}(1-\alpha/K)\). It controls the probability of at least one rejection at no more than \(\alpha\), because the probability of a union of events is no greater than the sum of their probabilities. Under independence, the maximum-based cutoff is exact, whereas Bonferroni is generally conservative.
Worked Example: A 5% Cutoff for the Maximum of Three Statistics
For three independent standard-normal statistics, find a cutoff that makes the global-null rejection probability exactly \(0.05\). Solve \(1-\Phi(c_3)^3=0.05\):
With the unrounded cutoff, the probability of no statistic exceeding it is \(\Phi(c_3)^3=0.95\), so the probability that the maximum exceeds it is \(1-0.95=0.05\). By comparison, the ordinary single-test cutoff is approximately \(1.6448536\), which gave a global-null rejection probability of \(0.142625\) in the three-candidate example. The adjusted cutoff is higher because it must account for searching across three statistics before deciding which one to test.
For comparison, the Bonferroni cutoff is \(\Phi^{-1}(1-0.05/3)\approx2.128\). It is slightly more stringent here: the union bound guarantees a global-null probability no greater than \(0.05\), while the maximum-based cutoff is calibrated to give exactly \(0.05\) under independence.
Assumptions, Diagnostics, and Scope
The exact expression \(1-(1-\alpha)^K\) relies on several conditions. The \(K\) statistics must each have the specified standard-normal null distribution, and they must be mutually independent. The selection rule must choose the largest statistic, and rejection must occur when that selected statistic exceeds the same one-sided cutoff. The calculation concerns the global null, where every candidate treatment is ineffective.
These assumptions should be checked against the design rather than inferred from the fact that \(K\) treatments were considered. If candidate statistics are correlated, then the product step in the derivation is not valid; the joint probability \(P(Z_1\leq c,\ldots,Z_K\leq c)\) depends on their joint distribution. For example, perfectly identical statistics have no selection-induced increase: their maximum is just the common statistic. With other dependence structures, the error probability lies between cases that can differ substantially, so the independent formula should not be presented as an exact answer without justification.
The statistic scale also matters. The formula applies when the selection rule actually picks the largest comparable standardized statistic and each candidate has the stated null distribution. Selecting by raw estimated treatment effects when their standard errors differ is not the same rule. Likewise, choosing the largest absolute statistic requires a two-sided formulation; the one-sided cutoff and tail probabilities used here cannot simply be reused.
A useful diagnostic is to write down the complete decision rule and ask whether its rejection event is “at least one candidate statistic crosses the cutoff.” If so, a single-test p-value for the selected statistic does not by itself account for the search. A second diagnostic is to distinguish interim selection data from new confirmatory data. If a treatment is selected using interim information and its effect is then assessed with genuinely independent data using a test that is valid conditional on the selection, this specific maximum-of-the-same-statistics calculation does not describe that procedure. Conversely, reusing selection evidence as though it came from a prespecified choice can produce the inflation shown here.
Common Mistake and Practical Interpretation
Wrong reasoning: “The selected treatment is tested at the 5% level, so the chance of a false rejection is 5%.” This treats the winner as if it had been named before any data were observed. In the three-candidate example, the rule rejects whenever any one of the three statistics exceeds the single-test cutoff, giving \(1-0.95^3=0.142625\), not \(0.05\).
Correction: Evaluate the full data-dependent rule under its null distribution. State how many candidates were considered, how selection was performed, and what assumptions justify the joint probability calculation. Use a cutoff or testing strategy designed for that rule, or use independent confirmatory evidence when appropriate. A reported p-value for the selected statistic that ignores selection does not automatically have its advertised single-test interpretation.
This counterexample is deliberately narrow: it demonstrates global-null error inflation for independent candidate statistics when the selected maximum is judged using a prespecified-treatment cutoff. The formula does not establish error control under partial nulls, arbitrary dependence, or a different adaptive design. Those settings require their own justification rather than an unqualified reuse of this calculation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Under the assumptions of this tutorial, what is the global-null rejection probability for four independent candidates tested at a single-test level of \(0.05\)?
- Why does selecting the largest statistic not change the marginal null distribution of each individual \(Z_i\), yet change the rejection probability of the selected statistic?
- For three independent candidates, write the equation for the exact cutoff giving a global-null rejection probability of \(0.05\). Why is it larger than the ordinary one-test cutoff?
- What step in the derivation fails if the candidate statistics are correlated?
- Would the formula \(1-(1-\alpha)^K\) automatically apply if treatment selection used interim data but the selected treatment were tested on independent new confirmatory data? Explain.
Solutions. (1) \(1-0.95^4=1-0.81450625=0.18549375\), or about \(18.55\%\). (2) Each fixed candidate still has its own standard-normal null distribution, but the selected value is the maximum, which exceeds a threshold whenever at least one candidate exceeds it. (3) Solve \(1-\Phi(c_3)^3=0.05\), giving \(c_3=\Phi^{-1}(0.95^{1/3})\approx2.121\); it is larger because the cutoff must control the chance that any of three statistics crosses it, not just one. (4) Independence was used to write the joint probability of all statistics being below the cutoff as a product of their marginal probabilities. (5) No. The formula models selection and testing using the same candidate statistics; independent confirmatory data require analysis of the conditional validity of that separate testing procedure.