Why selection changes the error probability
An interim decision can be useful: when several candidate treatments are being studied, a design may continue with the arm whose interim result looks most promising. But the selected result is not an ordinary, pre-specified single test. Even when every treatment is ineffective, random variation makes one arm look best. If that winner is then declared successful whenever its statistic passes an ordinary nominal threshold, the chance of a false positive can exceed the nominal significance level.
The key issue is not that a test statistic becomes invalid merely because it is large. It is that the rule has searched across several statistics and then focuses on the largest. The relevant false-positive event is therefore that at least one candidate statistic crosses the threshold, not that one particular pre-named statistic does so.
For clarity, begin with one-sided tests of treatment benefit, with each test statistic oriented so that larger values favor treatment. A nominal level \(\alpha\) test rejects its null when \(Z_i>c_\alpha\), where \(c_\alpha\) is the standard normal upper-tail critical value satisfying \(P(Z_i>c_\alpha)=\alpha\) under the null. The simple calculation below assumes that the statistics used for selection and testing are the same statistics. It is an illustration of the selection problem, not a universal formula for every adaptive design.
The independent-statistics derivation
Suppose there are \(K\) candidate treatments and, under the global null, their interim statistics \(Z_1,\ldots,Z_K\) are independent standard normal random variables. The rule selects \(Z_{\max}=\max(Z_1,\ldots,Z_K)\). Testing only the winner at nominal level \(\alpha\) rejects when \(Z_{\max}>c_\alpha\).
The third line uses independence. The last line uses the fact that each statistic has null rejection probability \(\alpha\). For \(K>1\) and \(0<\alpha<1\), \(1-(1-\alpha)^K>\alpha\). Thus, the rule does not have global-null Type I error \(\alpha\) in this model. The nominal p-value for the winner, calculated as if it had been named in advance, does not account for the search.
Worked Example: Three Independent Candidate Treatments
Suppose three candidate treatments have independent interim statistics, each distributed as \(N(0,1)\) under the global null. The design selects the largest statistic and uses a one-sided nominal level of \(\alpha=0.025\). For this level, \(c_\alpha\) is approximately \(1.960\).
For any one pre-specified treatment, the probability of exceeding \(1.960\) under the null is \(0.025\). For the selected winner, the probability that none of the three statistics exceeds the cutoff is \(\left(1-0.025\right)^3=0.975^3=0.926859\).
Therefore, the probability that the winner exceeds the cutoff is \(1-0.926859=0.073141\), or about \(7.31\%\). The false-positive probability is nearly three times the stated \(2.5\%\) level. This does not mean that every selected result is false; it means that under the stated global-null model, the selection-and-test procedure falsely declares benefit in about \(7.31\%\) of repeated trials.
What changes when treatment statistics are correlated?
Independence is a convenient starting assumption, but treatment comparisons often share a control group. Shared controls make the treatment-versus-control statistics positively correlated. For example, suppose there are \(K\) treatment arms and one control arm, with equal sample sizes \(n\), independent observations across arms, and a common outcome variance \(\sigma^2\). For a continuous outcome, each treatment-minus-control mean difference has variance \(2\sigma^2/n\). Two such differences share the control mean, so their covariance is \(\sigma^2/n\), giving correlation \((\sigma^2/n)/(2\sigma^2/n)=1/2\). The standardization used to form their test statistics leaves this correlation unchanged under these assumptions.
The independent formula \(1-(1-\alpha)^K\) does not apply to these correlated statistics. The exact probability depends on their joint null distribution. However, positive correlation does not in general restore the nominal error rate. For two distinct, non-identical statistics, let \(E_i\) be the event that statistic \(i\) exceeds the same one-sided nominal cutoff. Then \(P(E_1\cup E_2)=P(E_1)+P(E_2\setminus E_1)=\alpha+P(E_2\setminus E_1)\). If the joint distribution allows outcomes where one statistic crosses the cutoff and the other does not, the second term is positive. The chance of at least one crossing is then greater than \(\alpha\). Dependence affects how much the exceedance events overlap; it does not justify treating the selected test as if selection never occurred.
Worked Example: Two Comparisons Sharing a Control
Consider two treatment-versus-control statistics based on independent, normally distributed outcomes with equal sample sizes \(n\) and common variance \(\sigma^2\). Assume both treatment effects are zero. Each contrast estimate has variance \(2\sigma^2/n\). Because both contrasts subtract the same control mean, their covariance is \(\sigma^2/n\), so their correlation is \(1/2\).
At one-sided level \(\alpha=0.025\), each statistic individually has probability \(0.025\) of exceeding \(c_\alpha=1.960\). Let \(E_1\) and \(E_2\) denote the two exceedance events. The selection rule tests whichever statistic is larger, so it rejects exactly when \(E_1\cup E_2\) occurs. Its probability is \(P(E_1\cup E_2)=0.025+P(E_2\setminus E_1)\).
The bivariate normal distribution with correlation \(1/2\) is not perfectly dependent, so there are possible outcomes with \(Z_2>1.960\) and \(Z_1\le1.960\); consequently, \(P(E_2\setminus E_1)>0\). The rejection probability is therefore strictly greater than \(2.5\%\). Its exact value requires the joint normal probability; substituting the independent-statistics formula would be incorrect. This example shows why the dependence structure matters for quantifying inflation, even though selection is enough to identify the source of the problem.
Selection and later testing data
A crucial design distinction is whether the final test reuses information that helped select the treatment. When the same statistics are used for selection and testing, the maximum-statistic calculation directly describes the false-positive event under its assumptions. In many adaptive trials, the later analysis includes interim observations, or the final statistic otherwise depends on the data used to choose the arm. The selected analysis is then not simply a fresh, pre-specified test.
By contrast, selection based on interim data need not inflate the nominal error of a single test if confirmation uses genuinely independent validation data and the final test is valid under its null. The reason is conditional: given which arm the interim data select, the independent validation statistic for that arm still has its null distribution. The following example makes the distinction explicit.
Worked Example: Selection Followed by Independent Validation
Suppose two treatments are ranked using interim data, and one is chosen. Under the global null, assume the later validation statistic for either treatment is standard normal and that the validation data are independent of all interim data. The selected treatment is tested one-sided at \(\alpha=0.025\), using the cutoff \(1.960\).
Write \(S=i\) for the event that interim selection chooses treatment \(i\). By independence, conditional on \(S=i\), the validation statistic for treatment \(i\) still has a standard normal null distribution. Thus, \(P(\text{reject}\mid S=i)=0.025\) for either \(i=1\) or \(i=2\).
Averaging across the possible selections gives \(P(\text{reject})=\sum_{i=1}^{2}P(S=i)P(\text{reject}\mid S=i) =\sum_{i=1}^{2}P(S=i)(0.025)=0.025\), because \(P(S=1)+P(S=2)=1\). In this setting the one selected validation test retains its nominal global-null error rate. This conclusion relies on independent validation data and a valid test; it does not follow merely from calling a later analysis “confirmatory.”
Assumptions, diagnostics, and alternatives
The independent-normal derivation rests on several specific conditions: all candidate null hypotheses are true; each statistic has a standard normal null distribution; the tests are one-sided in the direction of benefit; statistics are independent; and the selected maximum is tested using the same cutoff that would apply to a single pre-specified test. If test statistics are \(t\)-distributed, have unequal variances, are correlated, or use different cutoffs, the displayed formula need not hold. The underlying concern remains: selection can change the probability of rejection, and the appropriate probability depends on the design’s joint sampling distribution.
Before interpreting an unadjusted selected result, check the design and analysis record for the following:
- How many candidates were eligible? Counting only the winner hides the number of opportunities for a chance high statistic.
- What quantity drove selection? Ranking raw estimates, standardized statistics, or a thresholded rule can produce different selection events.
- Which observations enter the test? Reusing interim data differs from testing with independent validation observations.
- How are the contrasts related? A shared control, unequal allocation, or other dependence changes the joint null distribution.
- Was the rule specified in advance? An unrecorded change in ranking or eligibility makes the actual selection process difficult to assess.
A pre-specified single-arm test avoids this particular search over candidate treatment statistics, but it may not answer a design’s treatment-selection objective. When several candidate hypotheses can lead to a claim, a multiplicity-adjusted procedure can control the probability of one or more false rejections; the adjustment must reflect the hypotheses and dependence structure. Independent validation is another option when it is feasible and truly independent. These approaches address different practical constraints, so the nominal level alone is not enough to compare them.
A common mistake is to say, “Only one treatment was tested, so the Type I error is \(\alpha\).” That reasoning counts only the final test and ignores the earlier search that determined which treatment received it. Under the independent three-treatment example, the relevant event is \(\max(Z_1,Z_2,Z_3)>1.960\), not merely that a pre-named statistic exceeds \(1.960\); its probability is \(7.31\%\), not \(2.5\%\). The correction is to evaluate the whole selection-and-testing rule under its null distribution, or to use a design and analysis that provides valid error control.
The result is a warning, not a claim that every selection rule causes the same amount of inflation. If candidate statistics are identical, selection adds no distinct opportunity to cross the threshold; if they are highly correlated but not identical, inflation may be smaller than under independence. If a selected treatment is tested only in independent validation data, the conditional argument above can preserve the nominal level. The limitation of the simple formula is precisely that it assumes independent, identically distributed statistics and a specific maximum rule.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Under the independent global-null model, three one-sided tests each have nominal level \(0.05\). What is the probability that the selected maximum exceeds its single-test cutoff?
- Why can the independent formula not be used unchanged for treatment-versus-control statistics that share a control group?
- Two arms are ranked using interim data, then one is tested using independent validation data. State the condition that lets the selected validation test retain its nominal global-null level.
- A team argues that selecting one winner means the final analysis has only one test and therefore has Type I error \(\alpha\). What part of the design does this argument omit?
Solutions. (1) The probability is \(1-(1-0.05)^3=1-0.95^3=1-0.857375=0.142625\), or about \(14.3\%\). (2) The product step in the derivation uses independence; shared-control statistics are correlated, so the joint probability that none exceeds the cutoff is not generally the product of their individual probabilities. (3) Conditional on which arm is selected, its validation statistic must still have the stated null distribution and be tested validly; independence of validation data from interim selection supplies that condition in the example. (4) The argument omits the search across candidate statistics that determined which treatment was tested. Under the global null, the relevant event is that the selection-and-testing rule produces a rejection, not merely that a single treatment was named at the end.