Why Selection Changes the Error-Control Question
At an interim analysis, biomarker-specific results may suggest continuing in one population, such as patients with a particular marker, rather than in a broader population. That choice can be clinically and operationally sensible. The statistical question is whether the subsequent confirmatory claim is protected against the fact that the population was chosen using data.
The key distinction is between choosing a population and choosing a population and then treating the evidence that drove the choice as if the target had been fixed in advance. Selection does not automatically inflate type I error. For example, a valid test in a genuinely independent confirmation sample can be applied to the selected population at its prespecified level. But selecting the most favorable interim result and testing it at an unadjusted level using that same evidence can make the chance of at least one false claim exceed the nominal level.
Let the prespecified candidate populations be indexed by \(j=1,\ldots,K\), with null hypotheses \(H_{0j}\). A selection rule based on interim data \(I\) chooses \(S(I)\), a candidate index. A confirmatory claim may then concern \(H_{0,S(I)}\). If the selection rule and the evidence used for testing are statistically dependent, the selected test is not generally distributed as though its index had been fixed before observing the data.
A Simple Derivation of Selection-Induced Inflation
Suppose, for illustration, there are two disjoint biomarker strata. Under the global null, each stratum-specific test statistic is standard normal, and the two statistics \(Z_1\) and \(Z_2\) are independent. The trial selects the stratum with the larger statistic, then declares a positive result if that selected statistic exceeds the one-sided 5% critical value, \(z_{0.95}=1.645\).
For a single prespecified test, the probability of exceeding 1.645 under its null is 0.05. After selecting the larger of two independent statistics, however, a false positive occurs if either statistic exceeds 1.645. Thus:
The probability is 9.75%, not 5%. The selection has created two opportunities for a favorable noise fluctuation, while the final decision still uses the cutoff for one test. With \(K\) independent null statistics and the same rule, the corresponding probability is \(1-(1-\alpha)^K\). If the statistics are dependent, that equality does not generally hold; dependence changes the exact probability. The Bonferroni bound, at most \(K\alpha\) for \(K\) tests each conducted at level \(\alpha\), does not require independence.
Worked Example: Selecting the Larger Interim Result
In a hypothetical trial, investigators examine one-sided standardized interim treatment-effect statistics for two disjoint biomarker strata. Under the global null, assume \(Z_1\) and \(Z_2\) are independent standard normal variables. The rule is to select the stratum with the larger statistic and call it positive if its statistic exceeds 1.645.
Suppose the observed interim values are \(Z_1=1.82\) and \(Z_2=0.74\). The rule selects stratum 1 because \(1.82>0.74\). Since \(1.82>1.645\), the selected result is called positive. Its one-sided nominal p-value is approximately 0.0344.
That nominal p-value describes a test of one fixed stratum; it does not account for the fact that the larger of two results was selected. Under the stated global-null assumptions, the actual probability of a positive selection is \(1-(0.95)^2=0.0975\). Thus the decision rule does not control the global type I error at 0.05, despite the selected result having an unadjusted p-value below 0.05.
This calculation is deliberately simple. In an actual adaptive trial, candidate populations may overlap, test statistics may be correlated, and selection may use conditional power or other interim summaries rather than the largest \(Z\)-statistic. The particular inflation then depends on the design and joint distribution. The general warning remains: a nominal level for a single prespecified hypothesis does not automatically protect a claim selected from several data-responsive candidates.
Prespecified Safeguards and Their Assumptions
A confirmatory procedure needs to specify, before relevant outcomes are examined, which populations may be selected, how selection is made, and how the final claim will be tested. The biomarker definition, analysis population, endpoint, direction of benefit, and handling of ties or unavailable interim data should also be fixed. These constraints make the decision rule auditable, but merely writing down a selection rule does not by itself control type I error. The testing procedure must account for selection or use evidence that is independent of it.
One direct safeguard is an independent confirmation sample. If the selected population is determined using interim data, and the confirmatory outcomes are independent of those data under each relevant null, then conditional on the interim results the selected index is fixed. A valid level-\(\alpha\) test in the confirmation sample therefore has conditional type I error at most \(\alpha\) for a true null selected for testing. This protects a selected-population claim, provided the confirmatory test and target are not altered after examining the confirmation outcomes.
Another safeguard is to adjust for the family of candidate claims. For example, testing each of \(K\) candidate null hypotheses at level \(\alpha/K\) controls familywise type I error at level \(\alpha\), regardless of dependence, by the union bound. More powerful procedures may use the prespecified logical structure among hypotheses, such as closed testing, or a prespecified allocation of alpha across candidate populations. Whatever procedure is chosen, its hypotheses and rejection rules must cover the claims that the selection process can produce.
Worked Example: Selecting First, Confirming Independently
Consider a hypothetical two-stage trial with two prespecified marker populations, A and B. The interim stage uses its outcomes only to select a population. The final-stage participants provide independent confirmatory outcomes, and the selected population is tested with a prespecified one-sided level-0.05 test. Assume the biomarker classification, endpoint, and test model are fixed, and that the confirmatory statistic is valid under the null for either population.
At interim, suppose the standardized results are \(Z_{A,I}=2.05\) and \(Z_{B,I}=1.31\). The rule selects A because \(2.05>1.31\). In the independent confirmation data, suppose the prespecified statistic for A is \(Z_{A,C}=1.78\), giving a one-sided p-value of approximately 0.0375. Since \(0.0375<0.05\), the trial rejects \(H_{0A}\) under this design.
The error-control argument is conditional on the interim results. Once those results are observed, A is the fixed selected target; under \(H_{0A}\), the independent confirmation test has rejection probability at most 0.05. Averaging over all possible interim outcomes preserves that bound. If A is false, rejection is not a type I error for A; if a true null is selected, its conditional rejection probability is still bounded by 0.05. The argument would fail if the confirmation data were used to switch to B after seeing A's result, or if the claimed test included the selection-stage outcomes without an appropriate adjustment.
Conditional Error: A General Preservation Principle
Independent confirmation is not the only possible safeguard. A general way to assess an adaptive change is to compare its conditional rejection probability with the conditional error allowed by the original design. Let \(I\) denote the interim data, and let \(\phi_0\) be the original design's final rejection indicator, equal to 1 when it rejects and 0 otherwise. Under a specified null hypothesis, define the original conditional error by \(e_0(I)=P_0(\phi_0=1\mid I)\). The original design has type I error at most \(\alpha\) if \(E_0\{\phi_0\}\leq\alpha\).
The justification follows from the law of total probability. Write \(\phi_A\) for the adaptive procedure's rejection indicator. If \(P_0(\phi_A=1\mid I)\leq e_0(I)\) almost surely, then:
This argument explains why an adaptive selection rule can be compatible with confirmatory inference: the adaptation must not spend more conditional error than the design has available. The condition is not satisfied merely because a rule was prespecified. The conditional rejection probability must actually be derived or otherwise justified under the relevant null, including any nuisance-parameter assumptions.
Worked Example: Bonferroni Protection for Three Candidate Populations
Suppose three prespecified biomarker populations are eligible for selection, and the same interim evidence is used both to select the smallest one-sided p-value and to make a confirmatory claim. Assume each candidate test has a valid p-value under its own null. To control familywise type I error at \(\alpha=0.05\) without relying on a particular dependence structure, test each candidate at \(0.05/3\).
The candidate-specific cutoff is \(0.01667\). Equivalently, the selected p-value can be multiplied by three, with the adjusted value capped at 1. Suppose the selected population has \(Z=2.20\), giving a one-sided p-value of approximately 0.0139. Its adjusted p-value is:
Because \(0.0417<0.05\), the selected claim passes this Bonferroni procedure. A selected p-value of 0.020, by contrast, would give \(3\times0.020=0.060\), and would not pass. The adjustment is valid under arbitrary dependence because the probability of at least one false rejection is no greater than the sum of the candidate-specific rejection probabilities, which is \(3(0.05/3)=0.05\).
Diagnostics, Alternatives, and Limits
Before interpreting a selected-population result, check whether the proposed procedure matches the data actually used. A nominal p-value may be insufficient if outcomes that influenced selection are also included in the final statistic. Likewise, a rule written as “select the most promising subgroup” is incomplete unless “promising,” the eligible populations, and the subsequent testing method are operationally defined. Investigators should verify that the chosen multiplicity procedure covers every possible confirmatory claim and that assumptions such as independence for a confirmation sample are credible.
| Approach | How it protects a claim | Main qualification |
|---|---|---|
| Independent confirmation | Selection is fixed before independent confirmatory outcomes are tested at level \(\alpha\). | Confirmation data must not influence selection or subsequent switching. |
| Multiplicity adjustment | Allocates a controlled error budget across candidate hypotheses. | Can be conservative; the candidate family and claim structure must be specified. |
| Conditional error preservation | Constrains adaptive rejection probability, given interim data, by the original design's conditional error. | Requires a valid calculation for the relevant null hypotheses. |
| Unadjusted reuse of selected evidence | Uses the ordinary single-test cutoff after choosing the favorable result. | Generally does not control the error rate for data-dependent selection. |
Selection thresholds based on conditional power, minimum estimated effect, or clinical feasibility can help decide which population to continue. They do not, by themselves, adjust the type I error of a later confirmatory claim. Similarly, a fixed rule can still be statistically invalid if it allows a nominal single-test threshold after searching over multiple candidates. The distinction is between a rule that determines trial conduct and a test that controls false claims.
These safeguards have limits. Bonferroni control may lose power, while conditional error calculations can be design-specific and technically demanding. Independent confirmation may require additional participants or resources, and its validity depends on genuine separation of the selection and confirmation evidence. Results from a selected population also support a claim about that defined population; they do not automatically establish benefit in other populations.
Common Pitfall
Incorrect reasoning: “The selected subgroup has a p-value below 0.05, and the rule for choosing it was written in the protocol, so the type I error is 5%.” Prespecification improves transparency, but it does not make a selected test behave like a single test fixed in advance.
Correction: Identify all populations the rule could select, determine whether the selection and test use overlapping evidence, and specify how the resulting family of claims is protected. Use independent confirmation, an appropriate multiplicity procedure, or a conditional-error argument. Then interpret the p-value against that complete procedure, not as an isolated single-test result.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Under a global null, two independent one-sided tests each use \(\alpha=0.05\). The larger statistic is selected and tested at 0.05. What is the probability of at least one false positive?
- Why can an independent confirmation sample protect a selected-population test at level 0.05?
- For four candidate populations, what Bonferroni level should be used per candidate to control familywise error at 0.05?
- State the conditional-error condition that preserves the original design's type I error.
- Does prespecifying a conditional-power rule alone guarantee confirmatory type I error control? Explain.
Solutions. (1) The probability is \(1-(1-0.05)^2=1-0.9025=0.0975\), or 9.75%. (2) Conditional on interim data, the selected population is fixed; if confirmation outcomes are independent and the test is valid under that population's null, its conditional rejection probability is at most 0.05. Averaging over interim results preserves the bound. (3) Use \(0.05/4=0.0125\) per candidate. (4) Under the null, for almost every interim result \(I\), require \(P_0(\phi_A=1\mid I)\leq e_0(I)\), where \(e_0(I)\) is the original design's conditional rejection probability. Taking expectations then bounds adaptive type I error by the original rate. (5) No. The rule can govern which population continues, but the confirmatory test must still account for selection, for example through independent confirmation, multiplicity adjustment, or conditional-error preservation.