Why Treatment Selection Need Not Invalidate a New Test
In a seamless phase 2/3 trial, phase 2 data may be used to choose which candidate treatment proceeds. That choice is data-dependent: a treatment with an unusually favorable phase 2 result may be selected even when its true effect is no better than the others. This selection can distort evidence drawn from those same data. It does not, by itself, invalidate a new test based only on independent phase 3 data.
The key is to distinguish the selection rule from the confirmatory test. The rule may depend on phase 2 outcomes, but the phase 3 test must be valid for the selected treatment under its null hypothesis, conditional on the phase 2 information that determined the selection. The validity claim concerns this design and analysis; it is not a blanket guarantee for every analysis performed after treatment selection.
A uniform null p-value satisfies this inequality with equality; a conservative p-value satisfies it with inequality. Conditioning on \(D_2\) is stronger than checking only that the p-value has the right marginal distribution over repeated trials. It says the test remains calibrated for every phase 2 result that could have led to a particular selection.
A standard sufficient design condition is that phase 3 outcomes used for the confirmatory test are independent of \(D_2\) under the null, and that the phase 3 analysis is itself valid for the selected treatment. For example, a new randomized comparison of the selected treatment with control can meet this condition if its participants and outcomes are distinct from the phase 2 evidence and the test is correctly calibrated. A phase 3 sample size may be chosen using phase 2 information, provided the test used at that sample size remains valid conditional on that choice.
Deriving Conditional Validity
Suppose the phase 3 test for each candidate is valid under its null, and the phase 3 data are independent of the phase 2 data. Under \(H_{0j}\), the unconditional p-value satisfies \(P(P_{3j}\leq u)\leq u\). Independence means that learning \(D_2\), including which treatment was selected, does not change the null distribution of the phase 3 test statistic. Thus:
Since \(J\) is determined by \(D_2\), conditioning on \(D_2\) also fixes the selected index. Therefore, when the selected treatment's null is true, its phase 3 p-value remains super-uniform conditional on the selection information. The argument does not require the selection rule to choose the treatment with the largest phase 2 estimate, or even to be a simple rule. It does require that the rule be based on information independent of the confirmatory phase 3 outcomes.
To see why this also controls the probability of a false claim across candidates, let \(\mathcal{T}\) be the set of candidates whose null hypotheses are true. A false rejection can occur only if the selected candidate belongs to \(\mathcal{T}\). Partitioning by the selected index gives:
Here \(I(J=j)\) equals 1 when treatment \(j\) is selected and 0 otherwise. The conditional validity inequality applies inside the expectation for each true null. The final sum is at most 1 because at most one candidate is selected. This is why testing only one selected treatment with a fresh, conditionally valid phase 3 test does not automatically require splitting \(\alpha\) among all candidates. It is different from making confirmatory claims about multiple candidates.
Worked Example: Testing the Selected Treatment With New Phase 3 Data
Suppose two candidate treatments are compared in phase 2, and the one with the larger estimated improvement is selected. The confirmatory endpoint is improvement in a prespecified score, measured in points. In the new phase 3 comparison, the selected treatment has an estimated improvement over control of 3.2 points with standard error 1.4 points. Assume independent phase 3 data and a valid large-sample normal test of \(H_{0J}\): no improvement, against a one-sided alternative of positive improvement. Use \(\alpha=0.025\).
The assumptions for this calculation are that the phase 3 test statistic is approximately standard normal under the null, its standard error is valid, and the phase 3 outcomes used here are independent of the phase 2 data that selected \(J\). The phase 3-only statistic is:
The one-sided p-value is \(P(Z\geq2.286)\approx0.0111\), where \(Z\sim N(0,1)\) under the null. Since \(0.0111<0.025\), the selected treatment passes this prespecified phase 3 test. Under the stated conditions, this is a level-0.025 confirmatory result for the selected treatment, despite selection based on phase 2 data. The estimate is expressed in score points; the p-value is not the probability that the null hypothesis is true.
Controlling the Chance of a False Selected Claim
Conditional validity is useful because the selected treatment's null may be true even when some other candidate is effective. The theorem controls the chance of falsely claiming benefit for the treatment actually selected, across such configurations. For example, if treatment A is ineffective and treatment B is effective, the chance of a false claim is the chance A is selected and its test rejects. It is not necessary for A to be selected with certainty for the test to retain overall error control.
Worked Example: Error Control When One Candidate Is Ineffective
Suppose candidates A and B are considered. A is ineffective, while B is effective. The phase 2 selection rule selects A with probability 0.70 and B with probability 0.30. Assume the phase 3 p-value for A is conditionally valid at level 0.05, including conditional on the phase 2 data that led to selecting A. A false confirmatory claim can occur only when A is selected and its null is rejected.
By conditioning on selection, the probability of a false claim is:
Thus the chance of a false claim is at most 3.5%, below the nominal 5% level. If both A and B were ineffective, the same conditional bound would apply to each selected candidate, and the selection probabilities would sum to 1, giving an overall bound of 5%. This calculation relies on conditional validity, not on phase 2 selecting the correct treatment most of the time.
What Breaks the Argument?
The derivation depends on the data and the test actually matching the conditions in the argument. Independence is especially important: if phase 2 outcomes affect the phase 3 statistic beyond the selection decision, the conditional distribution need not remain the null distribution used to calculate the p-value. Likewise, a valid p-value for a different endpoint, population, or treatment contrast does not establish the claim being tested.
- Reusing selected phase 2 evidence: a phase 2 estimate that helped select the treatment is not automatically independent evidence. Combining it with phase 3 evidence requires a method designed for stagewise evidence and its assumptions; simply treating the pooled result as a fresh phase 3 test is not justified.
- Overlapping or dependent data: shared participants, reused controls, or other dependence can undermine the independence argument. The actual joint data structure and the test's conditional calibration must be addressed.
- Changing the claim after seeing results: changing the endpoint, population, direction of the alternative, or analysis in response to observed phase 3 results can make the reported test different from the prespecified confirmatory test.
- Ignoring a selection-driven analysis choice: if phase 2 results determine more than the treatment—for example, which endpoint or test is used—then conditional validity must cover those choices too. Validity for treatment selection alone does not automatically cover additional data-dependent decisions.
A useful diagnostic is to write down exactly what information determines \(J\), then identify every observation entering \(P_{3J}\). Ask whether the conditional null distribution of that p-value is valid for every possible phase 2 history. If the answer relies on treating reused observations as independent, or on an unverified approximation, the central justification is missing.
Worked Example: Why the Phase 2 Winner's P-Value Is Not a Fresh Test
Consider two ineffective candidate treatments. Suppose their phase 2 test statistics \(Z_1\) and \(Z_2\) are independent standard normal variables under the null, and the treatment with the larger statistic is selected. A tempting but invalid procedure is to report the selected phase 2 statistic as if it had come from a single prespecified test, rejecting for a one-sided statistic above 1.645.
For either candidate alone, the probability of exceeding 1.645 is approximately 0.05. But the selected statistic is \(\max(Z_1,Z_2)\). The probability of a false rejection is:
The actual false-rejection probability is 9.75%, not 5%. The error arises because the same phase 2 values both chose the treatment and supplied the reported test statistic. By contrast, a new phase 3 statistic independent of phase 2 has a 5% conditional rejection probability under its null, no matter which treatment was selected. The normal calculation assumes independent standard normal phase 2 statistics; dependence between candidates would change the numerical probability, but would not make the naive single-test calibration valid by itself.
Comparing Confirmatory Strategies and Their Limits
A phase 3-only test is a direct route to a confirmatory claim when it is independent and valid conditional on the phase 2 selection. It can use a prespecified level-\(\alpha\) test for the one selected treatment. This approach sacrifices the statistical information in phase 2 for that final test, but it makes the validity argument comparatively transparent.
An alternative is to use evidence from both stages through a prespecified combination procedure. Such a procedure must account for the stagewise evidence and preserve error control under the relevant dependence and selection rules; the ordinary phase 3-only argument does not justify an arbitrary pooled analysis. A further alternative is to test multiple candidate treatments for multiple claims, which requires a procedure that controls multiplicity across those claims. These strategies answer different inferential questions and cannot be substituted for one another without checking their assumptions.
The conditional-validity result also has limits. It establishes control of false rejection for the specified null and analysis, not the clinical importance of an estimated effect, the adequacy of the trial's conduct, or the transportability of results to other populations. Independence and a valid test cannot repair a poorly defined estimand, substantial bias in outcome ascertainment, or an analysis that does not match the prespecified claim.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- What does conditional validity require of a phase 3 p-value under its null hypothesis?
- One of four candidates is selected using phase 2 data. Why does a valid phase 3 test for only the selected candidate not automatically require dividing \(\alpha\) by four?
- In the two-candidate example, what is the false-rejection probability if the selection rule chooses the larger of two independent standard normal statistics and the cutoff is 1.645?
- Name two ways a nominally phase 3 test could fail to meet the assumptions needed for confirmatory validity after selection.
Solutions. (1) For every \(u\) between 0 and 1, under the null, \(P(P_{3j}\leq u\mid D_2)\leq u\). (2) There is only one confirmatory claim, and the probabilities of selecting the different candidates sum to 1; conditional validity bounds the false-rejection probability within each selection event, so the overall bound is \(\alpha\). (3) \(1-(0.95)^2=0.0975\), or 9.75%, because either statistic can exceed the cutoff. (4) Examples include reusing phase 2 observations in the test, overlapping phase 2 and phase 3 participants in a way that induces unaccounted dependence, or changing the endpoint or analysis after observing results.