Why Selection Is Not Confirmation
A Phase 2 stage may compare several treatments or doses and use the results to choose what should proceed to Phase 3. That choice is a selection decision: it identifies a candidate for further study. It does not, by itself, establish that the candidate is effective. In particular, the treatment with the largest observed effect can win a comparison partly because its estimate happened to be unusually high.
A sound transition therefore connects three things: the candidates that were eligible for selection, the Phase 2 evidence used to choose among them, and the confirmatory claim that will be tested. The treatment effect must refer to the same target population, endpoint, and estimand across the boundary. The earlier material on preserving the estimand across stages is relevant here: a favorable Phase 2 result for one outcome or population cannot automatically support a confirmatory claim about another.
For example, if \(\theta_k\) is a mean improvement in a symptom score, measured in points with positive values favoring treatment, a one-sided confirmatory hypothesis may be \(H_{0k}:\theta_k\leq 0\) against \(H_{ak}:\theta_k>0\). If a clinically meaningful minimum effect is required, the null boundary should instead reflect that minimum. The choice of boundary belongs to the clinical claim; it should not be changed after seeing which regimen looks best.
Formulating a Treatment or Dose Choice
A useful selection formulation specifies, before examining the transition data, which regimens can be selected and what evidence matters. Phase 2 evidence may include estimated efficacy, uncertainty, adverse events, dose burden, or feasibility. These criteria need not all be combined into a single score. A transparent rule can first exclude candidates that fail a safety or feasibility constraint, then compare efficacy among those remaining.
The relevant estimates are noisy. With several candidates, choosing the largest estimate tends to select a regimen whose Phase 2 estimate includes positive random error. This is a selection effect, not evidence that the estimate is biased in every trial. It does mean the winning estimate should not be treated as an unbiased forecast of the effect in new patients. A Phase 3 study can provide fresh evidence, but only if its analysis and data use support the planned claim.
Worked Example: Selecting a Treatment Under a Safety Constraint
Suppose a hypothetical Phase 2 study compares three treatments with a control on a binary response endpoint. The estimated response rates are 40% for control and 50%, 56%, and 59% for treatments A, B, and C. Estimated excess rates of a prespecified adverse event, relative to control, are 1, 3, and 8 percentage points, respectively. Consider a rule that excludes any treatment with estimated excess adverse-event risk above 5 percentage points, then selects the eligible treatment with the largest estimated response-rate difference.
The estimated response-rate differences are:
Treatment C has the largest estimated efficacy difference, but its 8-percentage-point excess adverse-event rate exceeds the 5-point constraint, so it is ineligible under this rule. A and B meet the constraint. Since \(0.16>0.10\), the rule selects B. This decision says that B is the eligible candidate with the largest observed response-rate difference under the stated criterion; it does not establish that B is effective or that its true effect is exactly 16 percentage points.
A dose-selection rule can express a different clinical preference. For example, it may prefer the lowest dose whose estimated efficacy is close to the best eligible dose, rather than the dose with the maximum observed effect. This can avoid advancing a higher dose when its small apparent gain is accompanied by greater toxicity or treatment burden. The rule must still respect the confirmatory estimand: if Phase 3 is intended to test the selected dose against control, the claim should concern that dose and the same relevant outcome and population.
Worked Example: Choosing a Dose Rather Than the Maximum Estimate
In a hypothetical dose-ranging stage, the estimated mean symptom-score improvements over control are 5, 8, and 9 points at 5 mg, 10 mg, and 20 mg. The corresponding excess adverse-event rates are 1%, 3%, and 9%. Suppose the selection criterion is: exclude doses with excess adverse-event rate above 5%, then choose the lowest eligible dose whose estimated improvement is within 2 points of the largest eligible improvement.
The 20 mg dose is excluded because \(9\%>5\%\). The largest estimated improvement among the eligible doses is 8 points, at 10 mg. A dose is within 2 points of that value if its estimate is at least \(8-2=6\) points. The 5 mg estimate is 5, which is below 6; the 10 mg estimate is 8, which meets the criterion. Thus the rule selects 10 mg. This choice reflects the stated efficacy-and-safety preference, rather than an unsupported conclusion that 10 mg is the uniquely optimal biological dose.
When Selection Can Be Followed by a Valid Confirmatory Test
A particularly clear case arises when the Phase 3 confirmatory data are new and independent of the Phase 2 data used to select \(S\). Assume that (1) each candidate's confirmatory test is valid at level \(\alpha\) under its own null hypothesis, (2) the test statistic and analysis are specified so that the p-value is valid under that null, and (3) the Phase 3 data used for that test are independent of the Phase 2 selection data. Independence makes the confirmatory p-value valid even conditional on which candidate was selected.
For a single selected regimen and a single confirmatory claim, testing the selected candidate at level \(\alpha\) can then control the probability of a false rejection strongly: this holds whether all candidate null hypotheses are true or only some are true. Let \(K_0\) denote the set of candidates whose null hypotheses are true. A false rejection occurs only if the selected candidate belongs to \(K_0\) and its null is rejected. Since exactly one candidate is selected, the event can be partitioned by the selected candidate:
The second line uses the conditional validity of the confirmatory test, which follows here from independence of the Phase 3 data and the Phase 2 selection. The final inequality holds because the probabilities of selecting candidates in \(K_0\) sum to at most 1. Restricting the sum to true-null candidates is essential: a level-\(\alpha\) bound does not apply to rejection probabilities under false null hypotheses. This argument concerns one selected claim; it does not automatically cover multiple confirmatory claims, selection based on the confirmatory data, or reuse of Phase 2 observations in the test.
Worked Example: Testing the Selected Treatment on New Data
Suppose Phase 2 selects treatment B, and a separate hypothetical Phase 3 cohort is used for the confirmatory test. The prespecified estimand is the treatment-minus-control difference in mean symptom improvement, in points. The model-based estimate from Phase 3 is 4.20 points with standard error \(SE=2.004\) points. Assume the prespecified large-sample normal test is appropriate, the Phase 3 observations are independent of the selection data, and larger values favor treatment. Test \(H_{0B}:\theta_B\leq 0\) against \(H_{aB}:\theta_B>0\) at \(\alpha=0.025\).
At the boundary of the null, the test statistic is:
The one-sided normal p-value is \(P(Z\geq 2.096)=0.01804\), where \(Z\) has a standard normal distribution. Since \(0.01804<0.025\), reject \(H_{0B}\). Under the stated model and design assumptions, the Phase 3 data provide evidence that treatment B improves the mean symptom score relative to control. The p-value is calculated from the new confirmatory data; it is not a probability that the null hypothesis is true.
Data Reuse, Alternatives, and Diagnostics
If some Phase 2 observations are reused in the confirmatory statistic, the clean independence argument no longer applies automatically. Selection and testing may then depend on the same random variation. A nominal level-\(\alpha\) test applied as if the selected treatment had been fixed in advance can have an inflated false-rejection probability. The earlier material on reusing Phase 2 data in confirmatory testing addresses the need for a valid method when data cross the boundary. The selection decision must be compatible with that method; simply selecting the largest Phase 2 estimate and pooling all observations does not establish validity.
The main alternatives have different implications:
| Approach | What it offers | Key limitation |
|---|---|---|
| Select using Phase 2, then test with independent Phase 3 data | Selection does not affect the conditional validity of a single selected-candidate test. | Requires a genuinely separate confirmatory data contribution and a compatible estimand and analysis. |
| Select and reuse Phase 2 data | Can use information from both stages and may improve efficiency. | Requires an analysis that accounts for selection and data reuse; naive pooling is not justified by the independent-data argument. |
| Test all candidates as confirmatory claims | Can support claims about more than the selected regimen. | Multiplicity across claims must be addressed, and this is a different objective from making one selection. |
Several warning signs merit attention before a regimen advances. The apparent winner may have only a small advantage over alternatives relative to uncertainty; a safety estimate may be based on too few events to make an exclusion rule informative; or the dose-response pattern may be irregular. Also check that the endpoint, population, and handling of intercurrent events represented in Phase 2 match the planned confirmatory claim. These issues do not mechanically determine the selection, but they can show that a simple ranking is fragile or that the intended claim has changed.
A frequent mistake is to argue, “We selected the best-looking treatment, so its Phase 2 p-value is confirmatory,” or to treat the maximum observed effect as the expected effect in Phase 3. Ranking answers which candidate looked best among those examined; it does not account for the uncertainty induced by searching across candidates. A correction is to separate selection from confirmation: use Phase 2 evidence to choose a candidate under a clear clinical criterion, then use independent confirmatory data and a valid test for the stated claim—or use a method designed to account for data reuse.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study selects one of four treatments using Phase 2 data and tests it using an independent Phase 3 cohort. Why can the selected treatment be tested at level \(\alpha\) for one confirmatory claim?
- Three candidates have true null hypotheses and one has a false null. Why must a strong-control argument sum over true-null candidates rather than apply a level-\(\alpha\) bound to all four?
- A dose has the largest estimated benefit but fails a prespecified safety constraint. Under a rule that excludes doses failing that constraint, should it be selected? Explain.
- What is the problem with using a nominal Phase 2 p-value for the treatment selected because it had the largest Phase 2 estimate?
Solutions. (1) Conditional on the Phase 2 selection, the independent Phase 3 data still produce a valid level-\(\alpha\) test for the selected candidate's null. (2) A level-\(\alpha\) upper bound applies when a null is true; a false null may have rejection probability greater than \(\alpha\). Summing over the true-null candidates bounds the probability of a false rejection. (3) No. The rule excludes it, regardless of its efficacy ranking; selection follows the stated eligibility criterion. (4) The candidate was chosen using the same random Phase 2 evidence being tested, so the selected estimate and p-value need not have their nominal preselection interpretation.