Why the Role of Phase 2 Data Must Be Decided in Advance
In a seamless Phase 2/3 trial, observations gathered during Phase 2 may help investigators decide whether to continue, which treatment to advance, or how much Phase 3 recruitment is needed. The same observations might also be proposed for the primary confirmatory analysis. Those are distinct roles. Data that help choose a treatment are not automatically independent evidence that the chosen treatment works.
The central question is not simply whether Phase 2 patients were randomized or whether their measurements resemble Phase 3 measurements. It is whether the confirmatory hypothesis, target estimand, data handling, and inferential procedure were specified so that the final test has its intended error rate despite the stage-transition decision. In particular, a conventional test that treats a selected Phase 2 result as if the treatment had been fixed in advance can be misleading.
Conditions for Reusing Observations
Before Phase 2 observations can contribute to a primary confirmatory test, the protocol and statistical analysis plan should make clear what is being tested and how each stage contributes. The following conditions are fundamental; satisfying them does not, by itself, validate every possible test.
- A coherent estimand. The treatment contrast must refer to a defined population, treatment conditions, outcome, and summary measure across the stages. If the population or endpoint changes at the transition, the analysis must justify how the stagewise observations relate to the same estimand.
- Compatible conduct and measurement. Randomization, treatment delivery, endpoint definition, follow-up, and data quality procedures must support combining the observations. Material changes may create different stage-specific quantities rather than repeated measurements of one common target.
- Prespecified confirmatory role. The plan should state whether Phase 2 data are included, which hypotheses they inform, and what analysis will be used. A decision made after inspecting results cannot be treated as if it had been fixed before the data were observed.
- Valid calibration under adaptation. The primary test must control its type I error under the null for the actual design, including any treatment selection or other data-dependent transition. Ordinary pooled inference is not automatically valid after selection.
- Correct accounting of observations. Each participant contributes once to the analysis. Phase 2 data must not be counted again as though the same participants had supplied a second, independent sample in Phase 3.
These requirements separate a useful operational decision from a defensible confirmatory claim. For example, a Phase 2 estimate can be informative for choosing a dose even if it is excluded from the primary test. Conversely, an analysis may use Phase 2 outcomes confirmatorily when its design and test were planned to preserve valid inference under the transition rule.
Combining Stagewise Effects: Define the Target First
Suppose stage \(k\) estimates a treatment effect \(D_k\), with positive values favoring the experimental treatment. A fixed weighted average can summarize stage effects, but its weights define what is being averaged. To call it a weighted average, require nonnegative weights summing to one:
If the weights do not meet these conditions, the expression is a weighted contrast, not a weighted average. Even nonnegative weights summing to one do not guarantee that \(D\) estimates a clinically meaningful target: the stage populations, treatment effects, and intended target population must also be considered. For a common treatment effect, several reasonable weighting choices may estimate the same quantity. Under effect heterogeneity, different weights can define different average effects.
A simple justification for uncertainty calculations is available when the stage estimates are independent and approximately normal, with \(D_k\sim N(\theta_k,SE_k^2)\), and the weights are fixed in advance. Then linearity of expectation and independence give
Under the null hypothesis that the weighted target is zero, the standardized statistic \(Z=D/\sqrt{\sum_k a_k^2SE_k^2}\) has a standard normal distribution under the stated normal model. This derivation supports inference for fixed weights and the specified target. It does not establish validity when weights, treatment choices, or hypotheses are selected using the same outcomes; the confirmatory procedure must account for those adaptations.
Worked Example: A Fixed Weighted Target Across Two Stages
Suppose, hypothetically, that the protocol defines the target effect as a fixed mixture of the Phase 2 and Phase 3 stage populations, with weights \(a_1=0.40\) and \(a_2=0.60\). The stage estimates are \(D_1=3.0\) points and \(D_2=1.0\) point, with standard errors \(SE_1=1.5\) points and \(SE_2=1.0\) point. Assume the stage estimates are independent, approximately normal, and unbiased for their respective stage effects; assume also that the weights and target were fixed in advance and no data-dependent selection affects this analysis.
The estimated weighted target is
Its estimated variance and standard error are
For the two-sided test of a zero weighted target, the statistic is \(Z=1.80/0.8485=2.12\), giving an approximate two-sided \(p\)-value of 0.034. Under the assumptions and fixed-target interpretation above, the estimate is 1.80 points and the test is statistically significant at the 0.05 level. This calculation does not justify choosing the weights after seeing the stage estimates, nor does it show that the same weighted target represents every possible patient population.
Selection and Confirmation Are Not the Same Evidence
Selection creates a particular risk when the most promising treatment is chosen from several candidates using Phase 2 outcomes. Even if each candidate’s estimate is unbiased before selection, the selected estimate tends to be unusually favorable because selection favors high observed results. Treating that selected estimate as an ordinary, unbiased contribution to a conventional test can make the evidence look stronger than it is.
One straightforward strategy is to use Phase 2 outcomes for selection only, then test the selected treatment using independent Phase 3 data. If the Phase 3 test is valid for the selected treatment conditional on the Phase 2 decision, the confirmation does not reuse the selection noise. Whether additional multiplicity or other adjustments are needed depends on the claims and the design. Another strategy is to include both stages through a prespecified adaptive procedure calibrated for the transition rule. In that case, Phase 2 data can contribute to confirmation, but the test is not justified merely by pooling stage summaries.
Worked Example: Selection Followed by Independent Confirmation
Consider a hypothetical trial with three candidate treatments and one control. In Phase 2, each treatment-control effect is estimated with a standard error of 3 points. The observed estimates are 6, 4, and 0 points, so the first treatment is selected. In a separate Phase 3 cohort, its estimated effect is 2 points with standard error 1 point. Assume the Phase 3 data are independent of Phase 2, the confirmatory endpoint and population are aligned, and the prespecified primary test is one-sided at \(\alpha=0.025\).
For the Phase 3-only test of \(H_0:\theta\leq 0\) against \(H_a:\theta>0\), use the stated normal approximation:
Because \(0.0228<0.025\), the Phase 3-only test rejects \(H_0\) at the prespecified one-sided level. Its estimate is 2 points, and the evidence comes from data independent of the selection results. The Phase 2 estimates helped choose the candidate but are not included in this calculation. A different design could reuse them using a valid adaptive test, but simply adding the selected Phase 2 estimate to this Phase 3 estimate would not preserve the calculation’s stated calibration.
Worked Example: Why a Promising Selection Estimate Is Not Independent Confirmation
Suppose, hypothetically, that the same three Phase 2 estimates in the previous example are used to select a treatment, and that the largest estimate, 6 points, is then described as confirmatory evidence for that selected treatment. Its nominal stagewise statistic would be \(6/3=2.00\), with a one-sided standard-normal \(p\)-value of 0.0228. But this \(p\)-value treats the estimate as though the treatment had been fixed before those outcomes were observed. The treatment was chosen precisely because its observed result was largest.
For scale, if three independent standard-normal statistics were each tested at a one-sided 0.025 level and the largest were reported, the chance under a global null that at least one exceeded the threshold would be
Thus, in this simplified illustration, the chance of at least one nominal crossing is about 0.073, not 0.025. Real designs can have dependent outcomes, different selection rules, and other complications, so this calculation is illustrative rather than a general adjustment. It demonstrates why the selected Phase 2 result alone cannot be treated as an independent confirmatory test. A prespecified method must address the actual selection and testing procedure.
What to Check When Planning or Reviewing a Design
A practical review should begin with the primary claim and proceed backward to the data that will support it. Ask whether Phase 2 and Phase 3 measure the same endpoint at compatible times, whether the target population is stable or explicitly redefined, and whether all randomized participants remain handled consistently. Then inspect how the transition depends on Phase 2 outcomes. A fixed transition that does not depend on efficacy results poses a different inferential problem from selecting a treatment or hypothesis using those results.
Warning signs include an analysis plan that says “pool all available patients” without specifying the hypothesis and test; a treatment selected on Phase 2 efficacy followed by an ordinary pooled \(p\)-value; post hoc changes to the population or endpoint; and a claimed common effect despite substantial differences in stage populations or conduct. These are prompts to examine the design, not automatic proof that the analysis is invalid. The relevant question is whether the specified inferential method is valid for the actual design and estimand.
The main alternatives have different trade-offs:
| Approach | Role of Phase 2 observations | Main consideration |
|---|---|---|
| Selection only; test Phase 3 data | Inform the transition, but do not enter the primary test | Clear separation from selection evidence; may require more new recruitment |
| Prespecified adaptive confirmatory test | Contribute through a test designed for the adaptation | Requires design-stage specification and valid calibration for the actual rules |
| Ordinary pooled test after outcome-based selection | Selection and confirmation use the same outcomes without accounting for the selection | Nominal error rates may not hold; avoid assuming standard fixed-design calibration |
No single approach is universally preferable. Reuse can improve efficiency, but it also makes the design and analysis more demanding. If the adaptation is complex, the relevant null behavior should be evaluated for the entire decision-and-testing procedure, not just for a test statistic considered in isolation.
Common Mistakes and Limitations
- Calling every pooled analysis confirmatory. Combining estimates does not itself establish valid type I error control. The test must be appropriate for the adaptation.
- Assuming randomization resolves selection. Randomization supports valid within-stage comparisons, but it does not remove the favorable selection of the largest result among candidates.
- Confusing a weighted estimate with a universal effect. Fixed nonnegative weights summing to one define an average of stage effects. If those effects differ, the result depends on the weights and the populations represented.
- Counting Phase 2 participants twice. Reuse means including their observations once in a planned analysis, not treating them as additional independent Phase 3 participants.
- Changing the confirmatory question at the transition. A population, endpoint, or treatment contrast that changes may define a different estimand; the original primary claim cannot be assumed to carry over unchanged.
The basic lesson is deliberately conditional: Phase 2 observations can contribute to confirmatory testing when the target and analysis are specified and the test remains valid under the design’s selection and transition rules. If Phase 2 data are used to select the treatment, their confirmatory contribution needs an appropriate adaptive method; otherwise, independent Phase 3 evidence provides the cleaner separation.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A trial selects the treatment with the largest Phase 2 estimate, then reports a conventional pooled test using the selected treatment’s Phase 2 and Phase 3 observations. What is the central inferential concern?
- Two stage effects are 2 and 5 points. Are weights \(0.3\) and \(0.7\) a weighted average? What value do they produce?
- What must be established before Phase 2 observations can enter a primary confirmatory test?
- Why can a Phase 2 result be useful for treatment selection but insufficient as independent confirmation?
Solutions. (1) The treatment was selected using outcomes that also enter the test; ordinary fixed-design calibration may not account for that selection. (2) Yes: both weights are nonnegative and sum to one. The weighted estimate is \((0.3)(2)+(0.7)(5)=0.6+3.5=4.1\) points. (3) At minimum, define a coherent estimand, ensure compatible conduct and measurement, prespecify the confirmatory role, and use a test valid under the actual adaptation. (4) Selection favors promising observed results, so the selected estimate is not independent of the decision that chose it; independent Phase 3 data or a valid adaptive test is needed for confirmatory inference.