Why Multiple Phase 2 Outcomes Need a Rule
A phase 2 decision may depend on more than the largest estimated efficacy effect. A dose with a promising response may also have an unacceptable safety profile; a lower dose may appear safer but show little evidence of benefit; and the pattern across doses may help distinguish a plausible dose response from a noisy high-dose estimate. Because the same phase 2 data can influence several parts of the decision, the selection rule needs to say in advance how those outcomes will be used.
Selection is a decision about which treatment or dose to carry forward. It is not, by itself, proof that the selected treatment is effective or safe. In particular, the evidence that led to selection is statistically different from fresh confirmatory evidence collected after the transition. This distinction determines which stagewise p-values can be used without additional adjustment.
For candidate \(i\), a rule might use an estimated efficacy effect \(\hat{\theta}_{E,i}\), an efficacy p-value \(p_{E,1i}\), an estimate of excess harm \(\hat{\theta}_{H,i}\), and a dose-response statistic. The subscripts identify the endpoint and stage. Before the trial, specify the efficacy direction and estimand, the safety measure and unacceptable margin, the dose-response contrast, and the rule for ties, missing results, and no eligible candidate. Correlation among these outcomes is expected in many trials; it does not make a prespecified rule invalid, but it does mean that informal interpretations of each criterion as an independent piece of evidence may be misleading.
Constructing a Selection Rule
A useful design separates eligibility gates from ranking. For example, first remove doses that fail a safety constraint, then require a minimum level of efficacy evidence among the remaining doses, and finally choose among eligible doses using a prespecified ranking criterion. Dose-response evidence can inform that ranking or act as a separate gate. These choices are design decisions, not universal statistical rules.
Safety criteria should be stated in a direction and scale that can be evaluated. “No significant safety difference” is not a safety guarantee: a study may have too little information to detect a clinically important increase in harm. A more explicit rule might require the upper confidence bound for excess risk to be below a prespecified margin, or require that observed toxicity not exceed a prespecified limit. Such a rule guides selection; it does not replace a properly designed confirmatory safety assessment.
Likewise, a dose-response trend test can support the view that efficacy changes systematically across doses, but it does not automatically identify the best dose or establish that each dose is effective. The trend contrast and its direction must be specified, and the interpretation must match the dose pattern it tests. A rule that selects a dose after inspecting multiple endpoints, contrasts, and cutoffs should not be described as though only one test had been conducted.
Worked Example: Applying Efficacy, Safety, and Trend Criteria
Suppose a hypothetical phase 2 trial compares three doses with control. The planned selection rule excludes any dose with estimated excess serious adverse-event risk above 5 percentage points, requires a one-sided efficacy p-value below 0.20, and then selects the eligible dose with the largest estimated efficacy benefit. A prespecified dose-trend test is used as supporting evidence, not as a requirement for selection.
| Dose | Estimated efficacy benefit | One-sided efficacy p-value | Estimated excess serious-event risk |
|---|---|---|---|
| Low | 5 points | 0.16 | 1 percentage point |
| Middle | 7 points | 0.08 | 2 percentage points |
| High | 9 points | 0.03 | 7 percentage points |
The high dose fails the safety gate because \(7\% > 5\%\), despite having the largest estimated benefit and smallest efficacy p-value. The low and middle doses meet the stated safety and efficacy criteria: their efficacy p-values are both below 0.20, and their estimated excess risks are both at or below 5 percentage points. The middle dose has the larger estimated benefit of these eligible doses, 7 rather than 5 points, so the rule selects it. Suppose the prespecified trend test gives \(p=0.04\); that result supports a dose-related pattern, but under this rule it does not undo the high dose’s safety failure. The selected dose is a phase 2 decision, not a confirmatory finding of efficacy.
Which Stagewise P-Values Survive Selection?
Let \(S\) denote the selected candidate, and let \(F_1\) represent all phase 2 information available at the transition. If stage 2 uses new, independent increments, then the choice \(S\) is determined before the stage 2 data are observed. For each candidate \(i\), let \(p_{2i}\) be a stage 2 p-value for its prespecified confirmatory efficacy hypothesis. Under that candidate’s null hypothesis, the required validity condition is
This conditional super-uniformity means that, even after knowing the phase 2 results and which candidate was chosen, the stage 2 p-value remains valid under the corresponding null. It is the key reason a stage 2-only test can support a confirmatory claim for a selected candidate. The conclusion relies on genuinely independent stage 2 information and a valid stage 2 analysis under the null, not merely on labeling the later observations “stage 2.”
By contrast, a stage 1 p-value for the selected candidate is generally not uniform conditional on selection. If the design tends to choose candidates with the smallest efficacy p-values, the selected p-value will tend to be unusually small even when all candidates satisfy their null hypotheses. Selection based on correlated safety or dose-response results can also change the conditional distribution of the efficacy p-value. The fact that each candidate’s unselected stage 1 p-value was valid does not make the selected one valid for an unadjusted confirmatory combination test.
A Short Justification for Stage 2 Validity
The following argument shows why selecting one candidate does not, by itself, invalidate a valid stage 2 p-value. Assume that \(S\) is determined entirely by \(F_1\), and that the conditional super-uniformity condition above holds for every candidate under its own null hypothesis. If the selected candidate’s null is true, then
For each term, the event \(S=i\) is known from stage 1, so the conditional validity of \(p_{2i}\) bounds the probability of a small stage 2 p-value by \(u\). The last sum is at most one because only one candidate is selected. Thus a claim about the selected candidate based on its stage 2 p-value alone has type I error no greater than \(u\), under the stated assumptions. If the design instead intends to make several claims, or to test several endpoints, the appropriate multiplicity control is still required.
Worked Example: Testing the Selected Dose With Stage 2 Data
Suppose the middle dose is selected by the phase 2 rule above. In a hypothetical independent confirmatory stage, its prespecified efficacy analysis produces a one-sided test statistic \(z=2.05\). Under the null and the planned large-sample normal approximation, the stage 2 p-value is
At a prespecified one-sided significance level of \(0.025\), \(0.0202<0.025\), so the stage 2 result rejects the efficacy null for the selected dose, subject to the assumptions and analysis plan. The interpretation is that stage 2 data provide statistically significant evidence in the prespecified benefit direction at that level. This result does not claim that the probability the null is true is 2.02%, and it does not turn the phase 2 selection estimates into confirmatory estimates.
The relevant conditions are that the stage 2 observations provide independent increments from those used for selection, the treatment and efficacy hypothesis are defined consistently with the planned claim, and the test statistic and p-value are valid under the null. If the stage 2 analysis changes the endpoint, population, or estimand, this simple validity argument no longer suffices.
Combination Tests and the Role of Stage 1 Evidence
A combination test can use information from both stages, but it must use stagewise p-values and a calibration that remain valid after the actual selection rule. For example, an inverse-normal statistic may be written as
where \(p_1\) and \(p_2\) are one-sided p-values in the same direction and the weights are specified by the design. The formula alone does not guarantee validity. If \(p_1\) is the unadjusted phase 2 p-value for whichever candidate was selected using phase 2 evidence, treating it as an ordinary uniform input can be anti-conservative. A design may instead use an appropriate selection adjustment, conditional-error approach, or other prespecified calibration. The selection rule and the confirmatory test must be considered together.
The main alternatives have different costs. A stage 2-only test is simple and can retain validity under the independence conditions above, but it does not use phase 2 efficacy information to increase confirmatory power. A properly calibrated combination test can use evidence from both stages, but its calibration must account for selection and any multiplicity. A fixed-treatment design avoids treatment-selection bias in the confirmatory comparison, but gives up the opportunity to carry forward the most promising candidate. The earlier material on confirmatory validity after treatment selection provides the broader context for these options; the central point here is that a valid stage 2 p-value and an unadjusted selected stage 1 p-value are not interchangeable.
| Evidence used for the selected-treatment claim | Validity after selection | Design requirement |
|---|---|---|
| Stage 2 p-value alone | Can remain valid conditional on selection | Independent stage 2 increments and a valid prespecified test |
| Unadjusted selected stage 1 p-value | Not generally valid as an ordinary uniform p-value after selection | Do not use without a valid selection-aware method |
| Stage 1 and stage 2 combination | Depends on the calibration; not guaranteed by the combination formula | Prespecified adjustment or error-control method accounting for selection |
| Several selected-treatment claims | Individual stage 2 validity does not automatically control family-wise error | Multiplicity control for the planned claims |
Diagnostics, Failure Modes, and Common Pitfalls
Before interpreting confirmatory results, check that the implemented selection rule matches the prespecified one: the same eligibility thresholds, endpoint directions, dose comparisons, and tie-breaking procedure should have been used. Record which candidates failed each gate and which outcomes determined the final choice. If the rule was changed after seeing the data, the original calibration may not apply.
The stage 2 validity argument also depends on stage 1 selection being fixed before stage 2 information is examined and on the stage 2 test having the stated null distribution. Reusing information across stages, making unplanned changes to the endpoint or analysis, or adapting the confirmatory test based on stage 2 results can undermine that argument. When the data structure or design does not provide independent increments, the relevant dependence must be incorporated into a suitable error-control method rather than assumed away.
- Pitfall: “Each phase 2 p-value was valid, so the selected one is valid.” The selected p-value has been chosen using the observed results and is generally no longer uniform under the null. Use stage 2 data alone or a selection-aware method.
- Pitfall: “The selected dose passed a safety screen, so it is safe.” A selection gate is not proof of safety. Interpret it only as the stated rule allows, and retain appropriate confirmatory safety assessment.
- Pitfall: “A significant dose trend confirms the selected dose.” A trend test addresses its specified dose-response hypothesis; it is not automatically a test of the selected dose’s efficacy claim.
- Pitfall: “A valid stage 2 p-value permits any number of claims.” It supports the stated selected-treatment test under its conditions. Multiple endpoints or multiple confirmatory claims still require planned multiplicity control.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- Why should a multi-outcome selection rule distinguish safety eligibility from efficacy-based ranking?
- Two candidate treatments have valid phase 2 p-values under their null hypotheses. The design selects the candidate with the smaller p-value. Is that selected p-value generally uniform under the null?
- What condition makes a selected treatment’s stage 2 p-value valid conditional on the phase 2 selection?
- A trial selects one dose using phase 2 efficacy, safety, and trend evidence, then reports an unadjusted inverse-normal combination of that dose’s phase 2 and stage 2 p-values. What is the concern?
Solutions. (1) Eligibility and ranking answer different questions: a dose can have strong efficacy evidence yet fail a prespecified safety constraint, so the rule should say whether it is excluded before ranking. (2) No. Selection favors smaller values; for example, if two independent null p-values are uniform, the probability their minimum is below 0.05 is \(1-(1-0.05)^2=0.0975\), not 0.05. (3) Stage 2 must provide independent increments, and its p-value must be super-uniform under the null conditional on all phase 2 information used to select the dose. (4) The selected stage 1 p-value may be biased downward by the selection, so the combination statistic is not automatically calibrated; use a prespecified selection-aware adjustment or rely on a valid stage 2-only test.