Why Selection Changes the Evidence
A seamless trial may use Phase 2 results to choose which treatment proceeds to Phase 3. It is tempting to combine the selected treatment’s Phase 2 p-value with its Phase 3 p-value using a standard method for independent evidence. The problem is not necessarily that the stages share patients: they may be entirely separate. The problem is that the Phase 2 p-value helped determine which treatment was carried forward. After selection, that p-value is not distributed like an ordinary unselected null p-value.
A p-value used once for a prespecified test is calibrated under its null hypothesis: repeatedly applying the test when that null is true gives a uniform p-value under a continuous test statistic. Selecting a treatment because its p-value is unusually small changes that distribution. The selected value tends to be smaller even when all treatments have no effect. Treating it as an unselected p-value can therefore make combined evidence appear stronger than it is.
This issue is distinct from dependence caused by reusing a shared control group, which is addressed in earlier material on shared controls. Even if Phase 2 tests are mutually independent and Phase 3 uses new participants, selection alone can invalidate a naive combination. The adjustment must reflect both the selection rule and the dependence structure of the tests used in that rule.
A Simple Selection Adjustment
Suppose \(K\) treatments are tested in Phase 2, and their one-sided p-values are independent and continuous under the global null that none is effective. The design selects the treatment with the smallest p-value. Let \(M\) denote that selected p-value. For any \(u\) between 0 and 1, all \(K\) p-values exceed \(u\) with probability \((1-u)^K\). Therefore,
The selected p-value \(M\) is not uniform: its cumulative probability of being at most \(u\) exceeds \(u\). Applying its null cumulative distribution function gives the adjusted value \(P(M\leq M_{\mathrm{obs}})\). Under the stated assumptions, this transformation is uniform under the global null.
This derivation is deliberately specific. If Phase 2 comparisons share a control, the p-values will generally be dependent, so the formula for independent p-values does not apply. The appropriate selection-adjusted distribution must then be derived from, or simulated under, the joint null distribution and the actual selection rule. The same principle applies when selection uses a different criterion, such as a posterior probability, an estimated effect threshold, or a rule involving safety and efficacy.
Worked Example: Adjusting the Smallest of Three P-Values
Hypothetical design. Three treatments are compared with separate controls at the Phase 2 selection analysis. For this illustration, assume their one-sided p-values are independent and continuous under the global null. The treatment with the smallest Phase 2 p-value is selected, and the observed minimum is \(M_{\mathrm{obs}}=0.07763184\).
With \(K=3\), the selection-adjusted p-value is
Thus, under the global null and the independent-p-value assumption, a minimum as small as the observed value or smaller occurs about 21.5% of the time when the selection rule is applied. Reporting \(0.07763\) as though it were the p-value of a treatment specified in advance would understate how often this degree of apparent evidence arises after searching across three treatments.
Why Naive Fisher Combination Can Fail
For two independent, uniform null p-values \(p_1\) and \(p_2\), Fisher’s combination statistic is \(F=-2\{\log(p_1)+\log(p_2)\}\), which has a chi-square distribution with 4 degrees of freedom under the null. This reference distribution requires both p-values to be null-calibrated and independent. An independent Phase 3 p-value does not repair a Phase 2 p-value whose distribution was altered by selection.
Worked Example: Naive and Selection-Adjusted Combination
Hypothetical data. In the three-treatment design above, the selected Phase 2 p-value is \(0.07763184\). The independent Phase 3 analysis of that selected treatment gives a one-sided p-value of \(0.100\). Assume for this illustration that Phase 3 participants are new, the Phase 3 test is prespecified, and the Phase 3 p-value is independent of all Phase 2 results under the null.
Naive calculation. If the unadjusted selected p-value is treated as uniform, the Fisher statistic is
For a chi-square random variable with 4 degrees of freedom, the upper-tail probability is \(e^{-F/2}(1+F/2)\). The naive combined p-value is therefore
That calculation appears to cross a 0.05 threshold, but its reference distribution is not valid: the selected Phase 2 p-value is not uniform under the global null.
Selection-adjusted calculation. Under the stated independent Phase 2 p-value model, use \(p_{\mathrm{sel}}\approx0.2153\) instead. The transformed Phase 2 p-value is uniform under the global null and remains independent of the Phase 3 p-value under the assumptions above.
The corresponding chi-square upper-tail probability is
The adjusted combination does not reject at 0.05. The contrast shows why the selection step must be represented in the null calibration: the unadjusted calculation would overstate the combined evidence. The numerical adjustment here is valid only for the stated selection rule and independent Phase 2 p-values; it is not a universal correction for every seamless design.
A Valid Alternative: Use Independent Phase 3 Evidence
A simple alternative is to reserve Phase 2 for selection and base confirmatory inference on the Phase 3 data alone. Suppose the Phase 3 analysis plan specifies the treatment comparison, endpoint, direction of testing, and significance level before Phase 3 outcomes are examined. If the Phase 3 data are independent of the Phase 2 selection data, then, under the null for the selected treatment, the Phase 3 p-value is uniform conditional on the selection outcome. Testing that p-value at level \(\alpha\) controls the probability of a false rejection for the selected treatment.
For a design selecting among several treatments, the conditional argument applies to each possible selected treatment: given that treatment \(j\) was selected using Phase 2 data, its independent Phase 3 test has null rejection probability at most \(\alpha\). The probability of rejecting any true null is the sum over the possible selected true-null treatments of their selection probabilities times their conditional rejection probabilities, and is at most \(\alpha\). This claim depends on the independent confirmatory data and the prespecified Phase 3 testing procedure; it does not license unadjusted additional claims about every treatment or endpoint.
Worked Example: Phase 3-Only Confirmatory Test
Hypothetical data. Phase 2 selects Treatment B. In the independent Phase 3 comparison, the estimated improvement on a clinical score is 4.2 points in the prespecified favorable direction, with standard error 2.0 points. Assume the normal approximation for the standardized Phase 3 statistic is adequate, and the test is one-sided with \(\alpha=0.025\).
The Phase 3 statistic and p-value are
Since \(0.0179<0.025\), the prespecified Phase 3-only test rejects its null hypothesis at the 2.5% level. This conclusion is based on the independent Phase 3 comparison, not on adding the selected Phase 2 p-value. The standard error and test statistic must be appropriate for the Phase 3 design; for example, the analysis must respect its randomization, endpoint, and any prespecified adjustment.
Calibration Choices, Diagnostics, and Limits
When Phase 2 evidence is to contribute to a final test, a valid procedure must calibrate the final statistic under the full process that generated it: the Phase 2 data, the selection rule, and the Phase 3 data. One approach is to derive the conditional null distribution given the selection event and compute a conditional p-value. Another is to simulate the complete design under the null, repeating treatment selection in every simulation, and calibrate the observed final statistic against that simulated distribution. Such simulation must represent relevant dependence, nuisance parameters, missingness, and the actual adaptation rules; otherwise it merely reproduces a misspecified calculation.
A practical audit can proceed in sequence:
Specify exactly which Phase 2 estimates, p-values, safety results, or other criteria determine the treatment carried forward.
Check whether the selected statistic remains calibrated and whether tests are independent or correlated under the null.
Use a selection-adjusted combination procedure, or a prespecified independent Phase 3-only test, with its assumptions stated.
Under relevant null configurations, verify that the probability of a false positive is no greater than the planned significance level.
Useful warning signs include a final test whose derivation assumes a Phase 2 p-value is uniform despite selecting the smallest p-value; a simulation that holds the selected treatment fixed instead of repeating selection; and a calculation that assumes independent Phase 2 tests despite a shared control. Check as well that the Phase 3 p-value is genuinely independent of the selection data. Reusing patients or outcomes, allowing Phase 3 data to influence selection, or repeatedly examining unadjusted Phase 3 results can invalidate the simple conditional argument.
| Approach | What it uses | Key requirement |
|---|---|---|
| Naive combination of selected Phase 2 and Phase 3 p-values | Both stages, with the unadjusted selected Phase 2 p-value | Generally invalid when selection changes the Phase 2 p-value distribution |
| Selection-adjusted combination | Both stages, calibrated for selection | The joint null distribution and selection rule must be correctly represented |
| Independent Phase 3-only test | Confirmatory Phase 3 evidence | Independent Phase 3 data and a prespecified, appropriately controlled test |
Selection adjustment can reduce apparent evidence and may reduce power; that is the cost of accounting for a search. A Phase 3-only test can also be less efficient because it discards Phase 2 efficacy evidence, but it is often easier to interpret and protect. Neither approach automatically solves multiplicity across many endpoints, populations, or subsequent claims. Those testing families require their own prespecified error-control strategy.
Common Mistake
Wrong reasoning: “The Phase 3 p-value comes from new patients, so it is independent of the Phase 2 p-value. Therefore Fisher’s ordinary reference distribution is valid when I combine them.” The independence statement can be true while the conclusion is false. Fisher’s ordinary calibration also requires the Phase 2 input to be a valid uniform null p-value. Once it is selected for being small, that condition generally fails.
Correction: Either calibrate the combined statistic to account for the selection process, including the dependence among Phase 2 comparisons, or use an independent, prespecified Phase 3 test as the confirmatory test. State which data and decisions enter the procedure, and check its type I error under the complete adaptive rule.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Under the global null, three independent continuous Phase 2 p-values are generated, and the smallest is selected. What is the probability that the selected p-value is at most 0.10?
- Why does an independent Phase 3 p-value not, by itself, validate a naive Fisher combination with the selected Phase 2 p-value?
- For the example with minimum Phase 2 p-value 0.07763184 among three independent tests, what is the selection-adjusted value, approximately?
- Name two conditions needed for a Phase 3-only test to provide valid confirmatory inference after Phase 2 treatment selection.
Solutions: (1) \(1-(1-0.10)^3=1-0.729=0.271\), or 27.1%. (2) Fisher’s ordinary reference distribution requires both inputs to be null-calibrated as well as independent; selection makes the Phase 2 p-value nonuniform. (3) \(1-(1-0.07763184)^3\approx0.2153\). (4) Examples include independent Phase 3 participants and outcomes, a test and endpoint prespecified before Phase 3 outcomes are examined, and an analysis that respects the planned randomization and error-control rules.