Valid Stagewise P-Values After Treatment Selection

Why the Stage of the Data Matters

At a phase boundary, a design may select one treatment or dose from several candidates for further study. If the confirmatory test then uses only data collected after that choice, its p-value can have a different and useful status from a p-value based on all accumulated data. The key question is not simply whether treatment $j$ was selected using earlier outcomes. It is whether the confirmatory-stage p-value is valid conditional on the information and decision that led to testing treatment $j$.

This tutorial concerns a single selected-treatment claim tested with stage 3 data. It does not make the stage 1 evidence disappear: stage 1 may guide selection and contribute to the design. But a p-value calculated from stage 3 alone does not combine the stages or quantify the total evidence across them.

Definition: Let $J$ be the treatment selected at the phase boundary, let $S$ denote all stage 1 information used in the decision, and let $p_{j,3}$ be the p-value for treatment $j$ calculated using stage 3 data only. The selected-treatment stagewise p-value is $p_{J,3}$. It is conditionally valid for the null hypothesis $H_{0j}$ if, for every parameter value satisfying $H_{0j}$, and for almost every possible $S$ for which $J=j$, $$ P(p_{j,3}\leq u\mid S,J=j,H_{0j})\leq u \qquad\text{for every }0\leq u\leq 1. $$

The inequality is called conditional super-uniformity. It says that, under the null, the conditional probability of obtaining a p-value no larger than any threshold $u$ is at most $u$. Exact conditional calibration gives equality; conservative tests give an inequality. The condition must hold for the actual rule used to select $J$, not merely for a hypothetical fixed-treatment analysis.

Constructing a Stage 3 P-Value

A simple construction uses a prespecified one-sided test of a treatment effect in the selected population. Suppose the stage 3 estimator for the treatment-minus-control effect for treatment $j$ is $\hat{\theta}_{j,3}$, with standard error $SE_{j,3}$. Under a normal approximation, testing $H_{0j}:\theta_j\leq 0$ against $H_{aj}:\theta_j>0$ gives $$ Z_{j,3}=\frac{\hat{\theta}_{j,3}}{SE_{j,3}}, \qquad p_{j,3}=1-\Phi(Z_{j,3}), $$ where $\Phi$ is the standard normal cumulative distribution function. The formula is appropriate when, under the boundary null $\theta_j=0$, the standardized statistic has a standard normal distribution, or when that approximation is sufficiently accurate for the planned inference. For a composite one-sided null, the largest rejection probability over values $\theta_j\leq0$ must also be controlled; typically the boundary is least favorable for this upper-tail test.

The stage 3 standard error and test statistic must be calculated according to the prespecified analysis for the stage 3 population and endpoint. Randomization, outcome ascertainment, and the variance model must support the reference distribution. If the test uses an estimated variance or a different outcome model, its appropriate reference distribution replaces the normal one. A nominal p-value from a model that is not calibrated under the null does not become valid merely because its observations were collected later.

Worked Example: Testing the Selected Treatment With New Data

Suppose stage 1 selects treatment A for a phase 3 comparison. The phase 3 endpoint is a continuous symptom score, with lower scores better, so define the treatment effect as control mean minus treatment mean; positive values favor treatment. In the independent phase 3 cohort, the estimated effect is 2.40 score points, and its standard error is 1.20 points. Assume the prespecified large-sample model gives $Z_{A,3}\sim N(0,1)$ at the null boundary and that the test is one-sided.

The stage 3 statistic is $$ Z_{A,3}=\frac{2.40\text{ points}}{1.20\text{ points}}=2.00. $$

Therefore the stagewise p-value is $$ p_{A,3}=1-\Phi(2.00)=1-0.9772=0.0228. $$

The result is evidence against the no-benefit boundary at the 0.05 level using phase 3 data alone. Under the stated model and null, the one-sided p-value is conditionally calibrated only if, given the stage 1 information that selected A, the phase 3 statistic retains the stated null distribution. The numerical calculation does not establish that design assumption; it must be justified from how the trial is run and analyzed.

Why Selection Can Be Harmless for a Stagewise P-Value

Selecting the most promising treatment tends to make its stage 1 estimate unusually favorable, even when all treatments have no effect. That selection phenomenon does not, by itself, make a p-value based solely on later data too small. If the stage 3 data are generated and analyzed so that their null distribution remains calibrated conditional on stage 1 information and the selection decision, the selected treatment's stage 3 p-value remains valid.

The conditional requirement is important. Mere chronological separation is not enough. Stage 3 enrollment, randomization, endpoint definition, sample size, or analysis might depend on stage 1 results. Such adaptations are compatible with conditional validity only if the resulting stage 3 p-value accounts for them and still satisfies the conditional bound. A cohort described as “new” is not automatically independent in the inferential sense required by the test.

Worked Example: Averaging Over Stage 1 Information

Suppose two treatments, A and B, may be selected. Among stage 1 histories that select A, imagine two equally likely types: for one type, the conditional chance that A's stage 3 p-value is at most 0.05 is 0.04; for the other, it is 0.06. Assume $P(J=A)=0.60$. For B, assume the corresponding conditional chance is 0.05 for every history selecting B, and $P(J=B)=0.40$. All these probabilities are under the relevant treatment nulls.

For A, the conditional probability must be averaged over the stage 1 histories that selected A: $$ E[P(p_{A,3}\leq0.05\mid S,J=A,H_{0A})\mid J=A,H_{0A}] =\frac{0.04+0.06}{2}=0.05. $$ For B, that conditional average is 0.05. Thus the probability of a false rejection of the selected treatment, when both nulls hold, is $$ P(J=A)(0.05)+P(J=B)(0.05) =(0.60)(0.05)+(0.40)(0.05)=0.050. $$

The A-specific conditional rate is not at most 0.05 for every history in this illustration: one type has rate 0.06. Consequently, this example demonstrates only that the average conditional rate is 0.05; it does not establish the stronger conditional super-uniformity condition, which must hold for almost every relevant stage 1 history. If that stronger condition held, the same averaging argument would guarantee control.

Type I Error Control After Selection

The justification can be stated for any number of candidate treatments. Let $H_{0j}$ be the null hypothesis for treatment $j$. Suppose the final confirmatory claim is only about the selected treatment $J$. If, for every true $H_{0j}$, the stagewise p-value is conditionally super-uniform given $S$ and $J=j$, then testing the selected claim at level $\alpha$ controls the probability of falsely rejecting a true selected-treatment null at no more than $\alpha$.

To see this, partition a false rejection by the identity of the selected treatment. For a fixed configuration of true and false nulls, the probability of a false rejection is

\begin{aligned} P(\text{false rejection}) &=\sum_{j:H_{0j}\text{ true}}P(J=j,\ p_{j,3}\leq\alpha)\\ &=\sum_{j:H_{0j}\text{ true}}P(J=j) E\!\left[ P(p_{j,3}\leq\alpha\mid J=j,\ S,\ H_{0j}) \mid J=j,\ H_{0j} \right]\\ &\leq\sum_{j:H_{0j}\text{ true}}P(J=j)\alpha\\ &\leq \alpha. \end{aligned}

The expectation is essential: the inner conditional probability can vary with the stage 1 history, so it is not generally a single scalar to multiply by $P(J=j)$ until it has been averaged over those histories conditional on selection. Conditional super-uniformity bounds that inner probability by $\alpha$ almost surely. Finally, the probabilities of selecting the true-null treatments sum to at most one. This gives strong control for the single selected claim, including when some candidate treatments are truly effective, provided the stated conditional validity holds for each true null.

Key result: Selection from stage 1 does not inflate the type I error of a selected-treatment test based only on stage 3 data when each stage 3 p-value is conditionally super-uniform under its own null, given the full selection-stage information and the event that the treatment was selected. The result concerns the single selected claim; it does not by itself authorize multiple confirmatory claims or unadjusted testing of several endpoints.

When the Condition Fails

Several design features can invalidate the argument or change the claim being tested. The relevant question in each case is whether the conditional null distribution used to calculate $p_{J,3}$ remains correct.

Reusing stage 1 outcomes without adjustment. A pooled estimate can inherit the favorable selection that led to choosing the treatment. Its ordinary fixed-treatment p-value need not satisfy conditional super-uniformity.
Stage 3 design changes the null distribution. If sample size, allocation, or analysis is adapted using stage 1 information, the p-value calculation must account for that rule. An ordinary fixed-design reference distribution may no longer be calibrated conditionally.
Data or decisions are not separated as assumed. Shared participants, overlapping measurements, or selection based on outcomes that also enter the purported stage 3 analysis can create dependence. The conditional distribution must reflect the actual data structure.
The estimand changes at the boundary. If stage 1 selects a biomarker subgroup but the phase 3 test targets a different population or endpoint, the p-value may be valid for the new target while failing to support the intended cross-stage claim. The tested estimand must be specified and preserved.
More than one confirmatory claim is made. Conditional validity for each candidate's stage 3 p-value does not automatically control error when several treatments, subgroups, or endpoints can be declared successful. The claim set and multiplicity procedure must be addressed separately.

Worked Example: Why the Selected Stage 1 P-Value Is Not Stagewise Valid

Under a global null, suppose stage 1 produces independent standard normal statistics $Z_{A,1}$ and $Z_{B,1}$ for two treatments. The trial selects the treatment with the larger statistic. If it reports the ordinary one-sided stage 1 p-value for that selected treatment, its value is $1-\Phi(\max(Z_{A,1},Z_{B,1}))$.

Each unselected, prespecified one-sided p-value is uniform under the null. But the chance that at least one of the two p-values is at most 0.05 is $$ P(\min(p_{A,1},p_{B,1})\leq0.05) =1-P(p_{A,1}>0.05,\ p_{B,1}>0.05) =1-(0.95)^2=0.0975. $$

Thus the selected stage 1 p-value is not a level-0.05 test under this selection rule. In contrast, if the selected treatment is tested using a new stage 3 statistic whose p-value is conditionally super-uniform given the stage 1 history and selection, its conditional rejection probability is at most 0.05. This distinction is about which data generate the p-value, not a claim that all uses of later data are automatically valid.

Diagnostics and Practical Alternatives

Before relying on a stagewise p-value, check the selection algorithm, analysis population, endpoint, and stage 3 test as one complete design. In particular, document what stage 1 information is included in $S$, confirm that the selection rule is fully represented, and verify the null calibration conditional on that information. Simulation can diagnose a proposed design: generate data under relevant null configurations, apply the complete selection and stage 3 procedure, and estimate rejection rates within selection histories or prespecified strata. A low overall rejection rate alone may conceal conditional miscalibration in some histories.

A stage 3-only test is not the only possible approach. A design can instead control selection error by adjusting tests across candidate treatments, or use a prespecified procedure that combines evidence from both stages. These alternatives may use information more efficiently or support different claims, but they require their own calibration and assumptions. A stagewise p-value is comparatively transparent when the confirmatory claim is for one selected treatment and the stage 3 data provide a conditionally valid test. Its limitation is equally clear: it does not use stage 1 outcomes as confirmatory evidence in the p-value itself.

Worked Example: Checking a Proposed Design

A hypothetical trial selects one of three doses using stage 1 response data. At the boundary, stage 3 enrolls new participants, randomizes them between the selected dose and control, and measures the same primary endpoint in the same target population. The stage 3 sample size is fixed in advance, and the planned test is a one-sided randomized comparison with a valid null reference distribution.

These features support conditional validity: the selected dose is fixed once $S$ is observed; the stage 3 participants supply the test data; and the stage 3 analysis has a calibrated null distribution for each possible selected dose. The protocol must still ensure that the randomization and analysis are implemented as specified, and that the endpoint, population, and treatment contrast remain the same target claim. If the sample size were instead changed in response to stage 1 outcomes, the fixed-design calibration would need to be reassessed rather than assumed.

Common Mistake

Wrong reasoning: “The selected treatment had the best stage 1 result, so any p-value for it must be biased downward, including one computed from stage 3 alone.” This confuses selection-induced optimism in the data used to choose the treatment with the conditional distribution of genuinely later test data.

Correction: Selection makes the stage 1 result for the winner atypical under the null. It does not necessarily make a stage 3-only p-value atypical. The latter is valid if its conditional null distribution, given the stage 1 information and selection, is calibrated. Conversely, simply labeling observations “stage 3” is insufficient if the test reuses selection data or ignores design adaptations that alter its conditional distribution.

Check Your Understanding

Answer each question, then compare with the solutions.

What does conditional super-uniformity require of $p_{j,3}$ under $H_{0j}$?
In a two-treatment trial, $P(J=A)=0.70$ and $P(J=B)=0.30$. If the conditional rejection probability at level 0.05 is at most 0.05 for every stage 1 history under either true null, what bound follows for the false-rejection probability of the single selected claim when both nulls are true?
Why is an ordinary stage 1 p-value for the treatment with the largest of two independent null statistics not generally a valid 0.05 p-value?
Name two design features that must be checked before treating a stage 3-only p-value as conditionally valid.

Solutions. (1) For every threshold $u$ between zero and one, and almost every relevant stage 1 history, $P(p_{j,3}\leq u\mid S,J=j,H_{0j})\leq u$. (2) The false-rejection probability is at most $0.70(0.05)+0.30(0.05)=0.05$. (3) Selection favors an unusually large stage 1 statistic; for two independent null p-values the chance at least one is at most 0.05 is $1-0.95^2=0.0975$. (4) Examples include whether stage 3 data overlap with stage 1 data, whether sample size or allocation adapts to stage 1, whether the endpoint and estimand are unchanged, and whether the stage 3 reference distribution is calibrated for the actual randomization and analysis.