Evidence From Separate Trial Stages

Why Keep Trial-Stage Evidence Separate?

In an adaptive clinical trial, the design may allow a prespecified decision after an interim stage: for example, to stop for futility, change the planned sample size, or continue recruitment. The final analysis may therefore involve data collected under different stages of the trial. One way to preserve the evidence from each stage is to summarize its data separately and combine the resulting stagewise p-values using a prespecified test.

This approach does not mean that the stages are separate trials, nor that any two p-values can simply be put together. The stagewise analyses must address the intended treatment question, and the final combination rule must control the probability of a false rejection under its stated assumptions. The distinction is useful when the design or analysis changes between stages and participant-level data are not to be pooled for the final test.

Definition: A trial stage is a prespecified segment of trial conduct whose data are analyzed as a unit for the purpose of an interim decision or final inference. Stagewise evidence is the inferential summary from that segment, commonly a p-value calculated from that stage’s data under a specified null hypothesis and analysis. A combination test applies a prespecified function to stagewise p-values and compares the resulting statistic with a threshold calibrated to control the test’s type I error.

What a Stagewise P-Value Represents

A stagewise p-value is calculated using the observations assigned to that stage, its analysis plan, and the relevant null hypothesis. For a one-sided test of whether a treatment improves an outcome, a small stagewise p-value indicates that the stage’s data are relatively incompatible with the null in the direction of improvement. It is not the probability that the null hypothesis is true, and it is not by itself the final trial conclusion.

For a combination test, each stagewise p-value must be valid for the hypothesis being tested: under that null, its probability of being at or below any value $u$ between 0 and 1 must be no greater than $u$. In an adaptive design, this calibration must account for how the stages were defined and how decisions were made. The relevant joint behavior of the p-values also matters for many combination rules. For example, independence can justify some calibrations, while other rules remain valid under dependence.

A useful general form is $C(p_1,\ldots,p_K)$, where $p_i$ is the p-value from stage $i$, $K$ is the number of stages, and $C$ is a prespecified combination function. The final test rejects when the combination statistic falls in a rejection region calibrated for significance level $\alpha$. Equivalently, a calibrated combined p-value can be reported and compared with $\alpha$. The function and its calibration—not the mere act of combining numbers—make the test valid.

A Simple Combination Rule and Its Justification

For illustration, a conservative rule that works with any dependence among $K$ valid stagewise p-values is the Bonferroni combination rule. It rejects if at least one stagewise p-value is no greater than $\alpha/K$. The corresponding combined p-value is the minimum of 1 and $K$ times the smallest stagewise p-value. This rule is not the only possible combination test, but it makes the calibration principle explicit.

p_{\mathrm{comb}}=\min\left(1,\ K\min_{1\leq i\leq K}p_i\right).

The rule requires that each $p_i$ be a valid p-value for the same global null being tested. It does not require independent stagewise p-values. Its conservatism can be a cost: the rule protects the type I error rate even when the p-values are dependent, but may have less power than a rule calibrated to a known joint distribution.

Justification: Under the global null, each valid stagewise p-value satisfies $P(p_i\leq u)\leq u$. Rejection by the Bonferroni rule means that at least one $p_i\leq\alpha/K$. By the union bound, $$ P(\text{reject})=P\left(\bigcup_{i=1}^{K}\{p_i\leq\alpha/K\}\right) \leq \sum_{i=1}^{K}P(p_i\leq\alpha/K) \leq K\frac{\alpha}{K}=\alpha. $$ Thus the type I error is at most $\alpha$, regardless of dependence among the stagewise p-values. Also, $P(p_{\mathrm{comb}}\leq\alpha)\leq\alpha$, which is the defining validity condition for a combined p-value.

Worked Examples

Worked Example: Calculating Separate Stage Evidence

Consider a hypothetical two-stage randomized trial of a treatment intended to improve a continuous symptom score. Larger values of the treatment-minus-control difference mean greater improvement. The one-sided null is that the treatment does not improve the score. Stage 1 includes 60 participants and stage 2 includes 40 different participants. Suppose the stage-specific estimates and standard errors are 2.4 points and 1.2 points in stage 1, and 0.8 points and 1.0 point in stage 2. For illustration, assume the prespecified stagewise analyses use a standard normal test and yield valid p-values under the null.

For stage 1, the test statistic is $z_1=2.4/1.2=2.00$, giving the one-sided p-value $p_1=P(Z\geq 2.00)\approx0.0228$, where $Z\sim N(0,1)$. For stage 2, $z_2=0.8/1.0=0.800$, so $p_2=P(Z\geq0.800)\approx0.2119$. The results preserve two distinct pieces of evidence: relatively strong evidence in stage 1 and weaker evidence in stage 2. Neither p-value alone is the final two-stage test.

If the trial prespecified the two-stage Bonferroni combination rule at $\alpha=0.05$, then $p_{\mathrm{comb}}=\min(1,2\min(0.0228,0.2119))=0.0456$. Since $0.0456<0.05$, this rule rejects the global null at the 5% level. The conclusion is about the prespecified combined test, not a claim that the second stage independently demonstrated benefit.

Worked Example: An Adapted Second Stage

In a hypothetical trial, investigators plan an interim review after stage 1. A prespecified rule allows additional participants to be enrolled if the trial continues. The first stage yields $p_1=0.018$; following continuation, the second stage yields $p_2=0.120$. Assume both p-values are valid for the same global null under the trial’s prespecified design, and use the two-stage Bonferroni rule at $\alpha=0.025$.

The combined p-value is $p_{\mathrm{comb}}=\min(1,2\times\min(0.018,0.120))=0.036$. The rejection threshold is $0.025$, and $0.036>0.025$, so the test does not reject the null at the 2.5% level. The result remains a valid test under the stated validity assumptions, even if the stagewise p-values are dependent. However, the first-stage p-value cannot simply be treated as the final result after an interim continuation decision; the prespecified combination rule determines the final decision.

Worked Example: Three Stages With Unequal Evidence

Suppose a hypothetical trial has three non-overlapping stages, with valid p-values $p_1=0.080$, $p_2=0.040$, and $p_3=0.300$ for the same global null. The analysis plan specifies the Bonferroni combination test at $\alpha=0.05$. There are $K=3$ stages, so each stagewise rejection threshold is $\alpha/K=0.05/3\approx0.0167$.

The smallest stagewise p-value is $0.040$, which is greater than $0.0167$. Equivalently, $p_{\mathrm{comb}}=\min(1,3\times0.040)=0.120$. Because $0.120>0.05$, the combination test does not reject the null. The stagewise results can still be reported individually, but the test’s conclusion follows from the calibrated three-stage rule. Counting the number of p-values below 0.05 would not be an appropriate substitute.

What Separate Evidence Preserves—and What It Does Not

A combination test retains stage-level inferential summaries rather than requiring a single participant-level analysis across all stages. This can be valuable when the stages differ in planned sample size or other design features, or when the final analysis must use stagewise outputs. It also makes the role of each stage visible: the p-values can be reported alongside the final combined test.

The trade-off is that p-values compress data. They generally do not preserve the effect estimates, standard errors, covariate information, or other participant-level detail needed for a pooled estimate and its confidence interval. A combined p-value answers a testing question under its null and calibration; it does not automatically provide an overall effect estimate. Reporting stage-specific estimates and uncertainty, where appropriate, can therefore complement the final test.

A direct analysis of all participant-level data is a different approach, not a synonym for a combination test. It can use more of the data structure, but its validity depends on an analysis model and a design that account for adaptation and stage differences. Conversely, a combination test can be useful when stagewise inference is the intended basis for the final test, but it may lose information and power relative to an appropriate model-based analysis. The choice should be prespecified and matched to the scientific estimand and trial design.

Diagnostics and Common Failure Modes

Before interpreting a combined result, check that the stage boundaries, hypotheses, direction of testing, and stagewise analyses match the plan. Confirm that each p-value uses the data assigned to its stage and that any changes made after interim review are covered by the calibration argument. A mismatch between the stagewise test and the global null, unplanned selection of the most favorable analysis, or reuse of observations across stages can invalidate a claimed combination test.

Another diagnostic is to examine the assumptions behind the combination rule. If the rule’s calibration requires independent p-values, but stagewise data or decisions create dependence, that calibration may no longer control type I error. A rule justified under arbitrary dependence avoids that particular requirement, though it may be more conservative. In either case, checking the validity of each stagewise p-value remains essential.

Mistake: “The average of the stage p-values is the combined p-value.” An arithmetic average is not generally calibrated as a p-value. Use a prespecified combination function with a justified null distribution or an error-control argument.
Mistake: “Each stage has a p-value below 0.05, so the final test is automatically significant at 0.05.” Multiple stages create multiple opportunities for a small value. Apply the prespecified combination rule and its threshold.
Mistake: “Valid p-values can always be combined in any way.” Marginal validity alone does not validate every function of the p-values. The combination rule must be calibrated for the relevant joint behavior or justified by an argument that does not require a particular dependence structure.
Mistake: “A combined p-value is an overall effect estimate.” It is evidence for a hypothesis under a specified test, not a treatment-effect estimate with clinical units.

Key takeaway: Stagewise evidence consists of calibrated summaries from the separate trial stages. A valid combination test requires a prespecified rule, stagewise p-values valid for the intended null, and a justified calibration. It combines inferential evidence; it does not, by itself, pool participant-level data or estimate an overall treatment effect.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

What does a stagewise p-value summarize, and why is it not automatically the final trial result?
For two valid stagewise p-values of 0.030 and 0.400, calculate the Bonferroni combined p-value and state the decision at $\alpha=0.05$.
Does the Bonferroni combination justification require independent stagewise p-values? Explain.
Why is the arithmetic mean of two stagewise p-values not automatically a valid combined p-value?
Name one kind of information a combination test may not retain that a participant-level analysis can use.

Solutions. 1. It summarizes the evidence in the data assigned to one stage under a specified analysis and null; the final result depends on the prespecified rule that combines stages. 2. $p_{\mathrm{comb}}=\min(1,2\times0.030)=0.060$; since $0.060>0.05$, do not reject at the 5% level. 3. No. The union-bound justification controls type I error under arbitrary dependence, provided each stagewise p-value is valid for the global null. 4. An arbitrary function of valid p-values need not itself have a calibrated null distribution or error-control guarantee. 5. For example, participant-level effect estimates and covariate information may be compressed or unavailable when only stagewise p-values are combined.