Two Ways to Use Evidence From Multiple Stages
An adaptive trial may produce evidence in distinct stages, with an interim decision between them. After the trial, investigators can analyze participant-level observations together in one model, or combine valid p-values calculated separately by stage. These approaches can give similar answers, but they are not interchangeable by definition: they use different information and can rely on different assumptions.
A pooled-data test uses participant outcomes and covariates directly. Its validity depends on the analysis model, the estimand, and how the data were collected. A combination test starts from stagewise p-values and applies a prespecified rule whose rejection threshold is calibrated for their joint null distribution. The combination principle is useful when the trial adapts, but it does not make invalid stagewise p-values valid or erase differences in what the stages measured.
For a comparison of treatment and control, let \(\hat\theta_j\) be the estimated treatment effect at stage \(j\), and let \(V_j\) be its variance. A stagewise p-value summarizes evidence against a stagewise null using that stage's data. A pooled model instead specifies how the \(\hat\theta_j\), or the participant observations that produce them, relate across stages. In the simplest case, both stages estimate the same treatment effect using the same endpoint and a correct common model. Then pooling is natural, and a suitable combination test can preserve the same direction and much of the same evidence.
When Pooling and Stagewise Evidence Agree
Suppose the endpoint is normally distributed, treatment and control observations are independent, and the outcome standard deviation is known and equal to \(\sigma\) in both arms and both stages. Assume randomization within each stage, independent stage increments, the same endpoint and estimand throughout, and a common treatment effect. Under these conditions, pooling participants is straightforward. The pooled treatment effect is the sample-size-weighted average of the stage effects when each stage has equal treatment and control sample sizes.
A stagewise combination can also be well calibrated under the null if each stagewise p-value is valid for the stated null and the combination rule and its threshold account for the p-values' joint distribution. If the stages are independent under the null, their joint distribution is especially simple to specify. The combination need not reproduce the pooled analysis exactly: agreement depends on the combination rule and the model. But in a simple common-effect normal model, a precision-weighted summary of the stage estimates and the pooled estimate can coincide.
Worked Example: A Common Effect in Two Normal Stages
Suppose, hypothetically, that each stage enrolls 100 participants per arm, the outcome standard deviation is known to be 10 units in both arms, and the endpoint and treatment effect are the same in both stages. The estimated treatment effects are 1 unit in stage 1 and 3 units in stage 2. Assume independent observations and stages, normal outcomes, and no missing data. These assumptions permit the calculations below.
In either stage, the variance of the difference in sample means is \(10^2(1/100+1/100)=2\), so its standard error is \(\sqrt{2}\) units. The stagewise standardized statistics are \(Z_1=1/\sqrt{2}\approx0.7071\) and \(Z_2=3/\sqrt{2}\approx2.1213\). For one-sided tests of a positive treatment effect, the corresponding standard normal tail probabilities are approximately \(p_1=0.2398\) and \(p_2=0.0169\).
Across both stages, there are 200 participants per arm. The pooled effect estimate is
Its standard error is \(10\sqrt{1/200+1/200}=1\) unit, giving \(Z_{\text{pool}}=2/1=2.000\). The one-sided p-value is approximately 0.0228. In this example, the pooled estimate is also the average of the two stage estimates because stage sizes and their variances are equal. A stagewise combination procedure can be designed to use the same pattern of evidence, but its p-value depends on its specified combination rule and calibration. The pooled result alone does not establish that every possible combination test must give 0.0228.
Why a Pooled Contrast Can Be Misleading
Pooling all participants into a single unadjusted treatment-versus-control contrast can confound treatment with stage when the treatment allocation or participant mix differs across stages. This is not a failure of the combination principle; it is a mismatch between the pooled analysis and the data structure. A pooled model that includes stage can address this particular problem if its assumptions are appropriate. Stagewise tests also retain the within-stage comparisons, where treatment was randomized.
Worked Example: Stage Imbalance Changes the Unadjusted Pooled Contrast
Consider hypothetical normally distributed outcomes with known within-stage standard deviation 2 units. In stage 1, 10 treated participants have mean 11 and 90 controls have mean 10. In stage 2, 90 treated participants have mean 21 and 10 controls have mean 20. Assume independent observations, treatment randomized within each stage, and equal treatment effects across stages. Each within-stage treatment effect is 1 unit.
The unadjusted pooled treatment mean is \((10(11)+90(21))/100=20\) units. The unadjusted pooled control mean is \((90(10)+10(20))/100=11\) units. Their difference is 9 units, despite a 1-unit treatment effect in each stage. The reason is that most treated participants are from the higher-outcome stage, while most controls are from the lower-outcome stage.
A stage-adjusted estimate retains the within-stage contrasts: \(\hat\theta_1=11-10=1\) and \(\hat\theta_2=21-20=1\). Each stage contrast has variance \(2^2(1/10+1/90)=4(1/9)=4/9\) square units. With equal variances, the precision-weighted common-effect estimate is 1 unit, with variance \(1/(9/4+9/4)=2/9\) square units and standard error \(\sqrt{2/9}\approx0.4714\) units. Its standardized statistic is \(1/0.4714\approx2.121\), corresponding to a one-sided standard normal p-value of about 0.0169.
The calculation assumes the known common within-stage variance and common treatment effect. The unadjusted pooled contrast does not test that common within-stage effect; it mixes the stage outcome difference into the treatment contrast. A stage-adjusted pooled model or an appropriately calibrated stagewise combination avoids that particular mistake, though their inferential details are not identical.
Adaptation Can Change What the Overall Test Means
An interim decision may change the dose, eligibility criteria, endpoint, or participant population. A pooled analysis then needs to define how observations from the different stages contribute to its estimand. For example, pooling outcomes from two doses estimates an effect averaged over the doses actually assigned, not necessarily the effect of either dose alone. A combination test may still test a null that is meaningful across stages, but the null and alternative must be stated precisely. “No treatment effect in either stage” is not automatically the same question as “no effect for one fixed dose in the target population.”
Worked Example: A Dose Change Alters the Pooled Estimand
Suppose, hypothetically, stage 1 compares dose A with control and stage 2 compares dose B with control after an interim adaptation. Each arm has 100 participants per stage, outcomes are independent and normal, and the known standard deviation is 5 units. The estimated treatment effects are 2 units for dose A and 1 unit for dose B. Assume the stage-specific comparisons are valid for their respective dose-versus-control null hypotheses.
If the treatment groups are pooled together and treated as one group, the estimated average effect across the assigned doses is \((100(2)+100(1))/200=1.5\) units. Against the pooled control group, the standard error is \(5\sqrt{1/200+1/200}=0.5\) units, and the standardized statistic is \(1.5/0.5=3.000\). This test concerns the average effect of the mix of doses used in the trial under the stated pooling scheme. It does not estimate dose A's effect or dose B's effect separately.
For comparison, the stagewise standard errors are each \(5\sqrt{1/100+1/100}=\sqrt{0.5}\approx0.7071\) units. The stage statistics are \(2/\sqrt{0.5}\approx2.828\) and \(1/\sqrt{0.5}\approx1.414\). Their one-sided p-values are approximately 0.0023 and 0.0787. A calibrated combination test could use these stagewise p-values to test a prespecified null across the two dose comparisons. Its conclusion would depend on the specified combination rule, and it would not by itself turn the evidence into a claim about either dose's effect individually.
Validity, Diagnostics, and Limitations
A combination test requires valid stagewise p-values for the null being tested and a combination rule calibrated for the relevant joint distribution. In an adaptive design, validity of a stage 2 p-value must account for the way stage 2 was selected or conducted. Independent stage increments can support a simple calibration, but independence alone is not enough: endpoint definitions, null hypotheses, and the validity of each stagewise analysis still matter. A p-value from an invalid or mismatched stage analysis cannot be repaired merely by combining it with another.
A pooled-data test requires a suitable participant-level model and a clearly defined estimand. Check whether randomization is preserved within the factors that differ by stage, whether the analysis accounts for stage and relevant stratification variables, whether variance assumptions are credible, and whether outcomes and populations are sufficiently comparable for the intended pooled effect. Useful diagnostics include stage-by-treatment summaries, stage-specific estimates and standard errors, distributions of baseline characteristics, missingness by stage and arm, and evidence of stage-dependent measurement or outcome variance. Large differences between stage-specific effects may indicate genuine effect heterogeneity, a data-quality issue, or an estimand mismatch; they should not be hidden by a single pooled estimate.
Neither approach is universally more powerful or more robust. A correctly specified pooled model can use participant-level covariates and may estimate a common effect efficiently. A combination test can be attractive when adaptations make a single participant-level model awkward, or when the stages are analyzed separately and only valid stagewise evidence is retained. However, a combination test may discard useful information about effect size and covariate relationships. Pooling can estimate effects directly, but careless pooling can obscure stage differences or produce an answer for an unintended mixture of populations or treatments.
The following justification explains the calibration requirement. Let \(C\) be a prespecified function of stagewise p-values, and let \(c_\alpha\) be chosen so that under the null, \(P(C\leq c_\alpha)=\alpha\). If the null joint distribution used for calibration is correct, then the rejection rule \(C\leq c_\alpha\) has null rejection probability \(\alpha\). This follows directly from the threshold's definition; it does not require the stagewise p-values to be pooled into a participant-level model. If the assumed joint distribution is wrong, for example because dependence was ignored, that equality need not hold.
A common mistake is to say, “All participants were randomized, so the raw pooled treatment difference must be valid.” Randomization within stages does not guarantee that treatment and control groups have the same stage composition after pooling. The correction is to preserve the randomized comparisons, such as by including stage in the pooled model, or to use valid stagewise p-values with an appropriately calibrated combination rule. Another mistake is to interpret a combined p-value as evidence for one fixed treatment effect when the treatment or target population changed. State the null and estimand first, then interpret the test in those terms.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two independent stages estimate the same treatment effect using the same endpoint and a correct common-variance model. What makes participant-level pooling reasonable?
- In the stage-imbalance example, why is the unadjusted pooled contrast 9 units rather than 1 unit?
- A trial changes dose between stages. What does pooling both doses into one treatment group estimate, and what does it not necessarily estimate?
- Why does a combination test require both valid stagewise p-values and a calibrated combination rule?
Solutions. (1) The common endpoint, estimand, and effect model make the observations comparable, while the assumed variance model and independent sampling support the pooled analysis. (2) Treatment and control participants have different stage compositions: treated participants are mostly from the higher-outcome stage, and controls are mostly from the lower-outcome stage. The raw contrast therefore mixes stage differences with treatment differences. (3) Under the stated pooling scheme it estimates an average effect over the doses assigned; it does not necessarily estimate either dose's effect separately. (4) Invalid stagewise p-values do not provide valid evidence for the null, and an uncalibrated rule may not have the intended null rejection probability. Both conditions are needed for the stated error-rate control.