Mistake: Applying an Ordinary Pooled Test

Why Ordinary Pooling Can Fail After Selection

In a seamless phase 2/3 trial, stage-one data may help determine which treatment proceeds to stage two. A tempting analysis is to combine the selected treatment’s stage-one and stage-two observations, then apply the ordinary test that would have been used had the treatment been specified in advance. The problem is not pooling itself: it is treating selected evidence as though the treatment had been fixed before stage one. Selection tends to favor a treatment whose stage-one result looks unusually good, even when all treatments are ineffective. That favorable result is then carried into the pooled test.

The earlier material on selection bias in reported treatment effects describes why a selected estimate can be exaggerated. Here the focus is inferential: the selection can also invalidate the reference distribution used to calculate an ordinary pooled p-value. A prespecified combination test offers a way to use evidence from both stages while accounting for the selection step.

A Simple Null Example: Selection Changes the Distribution

Suppose two candidate treatments are compared with a control in stage one. For this illustration, assume their standardized one-sided test statistics \(Z_{11}\) and \(Z_{12}\) are independent standard normal random variables under the global null, and the design selects the treatment with the larger statistic. In stage two, a new independent sample gives a standard normal statistic \(Z_{2J}\) for the selected treatment \(J\) under its null. Assume equal stage information, so the ordinary pooled statistic would be

If the treatment had been fixed in advance, the numerator would be the sum of two independent standard normal statistics and \(Z_{\mathrm{pool}}\) would be standard normal. But \(Z_{1J}=\max(Z_{11},Z_{12})\), which is systematically larger than a standard normal variable. Thus the ordinary one-sided 5% critical value, 1.645, does not generally give a 5% test after this selection.

A Prespecified Combination Test

A combination test reduces each stage’s evidence to a stagewise p-value and combines those p-values according to a rule fixed in advance. To protect against selection among \(K\) candidate treatments, one simple approach adjusts the selected treatment’s stage-one p-value by Bonferroni. This is a transparent option, though it may be conservative.

The one-sided combination test rejects at level \(\alpha\) when \(Z_{\mathrm{comb}}>z_{1-\alpha}\), where \(z_{1-\alpha}=\Phi^{-1}(1-\alpha)\). Weights may reflect planned information in each stage, but they must be fixed independently of the observed stagewise outcomes. Equal weights are one possible choice; they are not required.

The validity conditions matter. Each stage-one p-value must be super-uniform under its own null: for every \(u\in[0,1]\), \(P(p_{1j}\leq u\mid H_j)\leq u\). The stage-two p-value for the selected treatment must be conditionally super-uniform under its null given all stage-one information used in selection and adaptation. This can be supported by a valid test using new stage-two data, an appropriate estimand and endpoint, and a correctly specified analysis. The stagewise p-values must use the prespecified direction of benefit. These conditions do not require a valid p-value to be exactly uniform; conservative or discrete p-values are allowed.

Why the Adjustment Controls the Selected-Null Error

The guarantee is specifically about false rejection when the selected treatment’s null is true. It is not a claim that the selected treatment’s adjusted stage-one p-value is unconditionally super-uniform under every configuration of true and false treatment effects. In particular, a false treatment can have a very small p-value, so a selected p-value that often belongs to a false null need not itself behave like a null p-value.

Here the event includes only occasions when the selected treatment’s null is true. The result holds even if some other candidates have false null hypotheses. Its short justification is the union bound restricted to the true nulls:

where \(m\leq K\) is the number of true null hypotheses. For \(u<1\), the event \(p^*_{1,j}\leq u\) implies \(p_{1j}\leq u/K\); at \(u=1\), the bound is immediate. Importantly, this argument does not apply null p-value validity to candidates whose null hypotheses are false.

For the combination test, write \(Z_1=\Phi^{-1}(1-p^*_{1,J})\) and \(Z_2=\Phi^{-1}(1-p_{2,J})\). The theorem implies \(P(Z_1\geq x,\ H_J\text{ true})\leq1-\Phi(x)\). Conditional super-uniformity of the stage-two p-value implies that, under the selected treatment’s null, \(Z_2\) has upper tails no larger than those of a standard normal variable, conditional on stage-one information. Since the chance of crossing the combination threshold increases with \(Z_1\), these two bounds give

To see the calibration, replace the selected-true-null stage-one contribution and the conditionally valid stage-two contribution by independent standard normal variables: their weighted sum has a standard normal distribution because \(w_1^2+w_2^2=1\). The tail bounds ensure that the actual rejection probability on the selected-true-null event is no greater. This is a strong selected-null guarantee: it remains valid when some unselected candidates are effective.

Worked Combination Analyses

Diagnostics, Alternatives, and Limitations

Before relying on the result, check that the written analysis matches how the trial actually operated. Confirm which treatments could be selected, how selection used stage-one information, whether the adjustment covers all eligible candidates, and whether weights and the one-sided direction were prespecified. Check that stage two supplies conditionally valid evidence under the selected treatment’s null. If participant populations, endpoint definitions, measurement procedures, or estimands change between stages, the stagewise tests must still address the same target or have a justified relationship; merely combining their p-values does not repair a mismatch.

The inverse-normal combination test is one alternative to ordinary pooling, not the only one. A design may use a more efficient prespecified multiplicity procedure for stage one, or an adaptive test whose calibration explicitly incorporates the selection rule. A separate confirmatory stage-two test can avoid reusing stage-one evidence, but may require more information. Ordinary pooling remains reasonable when the treatment comparison was fixed independently of stage-one outcomes and its model assumptions hold. After data-dependent selection, it is not made valid simply by reporting a larger sample size or by using a familiar pooled standard error.

Bonferroni adjustment can lose power, particularly when many candidates are considered or their test statistics are strongly related. The combination test also depends on its stated conditions: a misspecified stage-two test, outcome-dependent weights, an unrecorded change in the selection rule, or use of a p-value that does not test the intended null can undermine the guarantee. It controls the selected-true-null rejection probability; it does not by itself provide simultaneous conclusions about every candidate or correct a biased treatment-effect estimate.

• Mistake: “The pooled statistic is standard normal because the two stages are independent.” Independence of stagewise data is not enough: selection makes the chosen stage-one statistic nonstandard.
• Correction: Account for selection in the inferential procedure, for example by adjusting stage-one evidence and combining it with conditionally valid stage-two evidence.
• Mistake: “The selected adjusted p-value is always super-uniform, even when some treatments work.”
• Correction: The relevant guarantee is \(P(p^*_{1,J}\leq u,\ H_J\text{ true})\leq u\), which concerns occasions when the selected null is true.