Combining Stagewise P-Values From Different Tests

Why Conditional Validity Matters

An adaptive study may use different tests at different stages. For example, a first-stage analysis may use a score test while a second-stage analysis uses a test chosen after sample-size re-estimation. The resulting p-values need not come from the same statistic or reference distribution. They can still be combined, but the key requirement is not that the tests match: it is that each stagewise p-value be valid under the null, including after conditioning on the information used to make the adaptation.

This tutorial develops that requirement and shows how it supports Fisher and inverse-normal combination tests. The earlier material on combination tests gives the general motivation and the standard combination rules; here the emphasis is on why combining p-values from different tests can control Type I error, and on the conditional-validity failure that can invalidate an apparently reasonable analysis.

Definition: A p-value \(P_j\) for stage \(j\) is super-uniform under the null if, for every \(u\in[0,1]\), \(P(P_j\le u)\le u\) under every null parameter value covered by the test. It is conditionally super-uniform given prior-stage information if \(P(P_j\le u\mid\mathcal F_{j-1})\le u\) almost surely for every \(u\in[0,1]\), where \(\mathcal F_{j-1}\) contains the data and decisions available before stage \(j\).

For two stages, the required conditions are that \(P_1\) be super-uniform under the null and that \(P_2\) be super-uniform conditional on \(\mathcal F_1\). The second condition applies to the test actually selected and its actual sample size. If stage-two sample size, test choice, or another design feature depends on stage-one data, validity must hold for each relevant realized stage-one history—not merely after averaging over those histories.

Fisher Combination With Conditional Stagewise Validity

For two one-sided p-values aimed at the same prespecified direction of effect, Fisher’s combination statistic is

T=-2\{\log(P_1)+\log(P_2)\}.

If \(P_1\) and \(P_2\) are independent Uniform(0,1) random variables under the null, \(T\) has a chi-square distribution with 4 degrees of freedom. The combined p-value is therefore \(P(\chi^2_4\ge T)\). Conditional validity provides a route to Type I error control even when the stagewise tests differ and the second-stage design adapts.

Theorem: Suppose \(P_1\) is super-uniform under the null and \(P_2\) is conditionally super-uniform given \(\mathcal F_1\), with \(P_1\) and the stage-two design measurable from \(\mathcal F_1\). Then rejecting when \(T\ge q_{1-\alpha}\), where \(q_{1-\alpha}\) is the \(1-\alpha\) quantile of \(\chi^2_4\), has Type I error at most \(\alpha\). This conclusion holds even if the stagewise tests are different.

A short derivation makes the role of the conditioning explicit. Write \(q=q_{1-\alpha}\) and \(c=\exp(-q/2)\). Since \(T\ge q\) is equivalent to \(P_1P_2\le c\), for a realized \(P_1=x\) the rejection event is \(P_2\le c/x\). Conditional super-uniformity bounds its probability by \(\min(1,c/x)\). Consequently,

P(T\ge q)\le E\left[\min\left(1,\frac{c}{P_1}\right)\right] \le \int_0^1\min\left(1,\frac{c}{x}\right)\,dx =c+\int_c^1\frac{c}{x}\,dx =c(1-\log c)=\alpha.

The second inequality uses the fact that a super-uniform \(P_1\) is stochastically no smaller than a Uniform(0,1) variable, while \(x\mapsto\min(1,c/x)\) is decreasing. The final equality follows because the upper tail of a \(\chi^2_4\) variable at \(q\) is \(\exp(-q/2)(1+q/2)\), and \(c=\exp(-q/2)\). Thus the chi-square reference calibration is justified by the conditional validity assumptions; it is not justified simply because two quantities happen to be called p-values.

Worked Example: Fisher Combination of Two Different Tests

Suppose a hypothetical two-stage study tests the same one-sided null hypothesis at both stages. A prespecified first-stage test reports \(p_1=0.0800\). After the first-stage review, the sample size is increased, and a different, prespecified-for-that-adaptation second-stage test reports \(p_2=0.0150\). Assume the first-stage p-value is valid under the null and, for every stage-one history and the resulting stage-two design, the second-stage p-value is conditionally valid under the null.

The stagewise tests have different forms, but the assumptions needed for the combination test are met. First calculate the product and Fisher statistic:

p_1p_2=(0.0800)(0.0150)=0.001200,\qquad T=-2\log(0.001200)=13.4509.

For 4 degrees of freedom, the chi-square upper-tail probability can be written as \(\exp(-T/2)(1+T/2)\). Thus the combined p-value is

p_{\mathrm{Fisher}} =e^{-13.4509/2}\left(1+\frac{13.4509}{2}\right) \approx 0.009271.

At a 0.05 significance level, this combined result rejects the null hypothesis. The conclusion is about the combined evidence against the common null, not a claim that either stage’s p-value is the probability that the null is true.

What Conditional Validity Rules Out

Marginal validity alone is not enough. A p-value can be Uniform(0,1) before the interim analysis and still be invalid for combination if its conditional distribution is distorted by the interim information. The next hypothetical calculation shows how serious this can be.

Worked Example: Marginally Uniform P-Values That Fail After Adaptation

Let \(U\sim\mathrm{Uniform}(0,1)\) under the null and suppose an analysis reports \(P_1=U\) and \(P_2=U\). Each p-value is marginally uniform. But once \(P_1\) is observed, \(P_2\) is known exactly; it is not conditionally uniform.

At \(\alpha=0.05\), the Fisher critical value is approximately \(q_{0.95}=9.4877\). Equivalently, reject when \(P_1P_2\le c\), where

c=e^{-9.4877/2}\approx 0.008705.

Here \(P_1P_2=U^2\), so the actual null rejection probability is

P(U^2\le c)=P(U\le\sqrt{c})=\sqrt{0.008705}\approx 0.0933.

The test’s Type I error is about 9.33%, not 5%. The chi-square reference calculation fails because the two p-values are dependent in a way that violates conditional validity. This example also shows why checking that each stagewise p-value looks reasonable marginally does not establish that their combination is valid.

Inverse-Normal Combination and Test Choice

Fisher’s rule is not the only way to combine valid stagewise p-values. For a one-sided alternative, transform each p-value to a normal score \(Z_j=\Phi^{-1}(1-P_j)\). With fixed positive weights \(w_1,w_2\) satisfying \(w_1^2+w_2^2=1\), the inverse-normal statistic is

Z_{\mathrm{comb}}=w_1Z_1+w_2Z_2.

Under independent uniform p-values, this statistic is standard normal. More generally, if \(P_2\) is conditionally super-uniform given stage one, the combination’s rejection region is decreasing in each p-value. Conditional super-uniformity bounds the stage-two rejection probability for each stage-one history; stage-one super-uniformity then bounds the overall rejection probability. Weights and any permitted data dependence must meet the validity conditions for the particular combination rule.

Worked Example: Inverse-Normal Combination of Different Tests

Suppose two different one-sided tests yield hypothetical valid stagewise p-values \(p_1=0.0800\) and \(p_2=0.0150\), with the same conditional-validity assumptions as in the Fisher example. Use equal fixed weights \(w_1=w_2=1/\sqrt{2}\).

The normal scores are approximately \(z_1=\Phi^{-1}(0.9200)=1.4051\) and \(z_2=\Phi^{-1}(0.9850)=2.1701\). Therefore,

z_{\mathrm{comb}}=\frac{1.4051+2.1701}{\sqrt{2}} \approx 2.528.

The one-sided combined p-value is \(1-\Phi(2.528)\approx0.00573\). This is below 0.05, so the combined test rejects at that level. The numerical result differs from Fisher’s because the combination rules weight evidence differently; neither rule requires the underlying stagewise tests to have the same statistic, provided the required validity conditions hold.

As a useful check on the transformation, if \(p_1=0.5000\) and \(p_2=0.0500\), then \(z_1=0\) and \(z_2\approx1.645\). With equal weights, \(z_{\mathrm{comb}}=1.645/\sqrt{2}\), giving \(1-\Phi(1.645/\sqrt{2})\approx0.1224\). The combined p-value is not the second-stage p-value of 0.05.

Fisher’s method is convenient when evidence is summarized as p-values and a strong result at one stage can contribute substantially to the combination. Inverse-normal combination makes stage weights explicit and can reflect planned information contributions. Neither method repairs invalid stagewise p-values. In some designs, a conditional-error approach may be the more natural alternative because it directly preserves a specified original rejection rule. That approach has different inputs and should not be treated as interchangeable with a p-value combination without checking its assumptions.

Diagnostics, Limitations, and Common Mistakes

Before using a combination test, document the null hypothesis and alternative, the stagewise test at each stage, how each test’s reference distribution was obtained, and the information used to choose the stage-two test or sample size. The critical diagnostic question is: given any allowable stage-one history, is the selected stage-two p-value super-uniform under the null? Discreteness is generally compatible with validity: an exact p-value may be conservative, so its conditional rejection probability is no larger than the nominal bound. By contrast, a plug-in approximation that is anti-conservative, or a selection rule that favors unusually small stage-two p-values without accounting for selection, can defeat error control.

The usual combination interpretation also depends on aligning the tests. Their p-values should address the same prespecified null claim and the intended direction of evidence. Combining two-sided p-values can control a global null when the validity conditions hold, but evidence in opposite directions can then be combined without establishing a consistent treatment benefit. A small combined p-value is not itself an estimate of effect size, and it does not establish that the effect is clinically important.

Mistake: “Both p-values are valid on average, so the Fisher reference distribution must apply.” Correction: Stage two must be valid conditional on the information that determined its test and design.
Mistake: “Different tests cannot be combined.” Correction: Different tests can be used when they test the same null claim and their stagewise p-values satisfy the needed validity conditions.
Mistake: “The combined p-value is the probability the null is true.” Correction: It is a tail probability under the null for the chosen combination statistic; it is not a posterior probability of the null.

Key takeaway: The validity of a combination test comes from calibrated combination rules plus valid stagewise p-values. In adaptive designs, the decisive condition is conditional super-uniformity of each later p-value given the prior information used to shape that stage.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

State the conditional-validity requirement for a second-stage p-value.
For Fisher’s two-stage test, what product threshold corresponds to a chi-square critical value \(q\)?
Why can two marginally uniform p-values fail to give a valid Fisher test?
With equal inverse-normal weights, \(z_1=0\) and \(z_2=1.645\). Calculate the combined statistic and explain why its p-value is not 0.05.

Solutions. (1) For every \(u\in[0,1]\), the conditional probability \(P(P_2\le u\mid\mathcal F_1)\) must be at most \(u\), almost surely under the null, for the selected test and design. (2) Since \(T=-2\log(P_1P_2)\), \(T\ge q\) is equivalent to \(P_1P_2\le e^{-q/2}\). (3) Marginal uniformity does not prevent dependence on prior-stage information; the conditional distribution may concentrate on small values after stage one, increasing the combined rejection probability. (4) \(z_{\mathrm{comb}}=(0+1.645)/\sqrt{2}\approx1.163\), so the one-sided p-value is \(1-\Phi(1.163)\approx0.1224\), not 0.05.