Why Mixed Stagewise Tests Can Be Combined
In a multi-stage trial, the evidence from one stage may be evaluated with an exact test while another stage uses an asymptotic test. The first may have a discrete sampling distribution and the second a continuous approximation. This difference does not, by itself, prevent combination. What matters is whether each stage supplies a p-value with the calibration required by the chosen combination rule.
A p-value is not merely a statistic placed on a common numerical scale. Its defining role is to control the probability of being as small as it is under the null hypothesis. A combination procedure uses that calibration, along with its assumptions about dependence and any prespecified weights, to determine the null distribution of the combined evidence.
An exact test obtains its p-value from the null sampling distribution, including its discreteness. An asymptotic test instead uses an approximation, often a normal distribution for a standardized statistic. If that approximation is accurate, its p-value may be approximately valid, especially at larger sample sizes. But approximate validity is not the same as a finite-sample guarantee. The combination method cannot repair a stagewise p-value that is too small too often.
Conditions for Combining the P-Values
For the results below, consider a one-sided test in which smaller p-values indicate stronger evidence in the prespecified direction. Assume that each stage’s p-value is valid under the null being tested, and that the stagewise p-values are independent under that null. Independence may be justified, for example, by using separate patient cohorts at the stages and applying prespecified analyses. If the stages share data or otherwise produce dependent p-values, the independent reference distributions given here need not apply.
The stagewise test statistics need not have the same distribution, units, or definition. An exact binomial tail probability and an asymptotic standardized-statistic p-value can both be valid inputs because each is a probability calibrated against its own stagewise null distribution. The combination procedure acts on the p-values, not by pooling or equating the underlying statistics.
Two common choices are Fisher’s statistic and the inverse-normal statistic. In Fisher’s method, with \(k\) independent stagewise p-values, the statistic is \(T=-2\sum_{i=1}^{k}\log(P_i)\). If the p-values are independent continuous uniform random variables under the null, \(T\) has a chi-square distribution with \(2k\) degrees of freedom. For valid but discrete p-values, the usual chi-square calibration is conservative or exact in the sense of controlling the upper-tail error rate; it need not be an exact continuous reference distribution.
For inverse-normal combination, choose fixed weights \(w_i\) such that \(\sum_{i=1}^{k}w_i^2=1\), and define \(Z_i=\Phi^{-1}(1-P_i)\) for one-sided p-values. The combined statistic is \(Z_{\mathrm{comb}}=\sum_{i=1}^{k}w_iZ_i\). With independent uniform p-values, this has a standard normal null distribution. With independent valid p-values and fixed nonnegative weights satisfying \(\sum_{i=1}^{k}w_i^2=1\), the upper-tail test is conservative or exact. The weights must be fixed independently of the observed stagewise results for this calibration argument to apply.
Why Validity, Not Identical Statistics, Is the Key
The calibration argument can be seen from the distribution of each transformed p-value. For any real \(z\),
Thus a valid p-value transformed this way has no more upper-tail probability than a standard normal variable. For independent stages, the weighted sum with fixed nonnegative weights and sum of squared weights equal to one is no more prone to large values than the standard normal reference distribution. The corresponding upper-tail probability is therefore a valid combined p-value. This reasoning uses each stage’s p-value calibration and independence; it does not require the stagewise statistics to be identical.
For Fisher’s method, validity means each p-value is stochastically no smaller than a uniform random variable. Since \(-2\log(p)\) decreases as \(p\) increases, each transformed contribution is stochastically no larger than a chi-square variable with 2 degrees of freedom. Independence then yields the chi-square upper-tail calibration for the sum. When one or more p-values are discrete, that calibration can be conservative because the observed inputs do not range continuously over \([0,1]\).
These arguments also show the limitation: a stagewise p-value that is only asymptotically calibrated supports, at most, an asymptotic combined calibration unless additional finite-sample validity is established. The exact stage does not make the approximate stage exact.
Worked Example: Inverse-Normal Combination With One Exact Stage
Suppose two independent stages test a prespecified one-sided treatment effect. At stage 1, four independent binary outcomes are observed, and under the point null each has response probability \(0.5\). All four are responses. At stage 2, a large-sample standardized statistic is \(z_2=1.2816\), giving an asymptotic one-sided p-value of approximately \(0.1000\). Use equal inverse-normal weights.
Stage 1 exact p-value. Under the null, \(X\sim\operatorname{Binomial}(4,0.5)\). The exact upper-tail p-value is
This p-value is discrete and valid under the stated binomial null. Stage 2 has \(p_2=1-\Phi(1.2816)\approx0.1000\), using its asymptotic normal approximation. The independence assumption follows here from the stipulated separate stage cohorts; the accuracy of stage 2’s approximation remains a separate condition to assess.
Equal weights are \(w_1=w_2=1/\sqrt{2}\). The normal scores are \(Z_1=\Phi^{-1}(0.9375)=1.5341\) and \(Z_2=1.2816\). Therefore,
The observed combined statistic has an upper-tail standard normal p-value of \(0.02324\). At a nominal \(0.05\) level, this rejects the global null under the combination assumptions. Because the second input is asymptotic, the stated finite-sample conclusion is only as reliable as that stage’s approximation; the exact first-stage test does not remove this qualification.
Fisher Combination With a Discrete Input
The next calculation illustrates that an exact discrete p-value can be used directly in Fisher’s statistic. Its value is not adjusted to look continuous or to match the other stage’s statistic. Instead, the combined reference distribution is used with the understanding that discreteness may make the test conservative.
Worked Example: Fisher Combination of Exact and Asymptotic P-Values
In an independent first-stage cohort, \(X\sim\operatorname{Binomial}(8,0.5)\) under the point null, and \(X=6\) responses are observed. The second-stage asymptotic one-sided p-value is \(0.05000\). Assume the stages are independent and the second-stage approximation is adequate for the intended calibration.
The exact first-stage p-value is
The Fisher statistic and its observed value are
For two stages, the chi-square reference has \(2k=4\) degrees of freedom. Its upper-tail probability is \(e^{-T/2}(1+T/2)\), equivalently \(q(1-\log q)\) for \(q=p_1p_2\). Hence
The combined p-value is \(0.04285\), below \(0.05\). The calculation does not assume that the binomial count and the second-stage statistic share a common distribution. It does rely on stagewise validity and independence. Because the first p-value is discrete and the second is asymptotic, the result may be conservative in the first stage and approximate in the second.
When an Asymptotic P-Value Is Not Yet Valid
An asymptotic p-value can be a useful input without being an exact finite-sample p-value. The distinction matters most when sample sizes are small, outcomes are discrete, or the null distribution of the standardized statistic is poorly approximated. A nominal normal calculation can then yield too many small p-values under the null. If that happens, neither Fisher nor inverse-normal combination inherits the intended finite-sample error control.
Worked Example: A Small-Sample Normal Approximation Can Be Anti-Conservative
Let \(X\sim\operatorname{Binomial}(10,0.5)\) under a point null. A tempting approximation uses the null mean \(5\) and standard deviation \(\sqrt{10(0.5)(0.5)}=\sqrt{2.5}=1.5811\), without a continuity correction. For the one-sided result \(X=8\), it reports
This approximation appears to give a p-value at or below \(0.03\). But the event that this reported p-value is at most \(0.03\) includes every outcome \(X\geq8\), since still larger counts give still larger \(z\)-statistics. Its actual null probability is
Thus \(P(p_{\mathrm{normal}}\leq0.03\mid H_0)\geq0.05469>0.03\): this p-value calculation is anti-conservative at that threshold. An exact binomial p-value for \(X=8\) is \(0.05469\), not \(0.0289\). The example warns against treating every normal approximation as valid merely because it produces a number between zero and one. A continuity correction can improve approximation in some settings, but does not automatically establish validity for all thresholds.
Diagnostics, Alternatives, and Limitations
Before combining mixed exact and asymptotic p-values, check the following conditions and potential failure modes:
- Stagewise calibration: verify how each p-value was computed under its own null. For an exact test, check the tail definition and discreteness. For an asymptotic test, assess whether its approximation is defensible at the stage’s sample size and outcome distribution.
- Independence: establish that stagewise data and p-values are independent under the null, or use a calibration appropriate to their dependence. Separate cohorts can support independence; reused observations generally require more care.
- Prespecification: choose the combination method, direction of evidence, and any weights before inspecting the stage results. Data-dependent choices can invalidate the reference calibration.
- Numerical handling: retain sufficient p-value precision. Rounding a small p-value to zero makes \(\log(p)\) undefined and can distort a Fisher statistic; rounded values should not replace the underlying calculation.
Fisher’s method aggregates contributions through \(-2\log(p_i)\), so very small p-values can contribute strongly. The inverse-normal method combines normal-score transformations and permits prespecified stage weights. Both methods can combine exact and asymptotic inputs under their respective assumptions; neither makes a poorly calibrated input reliable. If a stagewise method is only approximately calibrated, describe the combined calibration as approximate and investigate finite-sample operating characteristics when that distinction is consequential.
A common mistake is to argue, “The stagewise statistics differ, so they cannot be combined,” or the reverse, “Both reported p-values, so the combined test must be exact.” The correction is that matching statistics is not required, but p-value validity and the combination rule’s dependence and weighting assumptions are. A p-value’s scale is comparable only insofar as its null calibration is sound.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Under a null hypothesis, a stagewise p-value satisfies \(P(P_i\leq u)\leq u\) for all \(u\in[0,1]\). What does this condition say about its calibration?
- Two independent stages have p-values \(0.20\) and \(0.10\). Calculate the two-stage Fisher statistic and its chi-square reference degrees of freedom.
- Why can an exact discrete p-value be a valid combination input even when its distribution is not uniform?
- In the binomial example with \(n=10\), \(p=0.5\), and observed \(X=8\), what is the exact one-sided p-value, and why is the uncorrected normal value \(0.0289\) not a finite-sample valid substitute?
- Name two conditions, other than the stagewise statistics being identical, that matter for the combination calibration.
Solutions. (1) The p-value is super-uniform: under the null, the chance it is at or below any threshold \(u\) is no greater than \(u\), which controls false small p-values. (2) \(T=-2\log(0.20\times0.10)=-2\log(0.02000)=7.824\), with \(2k=4\) reference degrees of freedom. (3) Validity requires the super-uniform inequality, not a continuous uniform distribution; discreteness may make the combined test conservative. (4) The exact tail is \((45+10+1)/1024=0.05469\). The uncorrected normal approximation gives \(0.0289\), yet under the null the probability of a reported value at most \(0.03\) is at least \(P(X\geq8)=0.05469\), exceeding \(0.03\). (5) Examples include stagewise p-value validity, independence (or an appropriate dependence calibration), and prespecified weights and combination method.