Combining Stagewise Evidence
In an adaptive design, the analysis may use evidence gathered at more than one stage, even when sample size or other design features change between stages. A combination test reduces the stagewise results to one statistic and compares it with a critical value calibrated under the null hypothesis. Its validity depends on both the properties of the stagewise p-values and the calibration of the combined statistic.
Let \(P_1,\ldots,P_K\) be the stagewise p-values for the same null hypothesis, with smaller values indicating stronger evidence against \(H_0\). Under \(H_0\), each p-value must be valid: for every \(u\) between zero and one, \(P(P_i\leq u)\leq u\). A continuous exact p-value has a uniform distribution on \((0,1)\) under the null. The stagewise p-values also need an appropriate joint null distribution. In the basic results below, they are independent and continuous uniform under \(H_0\).
A function that is nonincreasing in each p-value has the usual evidence direction: making any stagewise p-value smaller cannot weaken the combined evidence. This monotonicity is useful but does not, by itself, guarantee Type I error control. The joint null law and the critical value must still match the statistic being used.
Calibration: The Null Distribution Determines the Cutoff
Write \(P_0\) for probability under the null. If the null distribution of \(T\) is continuous and does not depend on an unknown nuisance parameter, the upper-tail critical value can be set by \(P_0(T>c_\alpha)=\alpha\). For a discrete statistic, an exact cutoff may not attain \(\alpha\); a conservative cutoff satisfies \(P_0(T>c_\alpha)\leq\alpha\). If the null law varies over a composite null, calibration must control the largest rejection probability across that null.
When the p-values are independent continuous uniforms, the null distribution can often be derived directly. When p-values are discrete or conservative, or when their dependence differs from the assumed model, the reference distribution may not apply. In those cases, the critical value must be obtained from the appropriate joint null distribution, using an exact calculation or a justified numerical calibration.
Weighted Stouffer Combination
For a one-sided test where small p-values represent evidence in the prespecified direction, transform each p-value to a standard normal score \(Z_i=\Phi^{-1}(1-P_i)\). Under the independent continuous uniform null model, each \(Z_i\) is standard normal. For fixed weights \(w_i\), not all zero, the weighted Stouffer statistic is
The denominator standardizes the weighted sum. Under independence, its numerator is normal with mean zero and variance \(\sum_i w_i^2\); therefore \(Z_{\mathrm{comb}}\sim N(0,1)\) under \(H_0\). The level-\(\alpha\) test rejects when \(Z_{\mathrm{comb}}>z_{1-\alpha}\). The weights can reflect prespecified stage information, but their values and the calibration must be compatible with the design.
Worked Example: Weighted Stouffer Combination
Suppose a hypothetical two-stage trial has independent, continuous, one-sided stagewise p-values \(P_1=0.12\) and \(P_2=0.025\) under \(H_0\). The second stage has more information, so the prespecified weights are \(w_1=1\) and \(w_2=2\). The fresh stage-two data are independent of stage one, and each stagewise test is valid under the null for its actual sample size.
Convert the p-values to normal scores: \(\Phi^{-1}(0.88)\approx1.175\), and \(\Phi^{-1}(0.975)\approx1.960\). Then
At one-sided \(\alpha=0.025\), the critical value is \(z_{0.975}=1.960\). Since \(2.279>1.960\), reject \(H_0\). The combined p-value is approximately \(1-\Phi(2.279)=0.0113\). Under the stated model, this is the null tail probability for a Stouffer statistic at least this large; it is not the probability that \(H_0\) is true.
This standard normal calibration depends on fixed weights and independent standard normal scores under the null. If weights are selected after observing stage-one evidence, the statistic is no longer generally standard normal: the random weights can be associated with \(P_1\). A design that allows such choices must calibrate the resulting procedure under its actual selection rule rather than simply use \(z_{1-\alpha}\).
Fisher’s Product Combination
Fisher’s method combines p-values through their product, or equivalently through the negative logarithm of that product. For independent continuous uniform p-values, each \(-2\log(P_i)\) has a chi-square distribution with two degrees of freedom. Their sum therefore has a chi-square distribution with \(2K\) degrees of freedom:
Small p-values produce large values of \(T_F\), so the test rejects in the upper tail. Unlike weighted Stouffer combination, Fisher’s statistic does not assign explicit stage weights in this basic form. Both methods combine evidence, but their statistics respond differently to the pattern of stagewise p-values.
Worked Example: Fisher Combination and Its Critical Value
In a hypothetical two-stage analysis, suppose the one-sided stagewise p-values are \(P_1=0.04\) and \(P_2=0.20\). Assume they are independent and exactly uniform under \(H_0\). For \(K=2\), Fisher’s statistic has a chi-square null distribution with four degrees of freedom.
The observed statistic is
At \(\alpha=0.05\), the upper-tail critical value is \(\chi^2_{4,0.95}\approx9.488\). Since \(9.657>9.488\), reject \(H_0\). The combined p-value for four degrees of freedom is
Thus the combined evidence is just sufficient for rejection at the five-percent level. The calculation requires the independence and continuous-uniform null assumptions stated above; a Fisher reference distribution cannot be assumed merely because the inputs are called p-values.
Calibration When the Reference Distribution Does Not Apply
Stagewise tests may yield discrete p-values, for example when outcomes have a limited number of possible values. Such p-values can be valid but conservative: their null probability of being at most \(u\) may be strictly less than \(u\). A statistic built from these values need not follow the continuous-uniform reference distribution used for Stouffer or Fisher calibration. The exact null distribution, including discreteness, is then the relevant basis for a critical value.
Worked Example: Exact Calibration With Discrete P-Values
Consider a hypothetical two-stage procedure under \(H_0\). At each stage, the p-value equals \(0.2\) with probability \(0.1\), and equals \(1\) with probability \(0.9\). Assume the stagewise p-values are independent. This is a valid, conservative p-value distribution: for \(0.2\leq u<1\), \(P(P_i\leq u)=0.1\leq u\), while for \(u<0.2\) that probability is zero.
Use Fisher’s statistic \(T_F=-2\log(P_1P_2)\). If both p-values are \(0.2\), \(T_F=-2\log(0.04)\approx6.438\), and this event has probability \(0.1^2=0.01\). If exactly one is \(0.2\), \(T_F=-2\log(0.2)\approx3.219\), with probability \(2(0.1)(0.9)=0.18\). If neither is \(0.2\), \(T_F=0\), with probability \(0.9^2=0.81\).
At \(\alpha=0.05\), rejecting only when \(T_F\geq6.438\) gives null rejection probability \(0.01\), which is at most \(0.05\). There is no nonrandomized cutoff that gives size exactly \(0.05\): lowering the threshold to include the outcome \(T_F\approx3.219\) raises the rejection probability to \(0.19\). The continuous-reference chi-square cutoff is not the exact calibration for these discrete p-values.
Assumptions, Diagnostics, and Alternatives
In sample size re-estimation, the key condition is often not that the stage-two sample size stay fixed, but that the second-stage p-value remain valid under the null conditional on the stage-one information and the adaptation choice. With independent fresh stage-two data and a valid stage-two test for the selected sample size, this conditional validity can support an independent-p-value combination procedure. If the second-stage data reuse stage-one observations, or if the adaptation changes the test in a way that compromises its null distribution, independence or validity may fail. The null distribution used for calibration must reflect what was actually done.
Before using a combination critical value, check that the stagewise p-values test the same null in the intended direction, that their null validity is justified, and that the joint distribution matches the calibration. For discrete outcomes, inspect whether the p-values are conservative and whether the intended continuous reference law is justified. For weights, verify that they were fixed as assumed or that the actual data-dependent rule was calibrated. A mismatch can make the nominal \(\alpha\) level inaccurate.
| Approach | How evidence is combined | Calibration condition |
|---|---|---|
| Weighted Stouffer | Weighted sum of normal scores | Independent continuous uniform p-values and fixed weights give a standard normal statistic |
| Fisher | Negative log of the product of p-values | Independent continuous uniform p-values give a chi-square statistic with \(2K\) degrees of freedom |
| Direct null calibration | Uses the null distribution of the chosen statistic under the actual design | Requires a justified distribution that includes discreteness, dependence, and adaptation as applicable |
A fixed-design pooled test is a different approach: it analyzes pooled data using its own model and assumptions rather than combining stagewise p-values. After adaptation, its null distribution must still be justified for the actual procedure. A combination test instead relies on valid stagewise p-values and a calibrated combination function. Neither method is automatically valid simply because it produces a familiar test statistic.
Why the Calibration Matters
A combination function determines how different patterns of stagewise evidence contribute to the final result. Its critical value determines how often that rule rejects when the null is true. These are separate choices: a familiar function does not excuse an incorrect reference distribution. Independence, continuity, prespecified weights, and a matching null calibration are substantive conditions, not cosmetic details.
- Wrong reasoning: “Both stagewise results are p-values, so Fisher’s chi-square cutoff always controls the error rate.”
- Correction: Fisher’s stated chi-square null distribution requires independent continuous uniform p-values. For other valid p-value distributions, calibrate using the actual joint null law or justify a conservative bound.
- Another pitfall: choosing weights after looking at stage-one results and then treating the weighted statistic as standard normal. The data-dependent selection changes its null distribution and must be included in calibration.
Check Your Understanding
Answer each question using the assumptions stated in the question. Brief solutions follow.
- Under what assumptions does the basic weighted Stouffer statistic have a standard normal null distribution?
- For three independent continuous uniform null p-values, what is the null distribution of Fisher’s statistic?
- If \(P_1\) and \(P_2\) are dependent, can the Fisher chi-square calibration automatically be used? Explain.
- Why can a discrete valid p-value lead to a conservative test when calibrated by a continuous-uniform reference distribution?
Solutions. (1) The stagewise p-values must be independent and continuous uniform under \(H_0\), and the weights must be fixed; then the transformed normal scores are independent standard normal variables and the normalized weighted sum is standard normal. (2) It is \(\chi^2_6\), because each of the three independent terms \(-2\log(P_i)\) is \(\chi^2_2\). (3) No. Dependence changes the joint null distribution of the statistic, so a chi-square calibration based on independence is not justified without further analysis. (4) A conservative p-value places less null probability in small-p-value regions than a uniform p-value does. The resulting statistic can therefore have a different, often less extreme, null distribution; calibrating its critical value from the continuous reference law need not give the intended size.