Fisher Combination Tests With Adaptive Stage Two

Why Combine Stagewise P-Values?

In a two-stage design, the second-stage sample size may be selected using information observed at the first stage. That flexibility creates a calibration question: does the final test still reject the global null hypothesis \(H_0\) with probability at most the chosen significance level \(\alpha\)? Fisher’s combination test offers a way to combine evidence from both stages without treating the adaptive sample-size rule as though it had been fixed in advance. Its validity, however, depends on how the stagewise p-values behave under \(H_0\).

The essential condition is not simply that the stages use different participants. The stage-two p-value must remain valid conditional on the stage-one information used to choose the second-stage sample size. When that condition holds, adaptive sample size does not by itself invalidate Fisher’s reference distribution.

Reference Distribution and Its Derivation

Suppose that under \(H_0\), \(p_1\) and \(p_2\) are independent and each has a Uniform(0,1) distribution. For a single Uniform(0,1) random variable \(U\), define \(Y=-2\log U\). For \(y\geq 0\),

The last expression is the cumulative distribution function of a chi-square random variable with two degrees of freedom. Thus \(-2\log p_1\) and \(-2\log p_2\) are independent chi-square variables with two degrees of freedom. Their sum has a chi-square distribution with four degrees of freedom.

For this reference distribution, the upper-tail probability at an observed value \(t\) is

This formula gives the combined p-value. Equivalently, since \(t=-2\log(p_1p_2)\), it can be evaluated directly as

The combined p-value is compared with \(\alpha\); reject \(H_0\) when \(p_{\mathrm{Fisher}}\leq\alpha\). At \(\alpha=0.05\), the upper-tail critical value for \(\chi^2_4\) is approximately 9.4877.

When Stage Two Is Adaptive

Let \(D_1\) denote all stage-one information available when the sample-size decision is made, and let the rule choose a stage-two sample size \(n_2=g(D_1)\). The relevant condition is that, under \(H_0\), the stage-two p-value is conditionally Uniform(0,1) given \(D_1\), including the selected value of \(n_2\). In symbols, for every relevant stage-one outcome,

If \(p_1\) is also Uniform(0,1) under \(H_0\), then \(p_1\) and \(p_2\) are independent. To see this, for any \(a,b\in[0,1]\), use that \(p_1\) is determined by \(D_1\):

This is the joint distribution of two independent Uniform(0,1) variables, so the chi-square reference distribution follows. The argument shows why adaptive sample size can be compatible with Fisher combination: conditional calibration, rather than a fixed value of \(n_2\), is what supports the reference law.

A standard setting satisfying the condition is one in which the stage-two observations are fresh and independent of stage one, the chosen \(n_2\) is fixed once \(D_1\) is observed, and the stage-two test produces an exact null p-value for that sample size. For example, with a known standard deviation and a normally distributed test statistic under \(H_0\), the properly calibrated one-sided normal-tail p-value is uniform at each fixed \(n_2\). That exact conditional property must be justified for the actual analysis; it should not be assumed merely because the samples are separate.

Worked Calculations and Interpretation

Assumptions, Diagnostics, and Alternatives

Before using the \(\chi^2_4\) reference distribution, check the conditions that create it. For the \(\chi^2_4\) reference distribution, the p-values must be independent and Uniform(0,1) under the global null; compatible, pre-specified directions may aid interpretation of the combined evidence but are not required for this null calibration. Under \(H_0\), each stagewise p-value must have the required uniform distribution, and \(p_2\) must remain uniform conditional on the stage-one information used for adaptation. If the design uses conservative p-values rather than exact uniform p-values, the exact chi-square claim does not automatically follow; the null calibration of the combined test needs separate justification.

Practical diagnostics focus on the design and analysis process, not on testing independence from the observed p-values in a single trial. Check that the sample-size rule was specified and implemented as intended, that stage-two data were not used in the rule before the second-stage test was formed, and that the calculation of \(p_2\) matches the selected sample size and test. Investigate whether exclusions, outcome definitions, or analysis choices were changed after looking at stage-two outcomes. Such changes can undermine conditional calibration even if the sample-size rule itself was permissible.

Fisher combination is one option among several. The inverse-normal combination test transforms stagewise p-values to normal scores and combines them using chosen weights; it can offer a transparent way to express the relative contribution of stages. Fisher’s method instead uses a fixed transformation of the p-values and, under the stated assumptions, a \(\chi^2_4\) reference distribution. An unadjusted pooled analysis may be efficient in a fixed design, but when the sample size or analysis is adapted, its ordinary fixed-design calibration cannot simply be presumed valid. The methods have different structures; none removes the need to justify the null distribution used for the final test.

Fisher’s test also has limits. It is a test of the global null, not an estimator of the treatment effect, and the combined p-value does not quantify the effect’s size or precision. The method can be sensitive to a very small p-value from one stage, so the combined result need not mean that both stages separately provide strong evidence. If the stage-two p-value is not conditionally valid, the standard calculation may not control Type I error as claimed. Incompatible alternatives can make the combined result difficult to interpret, but do not affect that null calibration when the p-values are independent and Uniform(0,1) under the null being tested.

Common Mistake