K-Stage Fisher Combination Tests

Why Extend Fisher’s Test to K Stages?

A trial with more than two stages can produce several stagewise p-values, each summarizing evidence from one stage. Fisher’s combination test gives a single test statistic for the collection. Its extension to $K$ stages is direct, but the calibration depends on a specific reference distribution: under the global null, the stagewise p-values must be independent and uniformly distributed on $(0,1)$. Without the required assumptions, the familiar chi-square calculation need not be valid.

This tutorial focuses on the unweighted Fisher statistic: every stage contributes through the same transformation, $-2\log(p_i)$. The statistic accumulates evidence across stages; it is not itself a p-value. A combined p-value is obtained by comparing the statistic with its null distribution.

Definition: For $K$ stagewise p-values $p_1,\ldots,p_K$, Fisher’s combination statistic is $$ X_F=-2\sum_{i=1}^{K}\log(p_i). $$ If, under the global null hypothesis, the $p_i$ are independent and each is continuously Uniform$(0,1)$, then $X_F\sim\chi^2_{2K}$. The Fisher combined p-value is the upper-tail probability $P(\chi^2_{2K}\ge X_F)$, evaluated at the observed statistic.

The logarithm is the natural logarithm. Since each $p_i$ lies between zero and one, $-\log(p_i)$ is nonnegative: smaller p-values contribute more to the sum. The degrees of freedom are $2K$, not $K$, because each transformed uniform p-value contributes an exponential random variable that becomes a chi-square variable with two degrees of freedom after multiplication by two.

Why the Chi-Square Calibration Holds

The derivation starts with a single continuous uniform p-value $P_i$ under the null. For $y\ge 0$,

P\{-\log(P_i)>y\} =P(P_i<e^{-y}) =e^{-y}.

This is the survival function of an exponential random variable with rate 1. Therefore, $-2\log(P_i)$ has a chi-square distribution with 2 degrees of freedom. If the $K$ p-values are independent, their transformed values are independent as well, and the sum of independent chi-square variables has degrees of freedom equal to the sum of their degrees of freedom. Thus,

X_F=\sum_{i=1}^{K}\{-2\log(P_i)\}\sim\chi^2_{2K}.

An equivalent calculation is often convenient. Define $t=-\sum_{i=1}^{K}\log(p_i)$, so $X_F=2t$. The combined p-value can then be written as

p_F=P(\chi^2_{2K}\ge 2t) =e^{-t}\sum_{j=0}^{K-1}\frac{t^j}{j!} =\left(\prod_{i=1}^{K}p_i\right)\sum_{j=0}^{K-1}\frac{t^j}{j!}.

The last equality follows because $e^{-t}=\prod_{i=1}^{K}p_i$. The formula is the upper tail of a chi-square distribution with an even number of degrees of freedom. It shows directly why the product of the stagewise p-values is not generally the combined p-value: the product is multiplied by a factor that depends on $t$ and $K$.

Worked Example: Combining Three Stagewise P-Values

Suppose a hypothetical three-stage trial has stagewise p-values $p_1=0.10$, $p_2=0.20$, and $p_3=0.05$. Assume that, under the global null, the p-values are independent and continuous Uniform$(0,1)$, as required for the chi-square reference distribution. Calculate the Fisher statistic and combined p-value, and compare the result with a two-sided test at $\alpha=0.05$.

First, multiply the p-values: $0.10(0.20)(0.05)=0.001$. The statistic is

X_F=-2\{\log(0.10)+\log(0.20)+\log(0.05)\} =-2\log(0.001) =13.8155.

For $K=3$, the reference distribution is $\chi^2_6$. With $t=X_F/2=6.9078$, the upper-tail probability is

p_F=e^{-t}\left(1+t+\frac{t^2}{2}\right) =0.001000\left(1+6.9078+\frac{6.9078^2}{2}\right) \approx 0.03177.

The combined p-value is approximately 0.03177, so the global null is rejected at $\alpha=0.05$ under the stated assumptions. Although these stagewise p-values are no greater than 0.20, their product, 0.001, is not the combined p-value; the product enters the statistic and must be calibrated against $\chi^2_6$.

Assumptions and Interpretation

The exact chi-square reference distribution requires two conditions under the global null: each stagewise p-value is Uniform$(0,1)$, and the p-values are mutually independent. Continuous tests with correctly calibrated null distributions can yield uniform p-values. Discrete tests commonly yield p-values that are conservative rather than exactly uniform; independence alone then does not give the exact chi-square distribution.

The independence condition concerns the p-values under the null, not merely whether the stages have different labels or were analyzed separately. Independent stage data with fixed, separately specified stagewise tests can provide a suitable basis for independence. Conversely, shared observations, overlapping data, or analysis choices that link stagewise results can make the p-values dependent. The chi-square calibration should not be used automatically when this condition is doubtful.

For independent p-values that are valid but conservative under the null, each p-value is stochastically no smaller than a Uniform$(0,1)$ variable. Applying the decreasing transformation $-2\log(p_i)$ makes each contribution stochastically no larger than its uniform-reference counterpart. With independence, the sum is likewise stochastically no larger than $\chi^2_{2K}$, so the usual upper-tail test is conservative. This argument does not make the exact chi-square distribution true; it only explains the direction of the calibration under those conditions.

A small combined p-value is evidence against the global null that all stages follow their null distributions. It does not establish that every stage independently shows compelling evidence, nor does it quantify the probability that the null is true. Fisher’s statistic can combine several moderate contributions, or be strongly influenced by one very small p-value; the combined result should be interpreted alongside the stagewise results and the trial’s scientific question.

Worked Example: Four Stages With a Non-Significant Combined Result

In a hypothetical four-stage trial, suppose the independent continuous-null p-values are $0.40$, $0.25$, $0.10$, and $0.50$. The chi-square calibration is justified here by the stated assumptions. Calculate the combined result and interpret it at $\alpha=0.05$.

The product is $0.40(0.25)(0.10)(0.50)=0.005$. Therefore,

X_F=-2\log(0.005)=10.5966.

With $K=4$, the reference distribution is $\chi^2_8$. Here $t=-\log(0.005)=5.2983$, and

p_F=e^{-t}\left(1+t+\frac{t^2}{2}+\frac{t^3}{6}\right) =0.005\left(1+5.2983+\frac{5.2983^2}{2}+\frac{5.2983^3}{6}\right) \approx 0.2256.

The combined p-value is approximately 0.2256, which is greater than 0.05. These four stagewise results do not provide sufficient evidence to reject the global null at the stated significance level. The product, 0.005, would give a very different and incorrect interpretation if treated as the p-value.

Calculation Checks and Failure Modes

For hand calculations and software output, several checks help catch errors before interpreting the result.

Check the degrees of freedom. With $K$ inputs the reference distribution is $\chi^2_{2K}$. Using $K$ degrees of freedom changes the tail probability.
Check the tail. Fisher’s combined p-value is an upper-tail probability. Larger $X_F$ means stronger evidence against the global null and a smaller upper-tail p-value.
Check the inputs. The formula requires $0<p_i\le 1$. A reported zero caused by rounding is not a literal zero; it cannot be inserted into the logarithm. Use an unrounded p-value or a justified method for handling the underlying test result.
Check independence and calibration. Dependence or non-uniform null p-values can invalidate the exact reference distribution. A small numerical result does not repair a failed assumption.
Check the arithmetic. Compute $X_F$ or $t$ from the log-sum, then obtain the chi-square upper tail. Do not report $\prod_i p_i$ as the combined p-value.

A common mistake is to argue that the product of several small p-values must itself be the probability of obtaining such evidence under the null. It is not. That product is a component in the tail formula; the null distribution of the full log-sum determines the calibrated probability. Another mistake is to use the test because the number of stages is large enough, without checking whether the stagewise p-values are independent. No choice of $K$ makes dependent inputs automatically satisfy the stated reference distribution.

How Fisher’s K-Stage Test Differs From the Inverse Normal Test

The inverse normal method, covered in the earlier material on K-stage inverse normal combination tests, transforms stagewise p-values to normal scores and combines those scores using specified weights. Fisher’s method instead adds $-2\log(p_i)$ contributions and calibrates their sum with $\chi^2_{2K}$ under independent uniform null p-values. These methods use different transformations and reference distributions; a numerical result from one cannot be substituted into the calibration for the other.

In the unweighted Fisher construction here, each stage enters through the same formula, and the null degrees of freedom are determined by the number of stages. The method can be especially responsive when one or more stages yield very small p-values, because the logarithm makes their contributions large. But the combined test does not, by itself, describe a common treatment effect or guarantee that every stage points in the same direction. It also cannot compensate for dependent or invalid stagewise p-values.

Fisher’s K-stage test is useful when the scientific aim is to combine valid stagewise evidence and its independence assumptions are defensible. Its limitations should shape reporting: give the individual p-values, the combined statistic, the degrees of freedom, the calibrated combined p-value, and a clear statement of the assumptions behind the reference distribution.

Worked Example: Three Stages and a Result Near the Decision Threshold

Consider a hypothetical trial with three independent, continuous-null stagewise p-values $0.08$, $0.15$, and $0.25$. Find the combined p-value and decide whether to reject at $\alpha=0.05$.

The product is $0.08(0.15)(0.25)=0.003$, so

X_F=-2\log(0.003)=11.6183,\qquad t=-\log(0.003)=5.8091.

For $K=3$, $X_F\sim\chi^2_6$ under the assumptions. Its upper tail is

p_F=e^{-t}\left(1+t+\frac{t^2}{2}\right) =0.003\left(1+5.8091+\frac{5.8091^2}{2}\right) \approx 0.07105.

Because $0.07105>0.05$, the test does not reject the global null at the 5% level. It would be inaccurate to call this result significant merely because the product of the p-values is 0.003. The combined p-value is computed from the $\chi^2_6$ tail and is approximately 0.07105.

Check Your Understanding

Answer each question, then compare with the brief solutions.

For $K=5$, what is the chi-square reference distribution for the Fisher statistic under independent continuous uniform null p-values?
For $K=2$, suppose the p-values are $0.10$ and $0.30$. Calculate $X_F$ and the combined p-value using the equivalent $t$-formula.
Why is a product of stagewise p-values not generally the Fisher combined p-value?
What is the exact-distribution concern if stagewise p-values are independent but conservative rather than uniform?
What assumption is threatened if the same participants contribute data to more than one stage?

Solutions. (1) The reference distribution is $\chi^2_{10}$, since its degrees of freedom are $2K=10$. (2) The product is $0.03$, so $X_F=-2\log(0.03)=7.0131$, and $t=-\log(0.03)=3.5066$. For $K=2$, $p_F=e^{-t}(1+t)=0.03(1+3.5066)\approx0.1352$. (3) The product is $e^{-t}$, but the upper-tail probability also includes the factor $\sum_{j=0}^{K-1}t^j/j!$. (4) The exact $\chi^2_{2K}$ distribution no longer follows; with independent conservative p-values, the upper-tail test is conservative rather than exact. (5) Overlap can make the stagewise p-values dependent, so the independent-p-value chi-square calibration may fail.