Two-Sided P-Values in Fisher Combination Tests

What Two-Sided Evidence Retains—and What It Discards

In a multi-stage trial, each stage can provide a p-value for a test of the null hypothesis. A two-sided p-value counts evidence in either direction: a sufficiently large effect in favour of treatment and a sufficiently large effect against treatment can both produce a small value. Fisher’s combination test can combine these p-values, but the combination operates on their magnitudes, not on the directions of the observed effects.

This distinction matters for interpretation. Fisher’s statistic can provide evidence against the global null that all stagewise null hypotheses hold. By itself, it does not establish that the effect has the same sign in every stage, or even identify which direction the evidence supports. Those are separate questions about the estimated effects and the design of the directional test.

Definition: For a two-stage combination test, let $p_1$ and $p_2$ be the stagewise two-sided p-values for their prespecified null hypotheses. Fisher’s combination statistic is $$ X_F=-2\ln(p_1)-2\ln(p_2)=-2\ln(p_1p_2). $$ A small stagewise p-value makes a larger positive contribution to $X_F$. The statistic contains no sign for either stage’s estimated effect.

For example, if a stage tests a standardized statistic $Z$ against a two-sided alternative and $Z$ has a standard normal null distribution, its p-value is $2[1-\Phi(|Z|)]$. Taking the absolute value means that statistics of equal magnitude but opposite signs produce the same p-value. For other tests, the two-sided p-value is defined by that test’s prespecified two-sided procedure; it need not be calculated from a normal statistic.

Reference Distribution and Its Assumptions

The usual two-stage Fisher reference distribution requires two independent stagewise p-values that are each exactly uniform on $(0,1)$ under the global null. A continuous test with a correctly specified null distribution can have this property. More generally, the stagewise tests must be valid for their null hypotheses, and the joint null distribution must justify the reference distribution being used. Independence is a key condition: two individually valid p-values do not automatically make a valid Fisher test if they are dependent.

Under the independent, continuous-uniform conditions, $X_F$ follows a chi-square distribution with four degrees of freedom under the global null. The combined p-value is the upper-tail probability $P(\chi^2_4\geq X_F)$. For two stages, this tail probability also has a convenient expression in terms of the product $q=p_1p_2$.

Result: If $p_1$ and $p_2$ are independent Uniform$(0,1)$ random variables under the global null, then $$ X_F=-2\ln(p_1p_2)\sim\chi^2_4. $$ Writing $q=p_1p_2$, the combined upper-tail p-value is $$ P(\chi^2_4\geq X_F)=e^{-X_F/2}\left(1+\frac{X_F}{2}\right) =q(1-\ln q). $$

Why the Chi-Square Distribution Appears

The distribution follows by transforming each uniform p-value and adding the transformed values. This short derivation also shows where independence enters.

Transform one p-value.
If $P\sim\text{Uniform}(0,1)$, then for $y\geq 0$, $P(-2\ln P\leq y)=P(P\geq e^{-y/2})=1-e^{-y/2}$. This is the cumulative distribution function of a chi-square variable with two degrees of freedom.

Add the independent contributions.
Under the stated assumptions, each $-2\ln(p_i)$ has a $\chi^2_2$ distribution and the two contributions are independent. The sum of independent chi-square variables has degrees of freedom equal to the sum of their degrees of freedom, so $X_F\sim\chi^2_4$.

Evaluate the upper tail.
The $\chi^2_4$ upper-tail probability at $X_F$ is $e^{-X_F/2}(1+X_F/2)$. Since $X_F/2=-\ln q$, this equals $q(1-\ln q)$.

The argument is not a claim that every computed two-sided p-value is exactly uniform under every null. For example, discrete tests can produce conservative, non-uniform p-values. In that setting, the ordinary chi-square reference distribution may not give exact calibration; use a method appropriate to the stagewise null distributions. Similarly, when stage two follows a data-dependent change, do not assume independence merely because the stages have different labels. The validity of the joint reference distribution must be justified for the design.

Worked Example: Combining Two Independent Two-Sided P-Values

Suppose a hypothetical two-stage trial produces valid, independent two-sided p-values $p_1=0.040$ and $p_2=0.200$ under the global null. Calculate Fisher’s statistic and the combined p-value.

The product of the stagewise p-values is $q=(0.040)(0.200)=0.00800$. Therefore,

X_F=-2\ln(0.040)-2\ln(0.200) =-2\ln(0.00800) =9.657.

Using the $\chi^2_4$ upper tail,

p_{\mathrm{Fisher}} =e^{-9.657/2}\left(1+\frac{9.657}{2}\right) \approx 0.00800(1+4.828) \approx 0.0466.

At a prespecified significance level of $\alpha=0.05$, this combined p-value is below the threshold. Under the stated assumptions, the result is evidence against the global null. It does not show that each stage individually rejects its null, nor does it say whether the effects are in a common direction.

The Combination Does Not Preserve Direction

A two-sided p-value is unchanged if the sign of a stage’s statistic is reversed. Consequently, Fisher’s statistic is unchanged if either stage’s effect estimate changes sign while its magnitude stays the same. The test can therefore detect evidence that at least one component null is false without resolving whether the evidence represents a consistent treatment benefit, a consistent harm, or effects that point in different directions.

This is not a defect in the arithmetic. It is a limit on what the statistic encodes. If the scientific claim requires a common direction, inspecting the signs after a Fisher test is not a substitute for a test designed to support that directional claim. The earlier material on directional consistency across stages distinguishes the observed signs from the evidence needed for a formal directional conclusion.

Worked Example: Identical Fisher Evidence, Opposite Direction Patterns

Consider two hypothetical data patterns. In both, the absolute standardized statistics are 2.054 in stage one and 1.282 in stage two, giving two-sided p-values approximately $0.040$ and $0.200$. In pattern A the signed statistics are $+2.054$ and $-1.282$; in pattern B they are $+2.054$ and $+1.282$.

For each pattern, the Fisher statistic is

X_F=-2\ln(0.040)-2\ln(0.200)=9.657, \qquad p_{\mathrm{Fisher}}\approx 0.0466.

The combined p-values are identical because Fisher’s statistic uses only the two p-values. Pattern A has opposing observed directions, while pattern B has aligned observed directions. Fisher’s result alone makes no distinction between them. Pattern B’s observed signs may be descriptively consistent, but the Fisher p-value itself is not a test that the effect is positive in both stages.

Worked Example: Converting Signed Statistics Before Combination

In a hypothetical trial, suppose independent stages yield signed standard normal statistics $Z_1=2.00$ and $Z_2=-1.00$. Calculate the two-sided p-values and combine them using Fisher’s method.

The two-sided p-values are $p_i=2[1-\Phi(|Z_i|)]$. Thus $p_1\approx 0.0455$ and $p_2\approx 0.3173$. The signs differ, but they do not enter these p-value calculations. The product is $q\approx(0.0455)(0.3173)=0.01444$, giving

X_F=-2\ln(0.01444)\approx 8.476, \qquad p_{\mathrm{Fisher}}=0.01444(1-\ln(0.01444))\approx 0.0756.

At $\alpha=0.05$, this combination does not reject the global null. The first stage has a smaller p-value than the second, but that does not make the combined result significant. The opposing signs also mean the data do not show a common observed direction; the Fisher statistic itself neither uses nor resolves that fact.

Interpretation, Diagnostics, and Alternatives

Before reporting a Fisher combination, check that the p-values answer the intended stagewise questions and that their directions have not been silently discarded. Useful checks include:

Confirm the tail choice. Each $p_i$ should be the prespecified two-sided p-value, not a one-sided value selected after observing the sign.
Check the null calibration. The reference distribution depends on the stagewise p-values being valid under their nulls; ordinary $\chi^2_4$ calibration is exact under the independent continuous-uniform conditions stated above.
Assess dependence. Shared participants, overlapping data, or data-dependent design choices can invalidate the assumed independent-p-value distribution. Verify the design justifies independence or use an appropriate calibration.
Keep the direction visible. Report stagewise effect estimates and their signs alongside the combined test when direction matters. The combined p-value cannot replace them.

An inverse normal combination test uses transformed stagewise evidence and can retain signed information when its construction uses signed statistics and prespecified weights. Fisher’s method instead combines p-value magnitudes through logarithms and is naturally suited to testing a global null against evidence in one or more stages, without requiring a shared direction. Neither method licenses an interpretation beyond its null hypothesis and assumptions. The choice should follow the scientific question and the planned inferential claim.

Common mistake: “The Fisher test is significant, so both stages show a treatment benefit.” This conclusion does not follow. Fisher’s statistic combines p-value magnitudes, and a small combined p-value is evidence against the global null under its assumptions; it does not prove that both stagewise effects are nonzero or positive. Examine the stagewise estimates and use a directional procedure if the claim requires a common direction.

A related mistake is to average the p-values or to treat the smallest stagewise p-value as the combined result. Fisher’s rule is neither operation: it adds $-2\ln(p_i)$ contributions and evaluates their sum against the chi-square reference distribution. In a two-stage test, the resulting combined p-value can differ substantially from either the arithmetic average or the minimum.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For independent, continuous-uniform stagewise p-values $p_1=0.10$ and $p_2=0.10$, calculate Fisher’s statistic and combined p-value.
Two stages have equal-magnitude statistics with opposite signs. What happens to their two-sided p-values if one sign is reversed, and what does that imply for Fisher’s statistic?
What assumption about the joint distribution of stagewise p-values supports the usual $\chi^2_4$ reference distribution?
A Fisher combination rejects the global null. Can one conclude that both stagewise effects are positive? Explain.

Solutions. (1) The product is $q=0.0100$, so $X_F=-2\ln(0.0100)=9.210$. The combined p-value is $q(1-\ln q)=0.0100(1+4.605)=0.0561$, which is above $0.05$. (2) Reversing a sign does not change the absolute statistic or its two-sided p-value, so Fisher’s statistic is unchanged. (3) The p-values must be independent and Uniform$(0,1)$ under the global null for the exact standard derivation. (4) No. Rejection is evidence against the global null under the test assumptions; Fisher’s statistic does not establish that both effects are positive. Stagewise estimates and an appropriate directional analysis are needed for a common-direction claim.