Why Direction Matters When Combining Evidence
A combination test can pool evidence collected in different stages of a clinical trial. Fisher’s method does so by transforming stagewise p-values, with smaller p-values contributing more strongly to the combined statistic. But a p-value is meaningful only relative to the alternative being tested. If one stage tests for benefit and another tests for harm, the two p-values do not represent evidence for one common directional claim.
For example, suppose larger values of an outcome mean better health and the treatment effect is defined as treatment minus control. A one-sided test for benefit uses the alternative that the treatment effect is greater than zero at every stage. A stage showing a negative estimated effect will then produce a large p-value for that alternative, rather than strong evidence for benefit. This is a feature of the test, not a defect in Fisher’s formula.
The direction should be set by the scientific question and analysis plan, not selected separately for each stage after observing its data. In a different study, the clinically relevant alternative might be \(\theta<\theta_0\); that is also a valid one-sided direction if it is chosen in advance and used consistently.
Statistic, Reference Distribution, and Assumptions
For \(m\) stages, let \(p_j\) be the one-sided p-value from stage \(j\), calculated for the same alternative. Fisher’s statistic is
Small stagewise p-values make \(X_F\) large. If, under the global null, the stagewise p-values are independent and each is exactly uniformly distributed on \((0,1)\), then \(X_F\) has a chi-square distribution with \(2m\) degrees of freedom. The combined p-value is the upper-tail probability \(P(\chi^2_{2m}\geq X_F)\). For two stages, this tail probability has the convenient form \(e^{-X_F/2}(1+X_F/2)\).
The assumptions concern the stagewise p-values, not merely the appearance of the formula. Under the null, each p-value must be calibrated for its one-sided test, and the p-values must be independent for the usual chi-square reference distribution to be exact. Independent data from non-overlapping stages can support this assumption when the stagewise tests and their designs are fixed. If stagewise p-values are discrete or conservative, the exact uniform reference may not apply as written. Dependence, unaccounted data reuse, or data-dependent changes to the design can also invalidate the standard calibration; validity must then be established for the actual design.
The direction is part of the calibration and interpretation. Under a common upper-tail alternative, a positive stage estimate tends to give a small p-value, while a negative estimate tends to give a large one. If analysts instead choose whichever tail gives the smaller p-value separately at each stage, the resulting values are not the prespecified one-sided p-values assumed by the calculation.
Why the Chi-Square Calibration Works
The key transformation can be checked directly. If \(U\) is uniformly distributed on \((0,1)\), then for \(y\geq0\),
This is the cumulative distribution function of a chi-square random variable with 2 degrees of freedom. Thus each independent uniform p-value contributes a \(\chi^2_2\) variable, and the sum of \(m\) independent such variables has a \(\chi^2_{2m}\) distribution. This derivation explains both the degrees of freedom and why independence matters.
The calculation does not make Fisher’s statistic directional by itself. Direction enters through the definition of each \(p_j\). Fisher’s transformation retains the strength of each p-value but no longer carries the sign of the stage estimate as a separate quantity. The common alternative must therefore be specified and respected before the p-values are combined.
Worked Calculations
Worked Example: Two Stages Testing for Benefit
Suppose a hypothetical two-stage trial defines benefit as a treatment effect greater than zero. The same upper-tail alternative is used in both stages. The stagewise p-values are \(p_1=0.0250\) and \(p_2=0.2000\). Assume the stagewise tests are valid under the null and their p-values are independent.
There are two stages, so the reference distribution has \(2m=4\) degrees of freedom. First calculate the statistic:
For four degrees of freedom, the upper-tail probability is
Thus the combined test rejects the global null at the 0.05 significance level, but not at the 0.01 level. In context, the result is evidence against the common null in the prespecified direction of benefit. It does not mean there is a 3.149% probability that the null hypothesis is true, nor does it establish that the treatment effect was beneficial in each stage.
Worked Example: Opposing Stage Results Under a Benefit Alternative
In a hypothetical trial, the prespecified alternative is again that the treatment effect is greater than zero. Suppose the stage Z statistics are \(z_1=2.00\) and \(z_2=-1.50\), with standard normal calibration and independent stage data. The p-values for this same upper-tail alternative are approximately \(p_1=0.02275\) and \(p_2=0.9332\).
The first stage supports benefit, while the second stage’s negative statistic is in the opposite direction. Nevertheless, both p-values are correctly calculated for the prespecified benefit alternative. Combining them gives
With four degrees of freedom, the combined upper-tail probability is approximately
The combined test does not reject at the 0.05 level. The large second-stage p-value for benefit offsets some of the first stage’s evidence. This result illustrates why the sign of each stage’s estimate should be checked alongside the combined statistic: a p-value alone does not tell the reader whether that stage’s estimate pointed toward or away from benefit.
Worked Example: A Prespecified Lower-Tail Alternative
A separate hypothetical trial defines a smaller outcome as clinically favorable and prespecifies the alternative \(\theta<0\). Its independent stages produce Z statistics \(z_1=-2.10\) and \(z_2=-1.80\). Under standard normal calibration, the lower-tail p-values are approximately \(p_1=0.0179\) and \(p_2=0.0359\). Both stages test the same direction.
The product and Fisher statistic are
Using the four-degree-of-freedom upper tail,
At a 0.05 significance level, this is evidence against the null in the prespecified lower-tail direction. The direction need not always be “greater than”; consistency means testing the same scientifically chosen direction at every stage.
Checks, Alternatives, and Limitations
Before reporting a one-sided Fisher test, verify that the stagewise p-values correspond to the same parameter, null boundary, and alternative direction. Then check that their calibration and independence justify the reference distribution. A small combined p-value is not a substitute for these checks.
Write the common \(H_a\) explicitly, such as \(\theta>\theta_0\), before examining stage results.
Confirm that every value is computed for that same one-sided alternative and is valid under the null.
Assess independence and whether the usual chi-square calibration is justified for the trial’s design.
Compute \(X_F\), use the \(2m\)-degree-of-freedom upper tail when its assumptions hold, and describe the conclusion in the prespecified direction.
A useful comparison is the inverse normal combination test. It transforms stagewise evidence to Z-scale and combines it with specified weights; when signed Z statistics are used, the stage directions remain visible in the combination. Fisher’s method instead combines the p-values through their logarithms and can be attractive when the inputs are valid p-values from different stagewise tests. Neither method repairs invalid p-values or dependence that has been ignored. Their combination statistics also weight evidence differently, so they need not produce the same result.
Fisher combination has limits even when its assumptions hold. A significant result is evidence against the combined null, not a guarantee of a clinically important effect, proof that every stage showed benefit, or an estimate of the treatment effect. The statistic can be strongly influenced by a very small p-value from one stage, while a stage with a large p-value contributes little evidence. Examine stage estimates and their uncertainty as well as the combined test.
Common Mistakes and Diagnostics
- Choosing the favorable tail separately in each stage. This uses the observed data to choose the direction and does not produce the prespecified one-sided p-values. Choose the common direction before looking at results.
- Assuming Fisher’s statistic records the sign of the effect. The statistic only sees the p-values. Report stage estimates and confirm that each p-value was calculated for the intended direction.
- Applying the chi-square reference without checking independence. If stage data or test statistics are dependent, the standard reference distribution need not be correct. Establish an appropriate calibration for the design.
- Interpreting a combined rejection as stage-by-stage success. Fisher’s test assesses the combined null; it does not require every stage to be individually significant or show an effect in the favorable direction.
- Entering a rounded p-value of zero into the logarithm. Since \(\log(0)\) is undefined, preserve numerical precision and obtain a valid nonzero p-value rather than treating a printed zero as an exact value.
The most important diagnostic is a simple stage-by-stage table of effect estimates, their directions, and the one-sided p-values used. It can reveal a direction mismatch that is hidden by the final combined p-value and can help distinguish consistent evidence from a result driven by one stage.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Two independent stages test the same one-sided alternative and have p-values 0.10 and 0.20. Calculate Fisher’s statistic and identify its reference degrees of freedom.
- Why is it invalid to use the smaller of the upper-tail and lower-tail p-values separately at each stage when the trial prespecified a single direction?
- In a benefit trial, one stage has a negative treatment-effect estimate. What should happen to its p-value if it is calculated correctly for the prespecified benefit alternative?
- Name two conditions needed for the usual chi-square reference distribution for Fisher’s statistic to be exact.
Solutions. (1) \(X_F=-2\log[(0.10)(0.20)]=-2\log(0.0200)\approx7.824\), with \(2m=4\) degrees of freedom. (2) Selecting the tail after observing the stage result makes the direction data-dependent, so those values are not the prespecified one-sided p-values used by the standard calibration. (3) A negative estimate points away from benefit and should generally yield a large upper-tail p-value, contributing little evidence for benefit. (4) Under the null, the p-values must be independent and exactly uniformly distributed; their stagewise tests must also be valid for the stated null and direction.