When Fisher and Inverse Normal Favor Different Evidence

Why the Same Stagewise Results Can Tell Different Stories

A combination test compresses stagewise evidence into one overall test. That compression depends on the chosen function: a very small p-value can have a different influence from a collection of moderate p-values. Consequently, Fisher’s combination test and the inverse-normal combination test can give different decisions from exactly the same stagewise results. Neither conclusion is automatically an error. The tests use different functions of the inputs and are sensitive to different patterns of evidence.

The aim here is to compare those functions through concrete calculations. The examples concern hypothetical two-stage clinical trials and a one-sided test of benefit. The p-values are assumed to have been calculated for the same direction of benefit, with smaller values indicating stronger evidence against the null hypothesis.

For a one-sided test at significance level \(\alpha\), Fisher’s test rejects when the upper-tail probability \(P(\chi^2_4\geq T_F)\) is at most \(\alpha\). The inverse-normal test rejects when \(Z_{IN}\geq\Phi^{-1}(1-\alpha)\), equivalently when its upper-tail p-value \(1-\Phi(Z_{IN})\) is at most \(\alpha\). In both cases the conclusion concerns the global null hypothesis that the stagewise tests are testing; it does not establish that every stage independently demonstrated benefit.

Assumptions Behind the Comparison

The reference distributions above are exact under a specific set of conditions. Under the global null, each stagewise p-value must be uniform on \((0,1)\), and the two p-values must be independent. Uniformity is the usual condition for a continuous, valid null p-value. Independence may be justified when the stages use non-overlapping data and the stagewise analyses are independent under the null; it should not be assumed just because the data are labelled as two stages.

The inverse-normal weights must be specified before the stagewise results are observed and must satisfy \(w_1^2+w_2^2=1\). For example, equal weights are \(w_1=w_2=1/\sqrt{2}\). If one stage is assigned more weight, the other must receive less in a way that preserves this normalization. Fisher’s statistic as defined here has no stage weights: it treats the p-values symmetrically through their product.

These assumptions concern calibration, not whether the tests will have equal power under every alternative. Fisher’s and inverse-normal tests are both valid at the stated level when their assumptions hold, but their power can differ across patterns of stagewise evidence. If stagewise p-values are dependent, discrete and not uniform, or otherwise not calibrated as assumed, the usual reference distributions may not provide the stated error control.

How the Combination Functions Weight Evidence

Fisher’s statistic can be written as a sum of stage contributions, \(-2\log(p_i)\). A smaller stagewise p-value gives a larger positive contribution. A p-value of 0.5 contributes \(1.386\) to the statistic, while a p-value of 0.05 contributes \(5.991\), and a p-value of 0.001 contributes \(13.816\). Every stage adds to the statistic, so a very small p-value can make a substantial contribution even when another stage has a large p-value.

The inverse-normal test first turns each p-value into a signed normal score, \(\Phi^{-1}(1-p_i)\), then multiplies that score by its stage weight. A p-value below 0.5 gives a positive score, a p-value above 0.5 gives a negative score, and a p-value of 0.5 gives a score of zero. Thus, a weak or opposing stage can offset evidence from another stage. Weights determine how much each signed score matters.

The contrast follows from the transformations themselves. Fisher’s contribution grows in proportion to \(\log(1/p)\) as \(p\) becomes very small. The inverse-normal score also grows as \(p\) decreases, but it grows on a normal-score scale, approximately \(\sqrt{2\log(1/p)}\) for very small \(p\). Meanwhile, the inverse-normal transformation preserves the sign of each stage’s evidence, whereas Fisher’s log contributions are always nonnegative. These differences make particular evidence patterns matter differently.

When the Inverse-Normal Test Favors the Evidence

The reverse pattern is also possible. Inverse normal can favor a particular pattern when the more compelling stage is assigned a larger weight. Fisher, as defined here, is symmetric in the stage p-values and does not use those weights. The next example illustrates why the prespecified weight choice matters to the comparison.

A Pattern Where Both Tests Reject

Different combined statistics do not always produce different decisions. When both stages have moderately small p-values, both procedures may reject, although their combined p-values need not be the same. Checking the actual calibrated tail probability is essential: comparing the raw values of \(T_F\) and \(Z_{IN}\) is meaningless because they have different scales and reference distributions.

Interpreting Disagreement and Diagnosing Problems

A disagreement is a reason to inspect the evidence pattern and assumptions, not to select whichever test gives the preferred conclusion after seeing the results. Before interpreting a difference, check that the inputs are p-values for the same global null and the same prespecified direction, that each stagewise p-value is valid, and that the assumed independence is credible. For inverse normal, also verify the weights and their normalization. The test’s significance level applies to its calibrated combined statistic, not to each stagewise p-value separately.

The most common error is to say, “Fisher found significance, so the inverse-normal result must be wrong,” or the reverse. That reasoning treats the tests as if they used the same evidence function. They do not. A second mistake is to replace a p-value above 0.5 with its complement before applying the inverse-normal transformation. For the stated one-sided convention, the score is \(\Phi^{-1}(1-p_i)\); a p-value of 0.80 therefore has score \(\Phi^{-1}(0.20)<0\). Changing it to 0.20 would reverse the evidence direction and produce a different test.

A third diagnostic issue is an input recorded as zero because software rounded a very small p-value. The logarithm in Fisher’s statistic is undefined at zero, and the normal quantile in inverse normal is infinite at \(p=0\). Use an unrounded valid p-value or an appropriate method for the test that generated it; do not treat a rounded zero as an exact probability of zero. Also investigate suspiciously identical p-values or unexpectedly extreme combined statistics for data-processing and dependence problems.

The comparison has limits. The examples assume independent stages and continuous, calibrated one-sided p-values; the numerical conclusions need not transfer unchanged when those conditions fail. The methods combine evidence against a global null, but neither by itself supplies an estimate of treatment effect, establishes clinical importance, or shows that effects were equal across stages. Nor does a smaller combined p-value imply that one test is universally preferable. The choice should reflect the design’s scientific rationale and be made before inspecting the results.