Comparing Inverse Normal and Fisher Calibration

Why Calibration Matters

Inverse normal and Fisher combination tests both turn stagewise p-values into a single test of the global null hypothesis. They do not, however, use the same transformation or the same reference distribution. A combined statistic has no calibrated meaning until its null distribution is specified: that distribution determines both the critical value and the p-value.

This comparison uses the same stagewise evidence for both methods. Unless stated otherwise, assume that each stage supplies a valid, continuous, one-sided p-value in a prespecified direction, and that under the global null the stagewise p-values are mutually independent and uniformly distributed on \((0,1)\). The combination rule, number of stages, and any inverse normal weights are fixed before the stagewise results are observed. These assumptions provide the standard exact reference distributions described below.

Definition: For \(K\) independent stagewise p-values \(p_1,\ldots,p_K\), the inverse normal combination statistic is \(Z_{\mathrm{IN}}=\sum_{i=1}^{K} w_i\Phi^{-1}(1-p_i)\), where \(\Phi\) is the standard normal cumulative distribution function and the prespecified weights satisfy \(\sum_{i=1}^{K}w_i^2=1\). The Fisher combination statistic is \(T_{\mathrm{F}}=-2\sum_{i=1}^{K}\ln(p_i)\). Under the stated assumptions, \(Z_{\mathrm{IN}}\sim N(0,1)\) and \(T_{\mathrm{F}}\sim\chi^2_{2K}\) under the global null.

Inverse Normal Calibration

For a one-sided p-value \(p_i\), the transformed value \(\Phi^{-1}(1-p_i)\) is large and positive when that stage gives evidence in the prespecified direction, and large and negative when it gives evidence in the opposite direction. Inverse normal combination adds these transformed values, with the prespecified weights, then compares the result with the standard normal distribution. The combined upper-tail p-value is \(1-\Phi(Z_{\mathrm{IN}})\).

The squared-weight condition is essential to the stated calibration. Under the global null, uniformity of \(p_i\) implies that \(1-p_i\) is also uniform on \((0,1)\). Applying the inverse standard normal distribution to a uniform variable gives a standard normal variable. Independence of the stagewise p-values therefore gives independent standard normal transformed values. A weighted sum of independent standard normal variables is normal, with mean zero and variance \(\sum_i w_i^2=1\). Thus the combined statistic is standard normal.

\sum_{i=1}^{K} w_i\Phi^{-1}(1-p_i)\sim N\left(0,\sum_{i=1}^{K}w_i^2\right)=N(0,1)

For a one-sided test at significance level \(\alpha\), reject \(H_0\) when \(Z_{\mathrm{IN}}>z_{1-\alpha}\), where \(z_{1-\alpha}\) is the \((1-\alpha)\) quantile of the standard normal distribution. Equivalently, reject when the combined p-value \(1-\Phi(Z_{\mathrm{IN}})\) is at most \(\alpha\). For example, at \(\alpha=0.05\), the critical value is approximately \(1.645\).

Worked Example: Two-Stage Inverse Normal Test

Suppose a hypothetical two-stage trial produces one-sided stagewise p-values \(p_1=0.10\) and \(p_2=0.30\). The design specified weights \(w_1=\sqrt{0.70}\) and \(w_2=\sqrt{0.30}\) before either result was observed. They are valid normalized weights because \(w_1^2+w_2^2=0.70+0.30=1\). Assume the p-values are independent under the global null.

The transformed stagewise values are \(\Phi^{-1}(0.90)\approx1.2816\) and \(\Phi^{-1}(0.70)\approx0.5244\). Therefore,

Z_{\mathrm{IN}} =\sqrt{0.70}(1.2816)+\sqrt{0.30}(0.5244) \approx 0.8367(1.2816)+0.5477(0.5244) \approx 1.360

The combined upper-tail p-value is \(1-\Phi(1.360)\approx0.0870\). Since \(1.360<1.645\), the test does not reject \(H_0\) at the 0.05 level. Under the assumed null model, this p-value is the probability of obtaining an inverse normal statistic at least as large as 1.360.

Fisher Calibration

Fisher’s statistic aggregates the negative logarithms of the stagewise p-values. Small p-values contribute larger positive terms, so evidence from several stages can accumulate even when no single stage is decisive. Its standard calibration follows from the distribution of a transformed uniform variable: if \(U\sim\mathrm{Uniform}(0,1)\), then \(-2\ln(U)\) has a chi-square distribution with two degrees of freedom.

To see this, for \(x\geq0\), the event \(-2\ln(U)\leq x\) is equivalent to \(U\geq e^{-x/2}\). Hence \(P(-2\ln(U)\leq x)=1-e^{-x/2}\), the chi-square-2 cumulative distribution function. With independent uniform p-values, the \(K\) transformed terms are independent chi-square-2 variables; their sum has a chi-square distribution with \(2K\) degrees of freedom.

-2\ln(p_i)\sim\chi^2_2 \qquad\Longrightarrow\qquad T_{\mathrm{F}}=\sum_{i=1}^{K}-2\ln(p_i)\sim\chi^2_{2K}

For Fisher’s one-sided combination test at level \(\alpha\), reject \(H_0\) when \(T_{\mathrm{F}}>\chi^2_{2K,1-\alpha}\), the \((1-\alpha)\) quantile of the chi-square distribution with \(2K\) degrees of freedom. The Fisher combined p-value is the upper-tail probability \(P(\chi^2_{2K}\geq T_{\mathrm{F}})\). Unlike inverse normal, the standard Fisher statistic has no stage weights.

Worked Example: Fisher Test on the Same Two-Stage Evidence

Use the same hypothetical stagewise p-values, \(p_1=0.10\) and \(p_2=0.30\), with the same independence assumption. Fisher’s statistic is

T_{\mathrm{F}} =-2\{\ln(0.10)+\ln(0.30)\} =-2\ln(0.030) \approx 7.013

There are two stages, so the null reference distribution is \(\chi^2_4\). Its 0.95 quantile is approximately \(9.488\). Since \(7.013<9.488\), Fisher’s test does not reject \(H_0\) at \(\alpha=0.05\). Its calibrated combined p-value is \(P(\chi^2_4\geq7.013)\approx0.135\).

The inverse normal and Fisher p-values for this evidence are not numerically equal: they are approximately 0.0870 and 0.135, respectively. Each is an upper-tail probability under its own specified reference distribution; neither should be interpreted using the other method’s statistic or cutoff.

Same Evidence, Different Reference Distributions

The next calculation puts the two calibrations side by side for a different hypothetical pair of stagewise results. It also shows why one must calculate each method’s critical value from its own reference distribution.

Worked Example: Comparing Critical Values and Combined P-Values

Suppose two independent stages produce one-sided p-values \(p_1=0.04\) and \(p_2=0.20\). Use equal inverse normal weights \(w_1=w_2=1/\sqrt{2}\), which satisfy \(w_1^2+w_2^2=1\). Under each method, the assumptions for its standard null calibration are the independent, valid uniform p-values specified above.

For inverse normal, \(\Phi^{-1}(0.96)\approx1.7507\) and \(\Phi^{-1}(0.80)\approx0.8416\), so

Z_{\mathrm{IN}} =\frac{1.7507+0.8416}{\sqrt{2}} \approx1.833

The inverse normal combined p-value is \(1-\Phi(1.833)\approx0.0334\). At \(\alpha=0.05\), \(1.833>1.645\), so this method rejects \(H_0\).

For Fisher, the product of the stagewise p-values is \(0.04(0.20)=0.008\), giving

T_{\mathrm{F}}=-2\ln(0.008)\approx9.657

The Fisher reference distribution is \(\chi^2_4\), whose 0.95 quantile is \(9.488\). Thus \(9.657>9.488\), so Fisher’s test also rejects \(H_0\). Its combined p-value is \(P(\chi^2_4\geq9.657)=e^{-9.657/2}(1+9.657/2)\approx0.0466\). Both methods reject at 0.05 here, but their calculated p-values differ because their transformations and null reference distributions differ.

Feature	Inverse normal	Fisher
Stagewise transformation	\(\Phi^{-1}(1-p_i)\)	\(-2\ln(p_i)\)
Combined statistic	Weighted sum with \(\sum_i w_i^2=1\)	Sum of transformed p-values
Standard null distribution	\(N(0,1)\)	\(\chi^2_{2K}\)
One-sided rejection rule	Compare with \(z_{1-\alpha}\)	Compare with \(\chi^2_{2K,1-\alpha}\)
Combined p-value	Standard normal upper tail	Chi-square upper tail

Assumptions, Diagnostics, and Common Errors

The reference distributions are not automatic properties of the formulas. They rely on valid stagewise p-values under the null, independence across stages, and the specified combination rule. Inverse normal also requires weights chosen in advance and normalized so their squared sum is one. If the p-values are dependent, the normal and chi-square reference distributions need not describe the resulting statistics; applying the usual cutoffs can then miscalibrate the test. Conditional validity alone should not be silently substituted for the independence assumption used in these standard calibrations.

A practical check is to record, before combining results, the hypothesis direction, stagewise p-value definition, number of stages, weights if applicable, dependence assumptions, and significance level. Also check that the reported combined p-value agrees with the statistic’s own reference distribution. For example, an inverse normal statistic must be evaluated using a normal upper tail, not a chi-square tail. Very small p-values rounded to zero also require care: the logarithm in Fisher’s formula is undefined at zero, and rounding can change the combined statistic. Use adequate numerical precision and a valid p-value representation rather than treating a printed zero as an exact input.

Common mistake: “Both methods use the same stagewise p-values, so they must have the same combined p-value and cutoff.” This is incorrect. The methods transform the inputs differently and use different reference distributions. Calculate the statistic, critical value, and upper-tail probability as a matched set for the chosen method. A second mistake is to use \(\Phi^{-1}(p_i)\) in place of \(\Phi^{-1}(1-p_i)\) for an upper-tail p-value: that reverses the evidence direction and can turn strong evidence into a negative transformed value.

Neither standard calibration is universally interchangeable with the other. Inverse normal makes the role of prespecified stage weights explicit; Fisher uses an unweighted sum of log p-values in its standard form. The calculations above compare their calibration, not which scientific design is preferable. If the independence or validity assumptions are not supported, one should not report the standard cutoffs as though they were justified; an appropriate joint calibration is needed. The specific behavior of the methods for different stagewise evidence patterns is a separate question from establishing their reference distributions.

Key takeaway: Inverse normal is calibrated against \(N(0,1)\) after squared-normalized weighting; Fisher is calibrated against \(\chi^2_{2K}\) under independent uniform stagewise p-values. Always pair each statistic with its own critical value and upper-tail p-value.

Check Your Understanding

For each question, use one-sided p-values and the standard independent-stage calibration unless the question states otherwise. Solutions are included below.

For two stages, what are the inverse normal and Fisher null reference distributions? Give the respective 0.05 upper-tail critical values to three significant figures.
With equal inverse normal weights and stagewise p-values \(0.20\) and \(0.20\), calculate the combined statistic and approximate combined p-value. Does the test reject at 0.05?
For two Fisher stages with p-values \(0.05\) and \(0.10\), calculate \(T_{\mathrm{F}}\), its degrees of freedom, and the approximate combined p-value.
Why is it not valid to use the usual standard-normal inverse normal calibration automatically when the stagewise p-values are dependent?

Solutions. (1) The inverse normal statistic has a standard normal null distribution, with critical value \(z_{0.95}\approx1.645\). Fisher’s statistic has a \(\chi^2_4\) null distribution, with critical value \(\chi^2_{4,0.95}\approx9.488\). (2) Each transformed value is \(\Phi^{-1}(0.80)\approx0.8416\), so \(Z_{\mathrm{IN}}=(0.8416+0.8416)/\sqrt{2}\approx1.190\). Its upper-tail p-value is \(1-\Phi(1.190)\approx0.117\); it does not reject at 0.05. (3) \(T_{\mathrm{F}}=-2\ln(0.05\cdot0.10)=-2\ln(0.005)\approx10.597\), with \(df=4\). The p-value is \(e^{-10.597/2}(1+10.597/2)=0.005(1+5.2985)\approx0.0315\). (4) Independence is used to establish the variance of the weighted normal sum and the chi-square distribution of the Fisher sum. Dependence can change those joint distributions, so the usual reference distribution and resulting cutoff are not guaranteed to control the stated error rate.