K-Stage Inverse Normal Combination Tests

Why Extend the Test to K Stages?

A trial may collect evidence in more than two stages. For example, investigators might plan several independent cohorts, each producing a stagewise test result, and then combine those results for one overall test. The inverse normal method extends directly: transform each stage’s p-value to a normal score, weight the scores according to a plan specified in advance, and add them.

The method combines evidence for a hypothesis; it does not, by itself, combine treatment-effect estimates or provide an estimate of the effect. Its calibration also depends on how the stagewise p-values are constructed and how the stages are related. In particular, the familiar standard-normal reference distribution requires independent stages and suitable stagewise p-values.

Definition: For $K$ stages, let $p_j$ be the one-sided p-value from stage $j$, with smaller values representing stronger evidence in the prespecified direction. Let $w_j$ be its prespecified, nonnegative weight, where $\sum_{j=1}^{K} w_j^2=1$. The inverse normal combination statistic is $$ Z_{\mathrm{comb}}=\sum_{j=1}^{K}w_j\Phi^{-1}(1-p_j). $$ For an upper-direction test at significance level $\alpha$, reject $H_0$ when $Z_{\mathrm{comb}}>z_{1-\alpha}$, where $z_{1-\alpha}=\Phi^{-1}(1-\alpha)$.

The p-values must refer to the same testing direction. For instance, if positive values of a stage statistic support a beneficial treatment effect, each $p_j$ must be a one-sided p-value for that positive direction. This formula is for an upper-direction combination test; unsigned two-sided p-values should not simply be inserted into it as though they were directionally aligned.

A transformed score is not necessarily positive. Since $\Phi^{-1}(1-p_j)$ is positive when $p_j<0.5$, zero when $p_j=0.5$, and negative when $p_j>0.5$, a stage with weak evidence in the specified direction can make a negative contribution. For a positive weight, a smaller p-value gives a larger score and a larger contribution; the score itself is positive only when $p_j<0.5$.

Why the Standard-Normal Calibration Works

Suppose that under the null hypothesis each stagewise p-value is continuous and uniformly distributed on $(0,1)$, and that the stagewise p-values are independent across stages. These assumptions hold, for example, when independent stage data produce valid continuous p-values under the null. The weights are fixed in advance and satisfy the normalization in the definition.

If $P_j\sim\operatorname{Uniform}(0,1)$, then $1-P_j$ is also uniform on $(0,1)$. Applying the standard normal quantile function to a uniform random variable gives a standard normal random variable. Thus each $Z_j=\Phi^{-1}(1-P_j)$ has distribution $N(0,1)$. Independence of the p-values gives independence of these transformed scores.

Justification: A weighted sum of independent normal random variables is normal. Its mean and variance under the null are $$ E(Z_{\mathrm{comb}})=\sum_{j=1}^{K}w_j E(Z_j)=0, \qquad \operatorname{Var}(Z_{\mathrm{comb}})=\sum_{j=1}^{K}w_j^2\operatorname{Var}(Z_j)=\sum_{j=1}^{K}w_j^2=1. $$ Therefore $Z_{\mathrm{comb}}\sim N(0,1)$ under these assumptions. Its upper-tail p-value is $1-\Phi(Z_{\mathrm{comb}})$, and comparing it with $z_{1-\alpha}$ gives a test with null rejection probability $\alpha$.

This result explains why the weights must be normalized by their squared sum, rather than by their ordinary sum. If $\sum_j w_j^2\ne1$, the combined statistic does not have variance one under the stated assumptions. One can rescale weights that were specified on another scale, but the values used in the formula must satisfy the normalization.

Worked Example: Combining Three Independent Stages

Hypothetical data. A trial has three independent stages, each testing whether the treatment improves the outcome. The prespecified weights are $w_1=0.5000$, $w_2=0.5477$, and $w_3=0.4472$, corresponding to squared weights $0.25$, $0.30$, and $0.20$. Their squared sum is $0.75$, not 1, so use the normalized weights $0.5000$, $0.5477$, and $0.4472$ only if normalized correctly?

Here the intended weights are $w_1=\sqrt{0.25}=0.5000$, $w_2=\sqrt{0.30}=0.5477$, and $w_3=\sqrt{0.20}=0.4472$. Their squared sum is $0.25+0.30+0.20=0.75$, so these weights are not normalized. To preserve the planned relative weights, divide each by $\sqrt{0.75}$: the normalized weights are $0.5774$, $0.6325$, and $0.5164$. The observed one-sided p-values are $0.10$, $0.04$, and $0.20$.

The stage scores are $$ \Phi^{-1}(1-0.10)=1.2816,\qquad \Phi^{-1}(1-0.04)=1.7507,\qquad \Phi^{-1}(1-0.20)=0.8416. $$

Therefore, $$ Z_{\mathrm{comb}} =(0.5774)(1.2816)+(0.6325)(1.7507)+(0.5164)(0.8416) =0.7400+1.1073+0.4346 =2.2819. $$

At one-sided $\alpha=0.025$, the critical value is $z_{0.975}=1.9600$. Since $2.2819>1.9600$, reject $H_0$. The combined p-value is approximately $1-\Phi(2.2819)=0.0112$. Under the assumed null calibration, this is evidence against no treatment benefit in the prespecified direction; it is not the probability that $H_0$ is true.

The normalization step matters in practice. The displayed weights in a protocol or analysis plan should already satisfy $\sum w_j^2=1$; the rescaling above makes explicit what to do if relative weights, rather than normalized weights, are initially supplied. In the remaining examples the weights are normalized as stated.

Reading Contributions and Making the Decision

Each term $w_j\Phi^{-1}(1-p_j)$ is stage $j$’s contribution to the total. A stage with a small p-value in the chosen direction contributes positively, while a p-value above 0.5 produces a negative score. A negative contribution is not an error: it records that the stage result points away from the specified direction. The total, rather than any single stage’s p-value, determines the combination-test decision.

Worked Example: A Stage Can Contribute a Negative Score

Hypothetical data. Three independent stages have equal weights $w_j=1/\sqrt{3}=0.5774$. Their one-sided p-values for benefit are $0.80$, $0.20$, and $0.05$. The transformed scores are $\Phi^{-1}(0.20)=-0.8416$, $\Phi^{-1}(0.80)=0.8416$, and $\Phi^{-1}(0.95)=1.6449$. The first score is negative because its p-value exceeds 0.5.

The combined statistic is $$ Z_{\mathrm{comb}} =\frac{-0.8416+0.8416+1.6449}{\sqrt{3}} =\frac{1.6449}{1.7321} =0.9497. $$

At one-sided $\alpha=0.05$, the critical value is $1.6449$. Since $0.9497<1.6449$, do not reject $H_0$. The combined upper-tail p-value is approximately $1-\Phi(0.9497)=0.171$. It would be wrong to describe the first stage’s score as positive merely because its p-value was transformed to a normal score.

Worked Example: Four Stages with Equal Weights

Hypothetical data. Four independent stages use equal prespecified weights. Since the squared weights must sum to 1, each weight is $w_j=1/\sqrt{4}=0.5000$. Suppose each stage yields a one-sided p-value of $0.10$ for benefit. Each normal score is $\Phi^{-1}(0.90)=1.2816$.

The combined statistic is $$ Z_{\mathrm{comb}}=4(0.5000)(1.2816)=2.5632. $$

At one-sided $\alpha=0.01$, the critical value is $z_{0.99}=2.3263$. The statistic exceeds this value, so reject $H_0$ at the 1% level. The combined p-value is approximately $1-\Phi(2.5632)=0.0052$. Although no individual stage has a p-value as small as 0.01, their aligned evidence produces a combined result below 0.01 under the stipulated independence and calibration assumptions.

Assumptions, Diagnostics, and Alternatives

Before interpreting a result, verify the conditions behind its reference distribution. The stagewise analyses must test the same null hypothesis in the same direction, and the p-values must be calibrated under that null. The stage data—and hence the stagewise p-values—must be independent for the simple standard-normal derivation above. Stages that reuse participants or otherwise share data may induce dependence, changing the variance of the weighted sum.

Check the direction. Confirm that small p-values consistently favor the same prespecified direction. Reversing the direction in one stage reverses the meaning of its transformed score.
Check independence. Confirm that stages use independent data under the null. Do not assume independence merely because results are reported separately.
Check the weights. Verify that weights were fixed without looking at the stage results and that their squared sum is 1.
Check p-value resolution. A continuous uniform null p-value gives the exact normal transform used in the derivation. Discrete or rounded p-values may not have that distribution; use a calibration appropriate to the stage tests rather than treating the standard-normal result as automatic.
Check the reference tail. For the upper-direction test, use $1-\Phi(Z_{\mathrm{comb}})$ and the upper critical value $z_{1-\alpha}$, not a two-sided cutoff unless a two-sided procedure has been specified.

The inverse normal method is not the only way to combine stagewise evidence. Fisher combination, introduced in the earlier material on Fisher’s combination statistic, combines p-values through their logarithms and uses a different reference distribution. The inverse normal method instead maps each p-value to a signed normal score and allows prespecified stage weights. These methods can respond differently to the pattern of stagewise evidence; choosing between them is a design decision, not a choice to make after seeing which gives the smaller p-value. A stagewise test considered alone is another alternative, but it does not use the full planned combination of evidence.

The procedure has limits. Its combined statistic is not automatically an estimate of treatment effect, and a significant result does not establish that every stage showed benefit. The standard-normal reference also cannot repair invalid stagewise p-values or unaddressed dependence. If the assumptions fail, the numerical result may still be calculable, but its stated type I error calibration is not justified by this derivation.

Common Mistakes

Calling every transformed score positive. A smaller p-value gives a larger normal score, but that score is positive only when $p_j<0.5$. For example, $p_j=0.80$ gives $\Phi^{-1}(0.20)\approx-0.8416$.
Using weights that do not have squared sum 1. The resulting statistic will not have unit variance under the stated null model. Normalize the weights before using the standard-normal reference.
Combining p-values for opposing directions. The transform is directional; align the one-sided alternatives before combining.
Treating separate reports as independent stages. Independence is a property of the underlying data and test results, not of how results are presented.
Interpreting the combined p-value as a probability that the null is true. It is a tail probability calculated under the null reference distribution, not a posterior probability for a hypothesis.

Key takeaway: With independent, properly calibrated stagewise p-values and fixed nonnegative weights satisfying $\sum_j w_j^2=1$, the K-stage inverse normal statistic has a standard normal null distribution. Preserve the testing direction, allow transformed scores to be negative when appropriate, and verify the assumptions before using the resulting p-value.

Check Your Understanding

Work each question, then compare your reasoning with the solutions.

For a stagewise p-value of $0.70$, is $\Phi^{-1}(1-p)$ positive or negative? What is its approximate value?
Three stages have weights $0.5$, $0.5$, and $0.5$. Do these satisfy the required normalization? If not, what common factor normalizes them?
Two independent stages have equal weights and one-sided p-values $0.10$ and $0.20$. Calculate the combination statistic and decide whether to reject at one-sided $\alpha=0.05$.
Why does dependence between stagewise p-values invalidate the simple variance calculation for the combination statistic?

Solutions. (1) It is negative because $1-p=0.30<0.5$; $\Phi^{-1}(0.30)\approx-0.5244$. (2) Their squared sum is $0.75$, so they are not normalized. Divide each by $\sqrt{0.75}$, giving weights approximately $0.5774$ each. (3) Equal weights are $1/\sqrt{2}=0.7071$. The scores are $1.2816$ and $0.8416$, so $Z_{\mathrm{comb}}=0.7071(1.2816+0.8416)=1.5013$. Since $1.5013<z_{0.95}=1.6449$, do not reject; the upper-tail p-value is approximately $0.0666$. (4) With dependence, the variance includes covariance terms: $\operatorname{Var}(\sum_jw_jZ_j)=\sum_jw_j^2\operatorname{Var}(Z_j)+2\sum_{i<j}w_iw_j\operatorname{Cov}(Z_i,Z_j)$. It need not equal 1, so the standard-normal reference derived under independence is not justified.