One-Sided Inverse Normal Combination Tests

Why Tail Direction Matters

An inverse normal combination test converts stagewise p-values to normal scores and combines those scores using weights. The transformation is deliberately chosen so that small p-values become large positive scores. For a one-sided test, that interpretation depends on how each stagewise p-value was defined: its small tail must represent evidence in the prespecified direction of the alternative hypothesis.

For example, if the alternative is that a treatment improves an outcome, a right-sided p-value may measure evidence for an increase. If the alternative is that treatment reduces an outcome, the relevant one-sided p-value may instead be left-sided. The combination calculation has one standard form, but the stagewise p-values supplied to it must all refer to the same null hypothesis and the same direction of evidence.

Definition: For $K$ stages, let $p_i$ be the one-sided p-value from stage $i$, defined so that smaller values indicate stronger evidence against $H_0$ in favor of the same prespecified $H_a$. Let $w_i$ be fixed, nonnegative weights satisfying $\sum_{i=1}^{K}w_i^2=1$. The one-sided inverse normal combination statistic is $$ C=\sum_{i=1}^{K}w_i\Phi^{-1}(1-p_i), $$ where $\Phi$ is the standard normal cumulative distribution function. Under the standard null conditions given below, reject $H_0$ at level $\alpha$ when $C>z_{1-\alpha}$, where $z_{1-\alpha}=\Phi^{-1}(1-\alpha)$.

Orienting the Stagewise P-Values

The inverse normal transformation uses $1-p_i$, not $p_i$. This is the upper-tail convention for the normal scores: a small p-value gives $1-p_i$ near one and therefore a positive score. A p-value near one-half gives a score near zero, while a large p-value gives a negative score.

Suppose a stage uses a standardized statistic $T_i$ with a standard normal null distribution. If large values support a right-sided alternative, its p-value is $p_i=1-\Phi(T_i)$. Then $\Phi^{-1}(1-p_i)=\Phi^{-1}(\Phi(T_i))=T_i$. For a left-sided alternative, small values of $T_i$ support $H_a$, and its p-value is $p_i=\Phi(T_i)$. In this case, $\Phi^{-1}(1-p_i)=\Phi^{-1}(1-\Phi(T_i))=-T_i$. Thus either direction can yield a positive score for evidence favoring $H_a$, provided that the p-value is computed using the correct alternative tail.

This orientation is not a license to reverse a p-value after seeing which direction looks favorable. The direction of $H_a$, the test statistic convention, and the corresponding tail must be specified before examining the stage results. A p-value calculated in the wrong tail does not become a valid one-sided p-value merely because it is passed through $\Phi^{-1}(1-p_i)$.

Null Distribution and Rejection Rule

The usual calibration has a clear set of conditions. Under $H_0$, each stagewise p-value must be exactly uniform on $(0,1)$, and the stagewise p-values must be independent. Continuous test statistics with correctly calibrated p-values give exact uniformity; discrete or otherwise conservative p-values generally do not. The weights must be nonnegative, fixed for the test, and have squared sum one. The p-values must test the same null hypothesis in the same alternative direction.

Independence is often justified by independent stage data or independent stage increments under the null. If the stagewise p-values are dependent, or if the weights are chosen using the stage results in a way not covered by the calibration, the standard normal reference distribution need not apply. In those circumstances, calibration must account for the actual joint design rather than assuming the result below.

Theorem: If $p_1,\ldots,p_K$ are independent and each is Uniform$(0,1)$ under $H_0$, and the weights satisfy $\sum_{i=1}^{K}w_i^2=1$, then $C\sim N(0,1)$ under $H_0$. The level-$\alpha$ one-sided test rejects for $C>z_{1-\alpha}$, and its combined p-value is $1-\Phi(C)$.

The result follows directly from the probability transform and the distribution of a weighted sum of independent normal variables.

P_i\sim \operatorname{Uniform}(0,1) \quad\Longrightarrow\quad 1-P_i\sim \operatorname{Uniform}(0,1) \quad\Longrightarrow\quad Z_i=\Phi^{-1}(1-P_i)\sim N(0,1).

Independence of the p-values implies independence of their transformed scores. A linear combination of independent normal variables is normal, with mean zero and variance equal to the sum of the squared coefficients:

C=\sum_{i=1}^{K}w_iZ_i \sim N\left(0,\sum_{i=1}^{K}w_i^2\right) =N(0,1).

Consequently, under $H_0$, $P(C>z_{1-\alpha})=\alpha$. For example, at $\alpha=0.05$, the right-tail critical value is approximately $1.645$. If the p-values are independent but conservative rather than exactly uniform, the resulting test is generally conservative under the corresponding null conditions; exact standard normal calibration should not be asserted without exact uniformity.

Worked Example: Combining Right-Sided Stagewise Evidence

Suppose two independent stages test the same right-sided alternative, with stagewise p-values $p_1=0.040$ and $p_2=0.200$. Use weights $w_1=0.6$ and $w_2=0.8$, which satisfy $0.6^2+0.8^2=1$. Test at one-sided $\alpha=0.05$.

Because the alternative is right-sided, each p-value is the probability under the null of a statistic at least as large as the observed statistic. The inverse normal scores use $1-p_i$:

z_1=\Phi^{-1}(1-0.040)=\Phi^{-1}(0.960)\approx1.7507, \qquad z_2=\Phi^{-1}(1-0.200)=\Phi^{-1}(0.800)\approx0.8416.

The combination statistic is

C=(0.6)(1.7507)+(0.8)(0.8416) \approx1.0504+0.6733 =1.7237.

The critical value is $z_{0.95}\approx1.6449$, and $1.7237>1.6449$, so reject $H_0$ at the 0.05 level. The combined p-value is $1-\Phi(1.7237)\approx0.0424$. In context, under the null and the stated combination-test conditions, the probability of a combined statistic at least this large is about 4.24%; this is below 5%.

Worked Example: A Left-Sided Alternative

The inverse normal formula is unchanged for a left-sided alternative, but the stagewise p-values must use the left tail. The next hypothetical example makes the sign change visible.

Worked Example: Combining Left-Tailed P-Values Correctly

Suppose a lower value of an outcome favors treatment, so the prespecified alternative is left-sided. Two independent stages report $p_1=0.080$ and $p_2=0.300$, each calculated as the null probability of a statistic at or below its observed value. Let $w_1=0.8$, $w_2=0.6$, and $\alpha=0.05$.

For illustration, if each stage statistic is standard normal under the null, these p-values correspond to statistics $T_1=\Phi^{-1}(0.080)\approx-1.4051$ and $T_2=\Phi^{-1}(0.300)\approx-0.5244$. The inverse normal combination scores are oriented toward the left-sided alternative:

\Phi^{-1}(1-p_1)=\Phi^{-1}(0.920)\approx1.4051, \qquad \Phi^{-1}(1-p_2)=\Phi^{-1}(0.700)\approx0.5244.

Thus

C=(0.8)(1.4051)+(0.6)(0.5244) \approx1.1241+0.3146 =1.4387.

The statistic is below the one-sided 5% critical value $1.6449$. The combined p-value is $1-\Phi(1.4387)\approx0.0751$, so this test does not reject $H_0$ at $\alpha=0.05$. The positive scores here do not mean the underlying stage statistics were positive: they reflect evidence oriented toward the specified left-sided alternative.

Worked Example: Retaining Precision in the Combination

The calculation can be sensitive to premature rounding, especially when the combined statistic is near a decision threshold. Carry enough digits through the weighted sum, then round the reported result.

Worked Example: Combining a Small and a Moderate P-Value

Consider two independent stagewise p-values $p_1=0.030$ and $p_2=0.400$, both for the same one-sided alternative. Use $w_1=0.6$ and $w_2=0.8$, with $\alpha=0.05$.

The transformed scores are

\Phi^{-1}(0.970)\approx1.880794, \qquad \Phi^{-1}(0.600)\approx0.253347.

Using these values without rounding the products prematurely gives

C=(0.6)(1.880794)+(0.8)(0.253347) \approx1.128476+0.202678 =1.331154 \approx1.331.

The combined p-value is $1-\Phi(1.331154)\approx0.0916$. Since $C<1.6449$, the test does not reject at the 0.05 level. The conclusion is about the combined test, not about whether either stagewise p-value alone passes a separate 0.05 threshold.

Reading the Result and Checking the Calculation

The combined p-value $1-\Phi(C)$ is a right-tail probability because large positive $C$ indicates evidence in favor of the one-sided alternative. It is not the probability that $H_0$ is true, nor is it the probability that the observed result arose “by chance” without specifying the null model and test procedure. The decision rule and p-value are two ways of expressing the same calibrated test: reject at level $\alpha$ exactly when the combined p-value is below $\alpha$, apart from equality at the boundary.

Several checks catch common errors before interpretation:

Confirm that every stagewise p-value is small in the direction specified by the same $H_a$.
Use $\Phi^{-1}(1-p_i)$, not $\Phi^{-1}(p_i)$, for the right-tail normal-score convention in this combination statistic.
Check that the weights are nonnegative and their squares sum to one; otherwise the stated $N(0,1)$ calibration does not follow as written.
Check the assumptions supporting independent, valid stagewise p-values under $H_0$. A familiar formula does not repair invalid p-values or unaccounted dependence.
Keep precision in the transformed scores and products until the weighted sum is complete.

A particularly tempting mistake is to say, “The stage used a left-sided test, so I should change the combination formula to $\Phi^{-1}(p_i)$.” That reverses the convention and can make evidence against the alternative produce a positive contribution. The correction is to compute the stagewise p-value from the left tail, then apply the defined transform $\Phi^{-1}(1-p_i)$. For example, $p=0.080$ gives $\Phi^{-1}(0.920)\approx1.405$, an evidence score in the favorable direction.

How This Test Differs From Other Decisions

The inverse normal combination test uses stagewise p-values as its inputs. It is not the same procedure as pooling all observations and applying a single test to the combined data. A pooled-data analysis may be appropriate for a prespecified design and model, but its statistic and null calibration depend on that model. The combination test instead has its own calibration, based on the joint behavior of the stagewise p-values and the weights.

It is also distinct from requiring every stagewise test to be individually significant. Such a rule can fail to reject even when several stages provide moderate evidence that combines to pass the prespecified threshold. Conversely, one very small stagewise p-value does not guarantee rejection: the combined statistic depends on all stagewise scores and their weights. The weights affect how strongly each stage contributes, but they do not change the direction convention.

The method’s limitations follow from its inputs. It provides a test of the specified null and alternative, not by itself a pooled treatment-effect estimate or confidence interval. Its standard reference distribution depends on correct stagewise calibration and independence. If those conditions are doubtful, or if the p-values refer to different hypotheses or different directions, the standard critical value and combined p-value may be invalid. These are design and calibration problems, not problems solved by changing the inverse normal transform.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For a right-sided stagewise p-value of $0.10$, what inverse normal score enters the combination statistic?
A left-sided test reports $p=0.05$. Explain why the combination score is positive under the stated convention, and give its approximate value.
Two independent stagewise p-values are $0.20$ and $0.50$, with weights $0.6$ and $0.8$. Calculate $C$ and the combined p-value to about three significant figures.
Why is it incorrect to use the standard $N(0,1)$ reference distribution automatically when the stagewise p-values are dependent?
A researcher computes $C=1.50$. At one-sided $\alpha=0.05$, state the decision and explain what the combined p-value represents.

Solutions. (1) Use $\Phi^{-1}(1-0.10)=\Phi^{-1}(0.90)\approx1.282$. (2) The left-tail p-value is first calculated so that small values support the left-sided alternative, then transformed by the same convention: $\Phi^{-1}(1-0.05)=\Phi^{-1}(0.95)\approx1.645$, a positive score. (3) The scores are $\Phi^{-1}(0.80)\approx0.8416$ and $\Phi^{-1}(0.50)=0$. Thus $C=(0.6)(0.8416)+(0.8)(0)\approx0.505$. The combined p-value is $1-\Phi(0.505)\approx0.307$. (4) Dependence changes the variance and potentially the full distribution of the weighted sum; the independent-normal derivation no longer establishes $C\sim N(0,1)$. Calibration must reflect the actual joint distribution. (5) Since $1.50<z_{0.95}\approx1.645$, do not reject $H_0$ at the 0.05 level. The combined p-value $1-\Phi(1.50)\approx0.0668$ is the null probability, under the test’s calibration conditions, of obtaining a combined statistic at least as large as 1.50 in the prespecified direction.