Two-Sided Inverse Normal Combination Tests

Why a Two-Sided Combination Needs Its Own Calibration

When trial evidence is accumulated across stages, the stagewise result may be reported as a two-sided p-value. Such a value records how surprising the data are in either direction, but it does not retain the direction of the estimated effect. A combination test built from these values therefore answers a different question from one that combines signed evidence: it can assess whether the stages jointly show strong evidence in either direction, without requiring their effects to point the same way.

The inverse normal transformation is useful here, but a two-sided p-value cannot simply be treated like the one-sided p-value used in the standard upper-tail inverse normal test. Under a continuous null, transforming a two-sided p-value into a nonnegative normal score produces the absolute value of a standard normal variable, not a standard normal variable. As a result, a weighted sum of those scores does not have the usual standard normal reference distribution. The test must be calibrated to the distribution it actually has.

Definition: For stage $i$, let $p_i$ be a valid two-sided p-value. Its nonnegative inverse normal score is $$ R_i=\Phi^{-1}(1-p_i/2), $$ where $\Phi$ is the standard normal cumulative distribution function. Given fixed weights $w_i\geq 0$ satisfying $\sum_i w_i^2=1$, define the two-sided inverse normal combination statistic $$ C=\sum_i w_iR_i. $$ The test rejects for large $C$, using a critical value calibrated under the joint null distribution of the $R_i$, rather than automatically using a standard normal critical value.

Null Distribution and Calibration

Suppose first that each stagewise two-sided p-value is exactly uniform under the null, and that the stagewise p-values are independent. For a standard normal variable $Z_i$, the two-sided p-value $2\{1-\Phi(|Z_i|)\}$ is uniform on $(0,1)$. Inverting this relation gives $R_i=\Phi^{-1}(1-p_i/2)$, which has the same null distribution as $|Z_i|$. Thus each $R_i$ follows a half-normal distribution: it is nonnegative and has the distribution of the absolute value of a standard normal variable.

This distributional fact explains why the usual inverse normal calibration does not apply. In the one-sided construction with exact uniform p-values, the transformed scores are standard normal and a normalized weighted sum is standard normal. Here the component scores are nonnegative half-normal variables. Their weighted sum is also nonnegative, so it cannot have a standard normal distribution, which assigns positive probability to negative values.

Calibration rule: Under independent, exactly uniform stagewise two-sided p-values and fixed nonnegative weights with squared sum one, choose $c_\alpha$ so that $$ P_0\left(\sum_i w_i|Z_i|>c_\alpha\right)=\alpha, $$ where the $Z_i$ are independent standard normal variables. Reject $H_0$ when $C>c_\alpha$. Equivalently, report the calibrated combination p-value $P_0(\sum_i w_i|Z_i|\geq C_{\mathrm{obs}})$.

The null probability in this definition can be evaluated by numerical integration or approximated by simulation from independent standard normal variables. With more than one positive weight, there is generally no standard normal shortcut for the critical value. Because the distribution is continuous in the usual case of at least one positive weight, the calibrated upper-tail test has null rejection probability $\alpha$.

Why the Calibration Is Correct

The justification has two parts: identify the component distribution, then use that joint distribution to set the rejection boundary. This is a distributional calibration, not an assumption that a sum of transformed two-sided p-values is normal.

Invert the two-sided p-value.
For $Z_i\sim N(0,1)$, $p_i=2\{1-\Phi(|Z_i|)\}$. Therefore $1-p_i/2=\Phi(|Z_i|)$, and applying $\Phi^{-1}$ gives $R_i=|Z_i|$.

Use stage independence.
If the stagewise p-values are independent under $H_0$, then the corresponding $R_i$ are independent. The null distribution of $C$ is therefore the distribution of the specified weighted sum of independent absolute standard normal variables.

Set the upper-tail cutoff.
Choose $c_\alpha$ so that the probability that this null weighted sum exceeds $c_\alpha$ is $\alpha$. The rejection event then has null probability $\alpha$, by the definition of the cutoff.

For valid but conservative p-values, exact uniformity may not hold. Validity means $P_0(p_i\leq u)\leq u$ for every $u\in[0,1]$. Since $R_i=\Phi^{-1}(1-p_i/2)$ decreases as $p_i$ increases, a conservative p-value produces a score that is stochastically no larger than the half-normal score from an exact uniform p-value. If the stagewise p-values are independent and all weights are nonnegative, the weighted sum is likewise stochastically no larger. The exact-uniform calibration therefore controls type I error at no more than $\alpha$. Nonnegative weights matter: a negative weight reverses the ordering of its contribution and does not support this conservativeness argument.

Worked Example: A Two-Stage Test With Equal Weights

Consider a hypothetical two-stage trial with independent, continuous stagewise two-sided p-values. The prespecified weights are $w_1=w_2=1/\sqrt{2}$, so their squared sum is one. Suppose the observed p-values are $p_1=0.0400$ and $p_2=0.1000$, and the test uses a two-sided combination procedure at $\alpha=0.05$.

Convert each stagewise p-value to its nonnegative score: $$ R_1=\Phi^{-1}(1-0.0400/2)=\Phi^{-1}(0.9800)\approx 2.054, \qquad R_2=\Phi^{-1}(1-0.1000/2)=\Phi^{-1}(0.9500)\approx 1.645. $$

The observed combination statistic is $$ C_{\mathrm{obs}}=\frac{R_1+R_2}{\sqrt{2}} =\frac{2.054+1.645}{1.4142} \approx 2.616. $$

The 5% upper critical value is obtained from the distribution of $(|Z_1|+|Z_2|)/\sqrt{2}$, not from a standard normal distribution. Numerical evaluation gives $c_{0.05}\approx 2.234$. Since $2.616>2.234$, this calibrated procedure rejects $H_0$ at the 5% level. The conclusion is that the stages jointly provide sufficiently strong two-sided evidence under this prespecified combination rule; the statistic itself does not indicate which direction the treatment effect takes.

Single-Stage Check and the Role of the Weights

A useful check is the one-stage case. With $w_1=1$, the statistic is $R_1=|Z_1|$ under an exact continuous null. Its level-$\alpha$ cutoff is the usual two-sided normal cutoff $z_{1-\alpha/2}=\Phi^{-1}(1-\alpha/2)$, because $P(|Z_1|>z_{1-\alpha/2})=\alpha$. Thus the construction reduces to the familiar two-sided test when there is only one stage.

For multiple stages, the critical value depends on the full weight vector. Larger weights give more influence to their corresponding stage scores. The constraint $\sum_i w_i^2=1$ is a useful normalization, but it does not make the sum of half-normal scores standard normal. The weights should be fixed by the design before observing the stagewise p-values; changing them in response to interim results requires a calibration appropriate to that adaptation.

Worked Example: Checking the One-Stage Boundary

Suppose a hypothetical single-stage test has a continuous standard normal test statistic under $H_0$ and uses $\alpha=0.05$. Its two-sided p-value is $p=0.0300$. The score is $$ R=\Phi^{-1}(1-0.0300/2)=\Phi^{-1}(0.9850)\approx 2.170. $$

For one stage, $C=R$, and $c_{0.05}=\Phi^{-1}(0.9750)\approx1.960$. Since $2.170>1.960$, the test rejects. This agrees with the ordinary two-sided p-value rule because $0.0300<0.05$. The agreement is a useful implementation check; applying the same $1.960$ cutoff to a multi-stage sum of absolute normal scores would not, in general, preserve the nominal level.

Magnitude-Only Evidence Versus Signed Evidence

A two-sided stagewise p-value is unchanged if the sign of a stage estimate is reversed. Consequently, the score $R_i$ contains evidence about the magnitude of departure from the null, but not its direction. The resulting combination statistic can be large when several stages have small p-values even if their estimated effects point in opposite directions.

This differs from combining signed stage evidence. For example, when each stage supplies a signed standard normal statistic $Z_i$, a weighted sum $\sum_i w_iZ_i$ can retain directional information; under independent standard normal null statistics and squared-sum-one weights, that sum is standard normal. A one-direction test then uses an upper tail, while a two-direction test based on the signed sum uses both tails. These are procedures based on signed statistics, not the nonnegative scores defined here. Their behavior can differ materially when stage estimates disagree in sign.

Worked Example: Opposing Signs Are Hidden by Two-Sided Scores

In a hypothetical two-stage trial, suppose the signed stage statistics are $Z_1=2.400$ and $Z_2=-2.000$, with equal weights $1/\sqrt{2}$. For illustration, assume independent standard normal stage statistics under the null. The two-sided stage p-values are approximately $0.0164$ and $0.0455$, respectively, and their nonnegative inverse normal scores are approximately $2.400$ and $2.000$.

The two-sided-score combination is $$ C=\frac{2.400+2.000}{\sqrt{2}}\approx 3.111. $$ Using the two-stage 5% cutoff $2.234$, this magnitude-only procedure rejects. It treats both stages as strong evidence, regardless of their signs.

By contrast, the signed weighted sum is $$ \frac{2.400+(-2.000)}{\sqrt{2}}\approx 0.283. $$ A two-sided test based on that signed sum would not reject at the 5% level. The difference is not a contradiction: the two procedures answer different questions. The first combines evidence of departure in either direction at each stage; the second allows opposing signed evidence to offset.

Diagnostics, Pitfalls, and Limitations

Before applying this procedure, check that the p-values correspond to the same null hypothesis and that their two-sided meaning is appropriate for the scientific question. Confirm that the stagewise values are valid under the null and have the independence required by the calibration. Verify that the weights are prespecified, nonnegative, and normalized as intended. Finally, ensure that the cutoff or combination p-value was computed from the half-normal-score distribution for the actual weights.

Using a standard normal cutoff automatically. A normalized sum of independent standard normal scores may be standard normal; the sum of nonnegative half-normal scores is not. Calibrate to the latter distribution.
Assuming the combined result reveals direction. Two-sided p-values discard the sign. A significant combination does not by itself establish whether the treatment effect is beneficial or harmful.
Allowing stage directions to cancel without noticing. The magnitude-only procedure does not cancel opposite signs. If directional coherence is required by the scientific objective, use a procedure designed for signed evidence and interpret it according to its own assumptions.
Using negative weights with a conservativeness claim. Negative weights can turn a smaller score into a smaller contribution, reversing the stochastic ordering argument. The conservative-p-value guarantee stated here requires nonnegative weights.
Ignoring dependence or adaptive weight choice. Independence and fixed weights are part of the stated calibration. If either fails, the reference distribution may be wrong and type I error control may not hold without a suitable alternative calibration.

The method is therefore valuable when stagewise evidence is intentionally treated as nondirectional and the stagewise p-values can be combined under a valid joint null model. Its principal limitation is the loss of directional information. It also requires a calibrated reference distribution that depends on the weights, rather than the convenient standard normal cutoff from the one-sided inverse normal construction.

Check Your Understanding

Answer each question, then compare with the concise solutions below.

Under an exact continuous null, what is the distribution of $R_i=\Phi^{-1}(1-p_i/2)$ when $p_i$ is a two-sided p-value?
Why is it generally incorrect to compare a multi-stage statistic $C=\sum_i w_iR_i$ with a standard normal critical value?
For one stage at $\alpha=0.05$, what is the critical value for $R_1$, and why does it match the ordinary two-sided test?
What assumptions allow conservative independent stagewise p-values to yield a conservative upper-tail combination test?
Two stages have large absolute evidence but opposite estimated directions. Which feature of the two-sided-score combination determines whether the evidence cancels?

Solutions. (1) $R_i$ has the distribution of $|Z_i|$ for $Z_i\sim N(0,1)$, a half-normal distribution. (2) The component scores are nonnegative half-normal variables, not standard normal variables, so their weighted sum has a different null distribution. (3) The cutoff is $\Phi^{-1}(0.975)\approx1.960$, since $P(|Z_1|>1.960)=0.05$. (4) The p-values must be independent and valid, the weights fixed and nonnegative, and the reference calibration based on exact-uniform null p-values; conservativeness then follows from the decreasing p-to-score transform and stochastic ordering. (5) Their signs are discarded, so the nonnegative scores add rather than cancel; a signed combination is a distinct procedure.