Directional Consistency Across Stages

Why Direction Matters

An inverse normal combination test aggregates evidence from separate stages, but the meaning of that aggregation depends on what each stage contributes. A small two-sided p-value says that a stage’s data are unusual under its null hypothesis; by itself, it does not say whether the estimated treatment effect is beneficial or harmful. If the treatment-effect estimate is positive in one stage and negative in another, discarding those signs can conceal an important disagreement.

This tutorial considers how to preserve direction when combining stagewise evidence. Assume throughout that the treatment effect is defined consistently across stages, with positive values representing the prespecified beneficial direction. A negative estimate then points against that direction. The signed combination statistic summarizes standardized evidence, not a pooled treatment-effect estimate.

Definition: For stage $i$, let $\hat{\theta}_i$ be the estimated treatment effect, with standard error $SE_i$, and let $Z_i=\hat{\theta}_i/SE_i$ be its signed stagewise statistic. If $p_i^{(2)}$ is the corresponding two-sided p-value, the signed normal score is $$ Y_i=\operatorname{sign}(\hat{\theta}_i)\Phi^{-1}(1-p_i^{(2)}/2). $$ When the stagewise statistic is a standard normal score, $Y_i=Z_i$. If $\hat{\theta}_i=0$, set $Y_i=0$. Here $\Phi$ is the standard normal cumulative distribution function.

The sign is not recoverable from a two-sided p-value alone. It must come from the stagewise estimate or signed test statistic, using a consistent treatment-minus-control definition and endpoint coding. If the stagewise test statistic is $Z_i$, then $p_i^{(2)}=2[1-\Phi(|Z_i|)]$, so transforming the p-value gives $|Z_i|$; restoring the sign gives $Z_i$.

Assumptions and the Combination Rule

For two stages, let $w_1$ and $w_2$ be prespecified weights satisfying $w_1^2+w_2^2=1$. The signed inverse normal statistic is

$$ C=w_1Y_1+w_2Y_2. $$

For a two-sided test at level $\alpha$, reject the global null when $|C|>z_{1-\alpha/2}$, where $z_q=\Phi^{-1}(q)$. Equivalently, the combined two-sided p-value is $2[1-\Phi(|C|)]$. For a prespecified positive-direction one-sided test, reject when $C>z_{1-\alpha}$.

The standard normal reference distribution requires the following conditions under the global null: each signed stage score has a standard normal null distribution; the scores from separate stages are independent; and the weights are fixed independently of the observed stagewise scores and normalized so that their squared sum is one. These conditions yield the calibration:

E(C\mid H_0)=0,\qquad \operatorname{Var}(C\mid H_0)=w_1^2+w_2^2=1, \qquad C\mid H_0\sim N(0,1).

The equality in distribution follows because a fixed linear combination of independent normal variables is normal. If a stagewise p-value is only approximately uniform under its null, or the signed score is only approximately standard normal, the resulting calibration is approximate as well. Independence and valid stagewise inference matter: a sign convention cannot correct dependent stage increments or an invalid stagewise p-value.

Why Opposing Signs Can Cancel

The contribution from each stage is its weight multiplied by its signed normal score. A stage pointing against the beneficial direction contributes negatively, so it can reduce the evidence from a stage pointing in favor. With positive weights and $Y_1>0$, $Y_2<0$, the combined score is positive only if $w_1Y_1>w_2|Y_2|$. Even when $C$ remains positive, a conflict between stage estimates may be important scientifically and should not be hidden by the final p-value.

Worked Example: Two Stages Point in Opposite Directions

Suppose a hypothetical trial measures treatment effect in points, with positive values beneficial. In stage 1, $\hat{\theta}_1=4.8$ points and $SE_1=2.0$ points; in stage 2, $\hat{\theta}_2=-3.0$ points and $SE_2=2.5$ points. Use prespecified weights $w_1=0.6$ and $w_2=0.8$.

The signed stage statistics are

Z_1=\frac{4.8}{2.0}=2.4,\qquad Z_2=\frac{-3.0}{2.5}=-1.2.

The two-sided stagewise p-values are approximately $0.0164$ and $0.2301$. Their signed normal scores are therefore $Y_1=2.4$ and $Y_2=-1.2$. The weights are normalized because $0.6^2+0.8^2=1$. Thus,

$$ C=(0.6)(2.4)+(0.8)(-1.2)=1.44-0.96=0.48. $$

The combined two-sided p-value is $2[1-\Phi(0.48)]\approx0.631$. At $\alpha=0.05$, the critical value is approximately $1.960$, and $|0.48|<1.960$, so the test does not reject the global null. The stage estimates point in opposite directions; the signed combination reflects that conflict rather than treating both stages as evidence in the beneficial direction.

This result is not a claim that the treatment has no effect, nor is the combined score an estimate in points. It is a test result under the specified combination rule. The stage estimates, their uncertainty, and the possibility that the treatment effect differs across stages remain relevant to interpretation.

Deriving the Null Calibration

The null reference distribution follows directly from the stagewise assumptions. If $Y_1$ and $Y_2$ are independent $N(0,1)$ variables, then their weighted sum has mean zero and variance equal to the sum of the squared weighted standard deviations:

$$ E(C)=w_1E(Y_1)+w_2E(Y_2)=0, $$

\operatorname{Var}(C) =w_1^2\operatorname{Var}(Y_1)+w_2^2\operatorname{Var}(Y_2) = w_1^2+w_2^2=1.

A linear combination of independent normal variables is normal, so $C\sim N(0,1)$ under the global null. The two-sided rejection rule therefore has null probability

P(|C|>z_{1-\alpha/2}\mid H_0)=\alpha.

The signs are essential to this derivation. Replacing $Y_i$ with $|Y_i|$ changes the distribution: absolute standard normal variables are not standard normal, and their weighted sum does not have the $N(0,1)$ reference distribution. Thus, the standard critical value cannot simply be applied to a weighted sum of unsigned scores.

Worked Example: Agreement Across Stages

Consider hypothetical estimates in the same beneficial direction: $\hat{\theta}_1=3.0$ points with $SE_1=2.0$ points, and $\hat{\theta}_2=5.0$ points with $SE_2=2.5$ points. Use $w_1=0.6$ and $w_2=0.8$.

The stage statistics are $Z_1=3.0/2.0=1.5$ and $Z_2=5.0/2.5=2.0$, so their signed scores are $Y_1=1.5$ and $Y_2=2.0$. The combined statistic is

$$ C=(0.6)(1.5)+(0.8)(2.0)=0.90+1.60=2.50. $$

The combined two-sided p-value is $2[1-\Phi(2.50)]\approx0.0124$. Since $2.50>1.960$, the test rejects the global null at the 0.05 level. Both stage estimates are positive, so the combined evidence and the observed directions are aligned. This does not establish that the effect is identical in both stages; it reports a test of the combined evidence under the stated assumptions.

Signed Scores Versus Other Choices

Approach	What it retains	Important qualification
Signed normal scores	Stagewise evidence and its direction	Uses the standard normal calibration under the stated independence, validity, and weight conditions.
Unsigned scores from two-sided p-values	Magnitude of two-sided evidence	Removes direction; the weighted sum does not have the signed standard normal reference distribution.
Prespecified one-sided stagewise p-values	Evidence in a common prespecified direction	Valid only when the direction is defined in advance and stagewise p-values test that same direction.
Pooled-data analysis	A treatment-effect estimate under its model	It is not interchangeable with a combination test and requires its own model and assumptions.

Combining unsigned normal quantiles can be part of a separately defined two-sided combination procedure, but it requires a reference distribution and rejection rule appropriate to that procedure. It is not equivalent to the signed rule above. Likewise, selecting a one-sided direction after seeing which way the estimates point and then treating the resulting p-value as prespecified does not preserve the stated one-sided error rate.

Worked Example: A Later Stage Can Reverse the Combined Direction

Suppose a hypothetical first stage estimates a beneficial effect of $2.0$ points with standard error $2.0$ points, while the second estimates an effect of $-6.0$ points with standard error $3.0$ points. Let $w_1=0.8$ and $w_2=0.6$, which satisfy $0.8^2+0.6^2=1$.

The stage scores are $Y_1=2.0/2.0=1.0$ and $Y_2=-6.0/3.0=-2.0$. Therefore,

$$ C=(0.8)(1.0)+(0.6)(-2.0)=0.8-1.2=-0.4. $$

The combined statistic points in the negative direction because the weighted negative score is larger in magnitude. The two-sided p-value is $2[1-\Phi(0.4)]\approx0.6892$, so the test does not reject at the 0.05 level. For a prespecified positive-direction test, the statistic also fails the rule $C>1.645$. The negative combined sign describes the direction of the aggregate standardized evidence; it is not itself proof of a harmful effect.

Diagnostics, Pitfalls, and Limits

Before interpreting a direction-aware result, check that the treatment contrast has the same orientation in every stage, that larger endpoint values have the same clinical meaning, and that each stage’s sign comes from its estimate rather than from the p-value alone. Confirm that stagewise tests are valid under their nulls, that the stage scores meet the required independence condition, and that the weights were fixed as assumed. If weights depend on observed evidence or stage two is adapted, the standard derivation above does not by itself establish calibration.

A common mistake is to convert each two-sided p-value to a positive normal quantile and then combine those values using the standard normal critical value, reasoning that a small p-value in either stage must increase evidence for treatment benefit. This both discards the direction of the estimates and uses a reference distribution that need not apply to the unsigned sum. Instead, retain the stage sign and use the signed normal score for the direction-preserving rule, or use a separately calibrated procedure designed to combine unsigned two-sided evidence.

A second limitation is interpretive: a nonsignificant signed combination can result from weak evidence in both stages or from substantial evidence in opposing directions. The final p-value does not distinguish those patterns. Report stagewise estimates and uncertainty alongside the combination result, especially when the scientific claim requires a stable beneficial direction. A significant result also does not establish equal stage effects or eliminate concerns about clinical heterogeneity.

Check Your Understanding

Answer each question, then compare with the solutions.

Under what conditions does the normalized signed inverse normal statistic have a standard normal null distribution?
With $w_1=0.6$, $w_2=0.8$, $Y_1=1.0$, and $Y_2=-1.0$, calculate $C$ and its direction.
Why is a two-sided p-value alone insufficient to determine a signed normal score?
What is wrong with using the usual standard normal critical value for a weighted sum of positive normal quantiles derived from two-sided p-values?

Solutions. (1) The signed stage scores must each be standard normal under the global null, independent, and combined with fixed weights whose squared sum is one. (2) $C=(0.6)(1.0)+(0.8)(-1.0)=0.6-0.8=-0.2$, so the combined score points in the negative direction. (3) A two-sided p-value retains the magnitude of evidence but omits whether the estimate is positive or negative; the sign must come from the estimate or signed statistic. (4) Taking absolute values changes the null distribution: the weighted unsigned sum is not generally standard normal, so its rejection threshold must be calibrated for its own distribution.