Preserving Hypothesis Direction When Combining Evidence

Why Direction Must Be Preserved

A combination test brings evidence from multiple stages into one final decision. But a small p-value only has its intended meaning when each stage’s test is aligned with the same scientific alternative. If one stage’s statistic is coded so that large values indicate benefit and another’s so that large values indicate harm, combining their p-values as though they point in the same direction can distort the interpretation. The numerical combination may still be computable; its clinical meaning may not be.

The issue often arises when stages use different outcomes or test statistics. For example, a positive treatment-minus-control difference might represent benefit for a symptom score where higher is better, but harm for an event rate where lower is better. The solution is not to force the raw estimates to use the same sign. It is to orient each stage’s evidence consistently toward the prespecified alternative before calculating its one-sided p-value.

Here the common alternative is clinical benefit. For clarity, suppose each stage has an estimate \(\hat{\theta}_j\) and standard error \(SE_j\). Define \(\hat{\theta}_j\) to be treatment minus control on the recorded outcome scale. Let \(a_j=+1\) when a larger treatment-minus-control value means greater benefit and \(a_j=-1\) when a smaller value means greater benefit. An oriented standardized statistic is then \(T_j=a_j\hat{\theta}_j/SE_j\). This definition assumes the estimate and standard error are on compatible scales and that the stagewise test uses this standardized statistic, or a valid corresponding null distribution.

The displayed normal-tail formula applies when the standardized statistic has a standard normal null distribution. With a different valid stagewise test, use its own null distribution and the tail corresponding to the common alternative. The scientific direction must be set from the endpoint’s meaning and the trial hypothesis, not chosen after seeing which sign produces a smaller p-value.

Orienting Contrasts Before Testing

The direction convention must be explicit for every endpoint. If the outcome is a favorable response rate, higher values may represent benefit, so a positive treatment-minus-control difference is favorable. If the outcome is a symptom burden, event rate, or time to an adverse event, lower values may be favorable, so a negative difference may represent benefit. A clear analysis plan records this mapping before the stagewise results are examined.

It is not necessary for every raw contrast to be positive under benefit. It is necessary that every statistic supplied to a directional combination procedure be interpreted consistently. For estimates on different scales, orienting the standardized statistic can make the direction comparable without pretending that the estimates themselves have common units. The scale and meaning of each endpoint should still be reported.

How a Sign Error Changes the Combination

Suppose two stages each provide a valid one-sided p-value for the same benefit alternative. An inverse-normal combination uses transformed stagewise p-values, while Fisher’s method uses their logarithms. In either case, the inputs must have the intended directional meaning. For the inverse-normal form used here, prespecified weights satisfy \(\sum_j w_j^2=1\), and the combined statistic is \(\sum_j w_j\Phi^{-1}(1-p_j)\). The combined upper-tail p-value is calculated from the corresponding reference distribution under the procedure’s calibration assumptions.

The next example assumes independent stages, valid stagewise p-values, and prespecified equal weights. These assumptions are stated to make the calculation interpretable; they are not a substitute for checking whether the chosen combination procedure is calibrated for the actual design.

Justification: Reversing the Tail Reverses the Meaning

For a standard normal statistic \(Z\), the upper-tail p-value for a positive-direction alternative is \(p=1-\Phi(Z)\). If the statistic is reversed to \(-Z\) but the same upper tail is used, the resulting value is

The identity uses the symmetry \(\Phi(-z)=1-\Phi(z)\) of the standard normal distribution. It shows why a direction reversal can turn a small p-value into a value near one. A combination procedure that transforms p-values to normal quantiles then receives a contribution with the opposite sign: \(\Phi^{-1}(1-p_{\mathrm{reversed}})=-Z\). For an asymmetric null distribution, the corresponding tail must be determined from that distribution rather than inferred from this normal identity, but the underlying requirement is unchanged: the tail must match the stated alternative.

A two-sided p-value answers a different question. It measures evidence against a null in either direction; by itself, it does not say that the result favors benefit. Taking a two-sided p-value and treating it as a directional input without a prespecified, valid conversion can therefore undermine the intended interpretation.

Diagnostics and Common Failure Modes

Before combining results, verify the direction at the level of the endpoint, the contrast, the test statistic, and the p-value. A useful audit records, for every stage, which group is subtracted from which, which outcome direction is clinically favorable, whether a positive statistic represents that direction, and which tail defines the p-value. These checks are especially important when different stages use different endpoints or analysis methods.

• Contrast order changes. Treatment-minus-control and control-minus-treatment estimates have opposite signs. Record the subtraction order rather than relying on a column label such as “difference.”
• Outcome direction changes. A higher response rate and a lower adverse-event rate may both represent benefit. State the clinical interpretation instead of assuming that positive always means better.
• Tail and statistic disagree. A p-value computed from the wrong tail may be close to one when the observed result favors benefit, or small when it favors harm. Recalculate from the stated alternative.
• Absolute values erase direction. Using \(|T_j|\) in a one-sided analysis can turn evidence against benefit into apparent evidence for benefit. Two-sided testing is a separate inferential choice.
• Labels are not enough. A p-value column called “benefit p-value” does not verify the code that produced it. Check the formula and the sign convention.

A practical diagnostic is to test the pipeline with a simple hypothetical input: enter a clearly favorable contrast and confirm that it produces a positive oriented statistic and a small directional p-value; then enter an equally large unfavorable contrast and confirm that it produces a negative oriented statistic and a large p-value. This is a coding check, not a replacement for statistical calibration.

Choosing and Reporting the Directional Input

Fisher and inverse-normal procedures use different transformations of the p-values, but neither transformation repairs a stagewise direction error. Fisher’s statistic \(-2\sum_j\ln(p_j)\) rewards small inputs; if a stage supplies a two-sided p-value or a small p-value for the wrong clinical direction, Fisher’s combination does not automatically restore the intended benefit interpretation. The inverse-normal procedure likewise treats a small p-value as evidence in the direction encoded by its one-sided tail.

The choice between combination procedures concerns how stagewise evidence is combined and calibrated, not how clinical direction is defined. Whichever procedure is used, report the common alternative, each stage’s contrast convention, the orientation rule, and the source of each one-sided p-value. Where the stages use distinct endpoints, also report each endpoint’s units and explain why its sign indicates benefit or harm.

Directional alignment has limits. It does not make unlike endpoints clinically interchangeable, remove bias, establish that stagewise p-values are valid, or guarantee that the combination procedure’s assumptions hold. A sign convention can also be ambiguous for outcomes where neither a simple increase nor decrease is uniformly favorable. Such cases require an explicitly defined estimand or decision rule before stagewise evidence is combined.