Directional Versus Two-Sided Replication Tests

Why the Direction of the Test Matters

A replication study asks whether an association identified in a discovery analysis is supported by new data. Before examining the replication results, the investigators should specify the variant, outcome, effect scale, and what result will count as evidence. The direction of the discovery estimate can also determine the replication alternative: if discovery estimated a positive effect, a directional test asks whether the replication effect is positive; a two-sided test asks whether it differs from zero in either direction.

These choices change the hypotheses and the rejection threshold. At a fixed significance level, a one-sided test places its full type-I error probability in one tail. A two-sided test divides that probability between both tails. A directional test can therefore have greater power for effects in its prespecified direction, but it does not treat a significant effect in the opposite direction as replication in the specified direction.

The effect must be oriented consistently across cohorts. For example, the same allele must be treated as the effect allele, and the effect must refer to the same phenotype and scale. Otherwise, a sign difference might reflect coding or harmonization rather than a biological reversal. The hypotheses and direction should be set before seeing the replication estimate; choosing whichever direction looks favorable afterward changes the test.

Test Statistics, P-Values, and Thresholds

For a sufficiently large-sample association analysis, suppose the replication estimate is approximately normal with standard error \(SE(\hat{\beta}_R)\). Under the null boundary \(\beta_R=0\), the signed test statistic is

Write \(\Phi(z)=P(Z\leq z)\) for the standard normal cumulative distribution function. If the prespecified direction is positive, the one-sided p-value is \(1-\Phi(Z)\); if it is negative, the p-value is \(\Phi(Z)\). The two-sided p-value is \(2[1-\Phi(|Z|)]\). For a directional test, evidence against the composite null is evaluated at its boundary, \(\beta_R=0\): values farther into the prespecified direction are less compatible with the null.

At significance level \(\alpha\), a positive-direction test rejects when \(Z>z_{1-\alpha}\), where \(\Phi(z_{1-\alpha})=1-\alpha\). A two-sided test rejects when \(|Z|>z_{1-\alpha/2}\). At \(\alpha=0.05\), these critical values are approximately 1.645 and 1.960, respectively. The two-sided test requires a more extreme statistic in either direction, because it allocates 0.025 to each tail rather than 0.05 to the prespecified tail.

For a statistic in the prespecified direction, the two-sided p-value is twice the directional p-value. This relationship does not mean that a directional p-value can always be halved: if the estimate points the other way, its prespecified-direction p-value is large, while the two-sided p-value may be small. A two-sided result describes evidence of a nonzero effect, not necessarily evidence that the effect agrees with discovery.

What a Reversed Replication Estimate Means

A reversed estimate points opposite to the direction specified from discovery. The directional test does not count this as support for its alternative. The two-sided test, in contrast, can reject the null for a sufficiently large effect in either direction. If a two-sided test is significant and the estimated sign is reversed, the result is evidence of a nonzero association in the replication cohort, but it is not same-direction replication.

This distinction is especially important when the scientific claim being checked is directional. If the prespecified replication criterion requires the same direction and statistical significance, a reversed estimate fails that criterion, even if its two-sided p-value is small. A two-sided test followed by a same-sign requirement is a different rule from a one-sided test at the same \(\alpha\): in the expected direction, it uses the more stringent two-sided critical value. Conversely, reporting a two-sided test can be useful when effects in either direction are scientifically relevant or when a reversal itself warrants attention.

Why the Threshold Difference Arises

Under the null boundary, \(Z\) has a standard normal distribution. A one-sided positive test with \(\alpha=0.05\) chooses a cutoff \(c\) such that the probability of a null statistic exceeding it is 0.05. A two-sided test chooses a cutoff \(d\) such that the combined probabilities of exceeding \(d\) or falling below \(-d\) total 0.05. By symmetry, each tail then has probability 0.025.

Thus the directional test gains sensitivity in its specified tail by not using that tail’s probability to detect an effect in the opposite direction. The gain is appropriate only when the direction was justified and fixed in advance. It is not a license to inspect the replication estimate, choose the matching one-sided alternative, and then use the one-sided threshold.

Assumptions, Diagnostics, and Choice of Test

The calculation is only as defensible as its design and model. The standard-normal test assumes that the estimate and standard error produce an adequate normal approximation under the null. The reported standard error must reflect the fitted association model and sampling design. If the analysis instead uses a test with a \(t\) reference distribution, use that test’s specified distribution and critical values rather than substituting standard-normal cutoffs.

For a confirmatory directional test, the direction must be prespecified, the tested variant and outcome must be fixed, and the replication data should not have been used to select the direction. Independent replication data make this separation particularly clear. A nominal threshold also does not automatically account for testing many variants or outcomes; the threshold must follow the prespecified analysis plan, including any applicable multiplicity adjustment. Do not assume that \(\alpha=0.05\) is suitable for every study.

Useful checks include verifying allele alignment and effect coding, confirming the outcome and effect scale, checking that the estimate and standard error come from the same model, and comparing the sign of \(\hat{\beta}_R\) with the prespecified direction. A reported directional p-value near one can be a correct result when the estimate points strongly the wrong way; it is not evidence for that opposite direction. Calculate or report the two-sided p-value separately if evidence in either direction is relevant.

Neither test removes the effects of selection in discovery, limited replication precision, or genuine differences between cohorts. The earlier material on power calculations for selected loci concerns the probability of rejection under specified effects and assumptions; the choice between one- and two-sided alternatives changes that probability and must be reflected in the calculation. A p-value does not measure the probability that the null hypothesis is true, and failure to reject does not establish that the effect is zero.

Common Mistake and Practical Decision

Choose a one-sided test when the scientific replication claim is genuinely directional, the direction was fixed before examining replication data, and an effect in the opposite direction would not count as success. Choose a two-sided test when departures in either direction are scientifically relevant or when the planned question is simply whether the replication effect differs from zero. In either case, state the estimate with units, its sign, the test used, the significance threshold, and the p-value. If a sign reversal is observed, describe it as such rather than allowing a two-sided p-value to conceal the direction.