Replication Criteria Based on Effect-Size Thresholds

Why Replication May Require More Than a Nonzero Effect

A replication result can be statistically distinguishable from zero without being large enough to matter for the scientific claim. For example, a small estimated genetic effect may have a small standard error in a large cohort and produce a small p-value, even though the effect is below a magnitude considered important in advance. A replication criterion can therefore ask a more demanding question: does the effect exceed a prespecified minimum magnitude?

The effect threshold must be defined on the scale of the replication estimand. An odds ratio is often analyzed on the log-odds scale, for example, so a threshold stated as an odds ratio must be converted before comparison with a log-odds estimate. The threshold should also be chosen before examining the replication estimate; choosing it afterward can make the criterion follow the observed result rather than the scientific question.

Here, \(s=1\) represents a positive direction and \(s=-1\) a negative direction. Orienting the effect this way lets one formula cover both cases. For instance, if the replication claim is a negative association, \(\psi=-\theta\); a larger positive value of \(\psi\) then means a stronger effect in the prespecified negative direction. The estimate and its standard error are oriented in the same way: \(\hat{\psi}=s\hat{\theta}\) and \(SE(\hat{\psi})=SE(\hat{\theta})\).

Threshold Hypotheses and Confidence Bounds

Suppose the replication estimate is approximately normal with standard error \(SE\), so that \((\hat{\psi}-\psi)/SE\) is approximately standard normal. This approximation is commonly justified by large-sample estimation. The cohort and outcome must match the prespecified replication estimand, the effect scale and direction must be harmonized, and the observations must support the standard-error calculation. The approximation may be unreliable with sparse data, strong model misspecification, or an estimator near a boundary; in those cases, use a suitably calibrated interval and test rather than relying automatically on a Wald calculation.

For a one-sided claim that the effect exceeds \(\delta\), the hypotheses are

This is a test against a boundary, not the usual test \(H_0:\psi=0\). The boundary value \(\delta\) is the least favorable null value: under the normal model, the probability of an estimate as large as the observed one is greatest among null values at \(\psi=\delta\). Thus the one-sided test statistic and p-value are

where \(\Phi\) is the standard normal cumulative distribution function. At significance level \(\alpha\), reject \(H_0\) when \(Z_\delta>z_{1-\alpha}\), equivalently when the one-sided lower confidence bound exceeds \(\delta\):

The correspondence follows directly from rearranging the rejection condition. The test rejects when \((\hat{\psi}-\delta)/SE>z_{1-\alpha}\). Multiplying by the positive \(SE\) and rearranging gives \(\hat{\psi}-z_{1-\alpha}SE>\delta\), which is exactly \(L>\delta\). Its level is controlled because, when \(\psi\leq\delta\), rejection requires the lower bound to exceed the true value. Under the normal model, that event has probability at most \(\alpha\), with the largest probability at the boundary \(\psi=\delta\).

For a negative-direction claim with \(\theta<0\), apply the same reasoning to \(\psi=-\theta\). Equivalently, test whether the upper confidence bound for \(\theta\) is below \(-\delta\). State the direction and scale explicitly so that a reader can see which side of the threshold is being tested.

Directional and Absolute-Magnitude Criteria

A directional criterion should be used when the effect direction was specified in advance. If the scientific claim is that an effect in one particular direction exceeds a minimum size, test that oriented effect against \(\delta\). If the claim instead concerns either direction, the question is whether \(|\theta|>\delta\). Its null is that the effect lies within the threshold band:

A confidence-interval approach rejects this null when a two-sided \(100(1-\alpha)\%\) confidence interval lies wholly above \(\delta\) or wholly below \(-\delta\). This is a valid, potentially conservative criterion: if the true value is in the null band, an interval wholly outside the band fails to cover the true value, an event with probability at most \(\alpha\). The direction of the effect is not chosen after seeing which side of the estimate looks more favorable.

This question differs from conventional equivalence testing. Equivalence testing asks whether the effect is small enough to lie within a prespecified interval, often \((-\delta,\delta)\); its alternative is that the effect is inside the band. A minimum-effect replication criterion asks whether the effect is beyond the band. Both use prespecified boundaries, but their null and alternative hypotheses point in opposite directions.

Choosing and Reporting the Criterion

The threshold has to be justified in the scientific context, not selected because it makes the replication succeed. It may be expressed as a minimum odds ratio, a change in a quantitative trait per allele, or another meaningful effect scale. Convert it to the scale on which the model estimates the effect, and retain the units in the interpretation. For example, a log odds ratio is dimensionless, whereas a mean difference in expression may be reported in specified expression units.

A report should identify the estimand, the direction or absolute-magnitude rule, the threshold and its scale, the estimate and standard error or interval, and the threshold-based p-value when one is used. A test against zero may be reported as additional information, but it must not substitute for the threshold test. If multiple variants, outcomes, or thresholds are evaluated, the stated error rate may not apply to the selection process as a whole; the analysis plan must address that multiplicity rather than treating every result as a single prespecified test.

Diagnostics, Limitations, and Common Mistakes

Check that the threshold, estimand, direction, and model scale agree. A mismatch—such as comparing a log-odds estimate with an odds-ratio threshold without transforming the threshold—can reverse the substantive meaning of the comparison. Inspect whether the estimate and standard error are credible under the fitted model. Sparse genotype counts, separation in logistic regression, influential observations, or poor outcome-model fit can undermine a Wald interval; use a method appropriate to the data and verify its calibration.

The confidence-bound criterion quantifies sampling uncertainty under its model and design assumptions. It does not correct for a threshold chosen after examining the data, nor does it by itself address a mismatch between discovery and replication populations. A replication conclusion is specific to the estimand and population defined by the analysis; an interval crossing the threshold indicates uncertainty, not automatic failure of the biological hypothesis.

• Mistake: “The p-value against zero is small, so the replication exceeds the meaningful threshold.” Correction: test against \(\delta\), using \(Z_\delta=(\hat{\psi}-\delta)/SE\), or check whether the appropriate lower bound exceeds \(\delta\).
• Mistake: “The estimate is larger than \(\delta\), so the criterion is met.” Correction: the estimate alone ignores uncertainty. The confidence bound must clear the threshold under the chosen procedure.
• Mistake: “The interval crosses \(\delta\), so the true effect is below \(\delta\).” Correction: crossing the boundary means the data do not establish which side contains the effect with the stated confidence; it does not prove the effect is below the threshold.
• Mistake: “A result in either direction can use the prespecified directional test.” Correction: an either-direction magnitude claim is a different hypothesis. Use an absolute-magnitude procedure that accounts for both sides of the threshold.