What Does a Failed Replication Mean?
A replication result is often summarized as “significant” or “not significant.” That summary is too coarse to diagnose why a replication did not meet its criterion. A nonsignificant result may be imprecise and compatible with the discovery effect; it may be precise enough to rule out effects of that size; or it may point in the opposite direction. Differences in phenotype definition, allele coding, or cohort context can also make two estimates hard to compare.
The central question is not simply whether the replication p-value exceeds a threshold. It is: which effect sizes remain plausible given the replication data, and are the discovery and replication estimates comparable? A failure to reject a no-effect null hypothesis is not proof that the effect is zero. Conversely, a wide interval is not evidence that the discovery effect has been confirmed: it says the replication data do not distinguish among a range of possibilities.
Let \(\hat{\beta}_D\) and \(\hat{\beta}_R\) denote the estimated effects in discovery and replication, with standard errors \(SE_D\) and \(SE_R\). An effect might be a change in a quantitative phenotype per additional copy of a specified allele, measured in phenotype units. For a large-sample Wald analysis, an approximate two-sided 95% confidence interval for the replication effect is
This approximation assumes that the estimate is approximately normally distributed around the target effect, the standard error estimates its sampling uncertainty, and the analysis model is appropriate. For a different model or small-sample setting, the appropriate interval may use a different reference distribution. The interval describes sampling uncertainty under the model; it does not account automatically for phenotype mismatch, bias, or other systematic errors.
Read the Interval Before Drawing a Conclusion
The interval provides more diagnostic information than the significance label alone. If it is wide and contains zero as well as the discovery estimate, the replication has not precisely distinguished no effect from an effect like the one initially observed. If the interval is narrow and excludes effects of the relevant size, the data are more directly inconsistent with that size of effect. If the entire interval is on the opposite side of zero, the estimated direction differs, and the data may provide evidence for an effect in that opposite direction.
The phrase “evidence of no effect” needs care. A conventional test of \(H_0:\beta_R=0\) asks whether the data are sufficiently incompatible with exactly zero under the model. Not rejecting \(H_0\) does not establish that \(\beta_R=0\). To support a claim of a negligible effect, investigators need a scientifically justified range around zero that counts as negligible, and an interval precise enough to lie within that range. The range should be defined on the relevant effect scale, not chosen after seeing the replication estimate.
Worked Example: A Wide Interval After a Nonsignificant Test
Suppose a discovery cohort estimates an allele effect of \(0.20\) phenotype units per allele. In an independent replication cohort, the estimate is \(0.20\) units with \(SE_R=0.12\) units. Assume the same allele, phenotype scale, and adjustment model are used, and that the estimate has an approximately normal sampling distribution.
The replication Wald statistic is
The two-sided p-value is approximately \(2\{1-\Phi(1.667)\}=0.0955\), so the result is not significant at the 0.05 level. Its approximate 95% confidence interval is
This interval includes zero, the discovery estimate of \(0.20\), and effects larger than the discovery estimate. The result is therefore inconclusive about whether an effect of \(0.20\) replicates; it does not show that the effect is absent. The interval is also too broad to support a claim that the effect is negligible without a much broader definition of “negligible.”
Power Explains One Kind of Failure, Not Every Kind
Power is the probability that a specified test rejects its null hypothesis under a specified alternative, assuming the model and design used for the calculation. It depends on the effect size used as the alternative, the standard error, the significance level, and whether the test is one-sided or two-sided. Thus, “the replication was underpowered” is incomplete unless it specifies the effect size and test criterion.
For a two-sided Wald test at level \(\alpha=0.05\), an estimate with known standard error \(SE_R\) has approximate power at effect \(\beta\)
This calculation helps assess whether a nonsignificant result was likely under a particular effect. It does not diagnose a phenotype mismatch or systematic bias, and it does not turn a nonsignificant result into positive evidence for the alternative. Also, power calculated using the observed replication estimate as if it were the true effect is generally not a useful explanation of the result: power should be evaluated at a prespecified scientifically relevant effect, such as the target used in planning.
Worked Example: How Much Power Did the Replication Have?
Continue with the hypothetical replication above: \(SE_R=0.12\) phenotype units, a two-sided test at \(\alpha=0.05\), and a planning target \(\beta=0.20\) units. These assumptions use the normal approximation and treat the standard error as fixed for planning.
The standardized target effect is \(\beta/SE_R=0.20/0.12=1.667\). The probability of a positive-tail rejection is \(P(Z>1.96-1.667)=P(Z>0.293)\), approximately \(0.385\). The probability of a negative-tail rejection is \(P(Z<-1.96-1.667)=P(Z<-3.627)\), approximately \(0.00014\). Therefore, approximate power is \(0.385+0.00014=0.385\), or about 38.5%.
A nonsignificant outcome is not surprising under this design even if the target effect is truly \(0.20\) units. The calculation makes low power a plausible explanation; it does not establish that this is the explanation for the observed result. For scale, an effect would need to be about \(SE_R(1.96+1.282)=0.12(3.242)=0.389\) units for approximately 90% power under the same two-sided normal approximation. That is substantially larger than the planning target.
Direction and Comparability Must Be Checked
A direction disagreement is meaningful only after the estimates have been placed on comparable scales. In genetic association analyses, confirm that both estimates refer to the same effect allele, that the allele is not coded as its complement in one cohort, and that the modeled effect has the same interpretation. For a quantitative phenotype, also check whether the same measurement, transformation, units, and covariate adjustment were used. For a binary phenotype, check that case and control coding and the effect scale (for example, log odds rather than odds ratio) agree.
Even with harmonized coding, cohorts may differ in phenotype measurement, inclusion criteria, environmental context, ancestry composition, or other effect modifiers. A change in phenotype definition can change the target being estimated, rather than merely add noise. A genuine effect difference across populations may also be scientifically important, but it should not be attributed to a biological mechanism from a pair of estimates alone. First establish that the analyses estimate comparable quantities; then describe the observed difference and its uncertainty.
Worked Example: An Opposite-Signed but Nonsignificant Estimate
Suppose discovery reports \(\hat{\beta}_D=0.12\) units per copy of allele A. A replication reports \(\hat{\beta}_R=-0.03\) units and \(SE_R=0.04\) units. Assume allele A is the effect allele in both cohorts, the phenotype and units match, and the large-sample Wald approximation is appropriate.
The replication statistic is \(z=-0.03/0.04=-0.750\), giving a two-sided p-value of approximately \(2\{1-\Phi(0.750)\}=0.453\). The 95% interval is
The point estimate has the opposite sign from discovery, but the interval includes zero and positive effects, including effects close to the discovery estimate. The data do not establish a reversal of direction. If the allele coding or phenotype alignment had not been checked, even the sign comparison would be premature. Here the harmonized result is best described as an imprecise estimate with a negative point estimate, not as evidence that the effect has reversed.
Compare Cohorts Directly to Diagnose Heterogeneity
A common error is to infer a difference because one cohort has a significant result and another does not. Significance depends on both the estimated effect and its standard error; two estimates can be statistically compatible even when only one crosses a significance threshold. Conversely, a direct comparison may reveal a difference even when each cohort estimate is individually imprecise. The appropriate question is whether the estimated effects differ by more than can reasonably be explained by their sampling errors.
For two independent cohorts estimating the same effect on the same scale, define the contrast \(\hat{\Delta}=\hat{\beta}_1-\hat{\beta}_2\). If the estimates are approximately normal and their sampling errors are independent, then
This follows because the variance of a difference of independent random variables is the sum of their variances: \(\operatorname{Var}(\hat{\beta}_1-\hat{\beta}_2)=SE_1^2+SE_2^2\). A Wald interval for the contrast is \(\hat{\Delta}\pm1.96SE(\hat{\Delta})\). If the estimates are correlated, this formula is not valid as written; the covariance must be included. This simple contrast is a diagnostic for a pair of independent estimates, not a full model for several cohorts or for a distribution of effects.
Worked Example: A Direct Contrast Between Two Cohorts
Two independent hypothetical cohorts use the same allele coding, phenotype definition, units, and analysis model. Cohort 1 estimates an effect of \(0.20\) units per allele with \(SE_1=0.05\); cohort 2 estimates \(0.02\) units with \(SE_2=0.05\). Each estimate is approximately normal with its stated standard error.
The estimated difference is \(\hat{\Delta}=0.20-0.02=0.18\) units. Its standard error is
The contrast statistic is \(z=0.18/0.07071=2.546\), with a two-sided p-value of approximately \(0.0109\). Its 95% interval is \(0.18\pm1.96(0.07071)=[0.0414,\ 0.3186]\) units. Under the stated assumptions, this provides evidence that the two cohort-specific estimates differ.
Cohort 2 alone has interval \(0.02\pm1.96(0.05)=[-0.078,\ 0.118]\) units, which includes zero. That fact alone would not establish a difference from cohort 1. The direct contrast does. Still, it does not identify why the estimates differ: chance, unmeasured design differences, phenotype or population differences, or genuine effect modification remain possible explanations to investigate.
A Practical Diagnostic Sequence
Check the prespecified variant, effect allele, phenotype, effect scale, and test direction. If these differ from discovery, state precisely what was replicated instead.
Report \(\hat{\beta}_R\), \(SE_R\), and a confidence interval in context and units. Note whether the interval includes zero, the discovery estimate, or effects judged scientifically important.
Use the prespecified effect size and the actual test criterion. Low power can explain why a meaningful effect failed to reach significance, but it cannot establish that such an effect is present.
For two independent, comparable cohorts, examine the difference and its uncertainty. Avoid attributing a cause to a difference until measurement and cohort features have been investigated.
Common Pitfalls and Limits
- “The result was nonsignificant, so there is no effect.” A p-value above the cutoff means the test did not reject its null at that cutoff. Inspect the interval and its precision before making a claim about the effect.
- “One cohort is significant and the other is not, so they differ.” This compares two significance decisions, not the estimates. Test or interval the contrast directly when cohorts are independent and the estimands are comparable.
- “The replication estimate has the opposite sign, so the effect reversed.” A noisy estimate can have the opposite sign by chance. Check allele harmonization and the interval; direction evidence is weak when the interval spans both signs.
- “Low power explains the failure.” Low power at a chosen target makes a nonsignificant result plausible, but is not proof of the target effect. Bias, phenotype mismatch, or genuine cohort differences may also matter.
- “A difference proves a biological interaction.” A cohort contrast identifies a statistical difference under its assumptions. It does not isolate the cause or establish a mechanism.
These diagnostics have limits. A confidence interval can be narrow and still misleading if the standard error or model is wrong. Cohorts that appear to measure the same phenotype may differ in ascertainment or measurement. Pairwise contrasts do not describe all sources of variation when many cohorts are involved. Conclusions should therefore report the effect estimates, uncertainty, comparability checks, and remaining uncertainty—not only a replication label.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A replication estimate is \(0.10\) units with \(SE=0.09\). Its approximate 95% interval is \([-0.0764,\ 0.2764]\). Does a nonsignificant test establish no effect?
- Why is “discovery is significant, replication is not” insufficient to conclude that the cohort effects differ?
- A replication point estimate is negative, but its confidence interval includes zero and positive values. What is the defensible interpretation of direction?
- Name two checks needed before comparing genetic effect directions across cohorts.
Solutions. (1) No. The interval includes zero and a broad range of positive effects, so the result is inconclusive about effects in that range. (2) Significance depends on both estimate and standard error; compare the estimates directly and quantify uncertainty in their difference. (3) The point estimate is negative, but the data do not establish a negative effect or a reversal; the interval remains compatible with either direction. (4) At minimum, verify the effect-allele coding and that phenotype definition, scale, and units have the same interpretation.