What Does “Failed Replication” Mean?
A replication is often called a “failure” when its result does not meet a prespecified replication criterion. That label alone does not say why the criterion was missed. A replication estimate may be too imprecise to distinguish a meaningful effect from no effect; it may be precise but substantially smaller than the original estimate; or it may point reliably in the opposite direction. These situations support different conclusions.
Interpretation begins by asking whether the studies estimate the same quantity. For example, an effect measured as a change in log-base-2 expression per treatment contrast cannot be compared directly with an effect measured in a different unit or for a different contrast without a justified conversion. The earlier material on cross-platform measurement comparability, frozen versus refit analyses, and replication effect estimates provides useful context: here the focus is how to reason from the pattern of evidence when a replication criterion is not met.
The distinction depends on the claim being assessed. A replication can challenge the original effect’s reported magnitude without showing that the effect is zero. Conversely, a confidence interval that includes zero does not by itself show that the original finding is wrong: the interval may also include effects close to the original estimate.
Read the Estimate, Its Uncertainty, and Its Direction Together
Let \(\hat{\beta}_O\) denote an estimated effect in the original cohort and \(\hat{\beta}_R\) the corresponding replication estimate, both on a common scale. Let \(SE_R\) be the replication estimate’s standard error. When a large-sample normal approximation is reasonable, a two-sided 95% confidence interval for the replication effect is
The interval gives a range of effect values reasonably compatible with the data under the model and sampling procedure. It does not give the probability that a fixed parameter is in this particular interval. Its width is a direct diagnostic of precision: all else equal, a larger standard error produces a wider interval.
Interpret the interval against the replication question, not in isolation. Does it include zero? Does it include the original estimate? Does it include the range of effects considered scientifically meaningful in the locked protocol? These are different comparisons. If the protocol specified a minimum meaningful effect \(\delta>0\), a precise interval lying wholly below \(\delta\) can challenge a claim that the effect is at least that large. It does not establish no effect unless the interval is also sufficiently concentrated around zero relative to prespecified bounds.
Worked Examples: Three Different Patterns of Evidence
Worked Example: A Wide Interval Leaves the Result Inconclusive
Suppose a hypothetical original study estimates a positive expression contrast of \(\hat{\beta}_O=0.40\) log-base-2 expression units, and a replication estimates \(\hat{\beta}_R=0.18\) log-base-2 expression units with \(SE_R=0.15\). Assume the same contrast, outcome definition, and effect scale were used; the cohorts are independent; and the replication estimate is approximately normal with the stated standard error.
The replication interval is
The estimate is in the original positive direction, but the interval includes zero and the original estimate of \(0.40\). As a check on the usual two-sided test of \(H_0:\beta_R=0\), the statistic is \(z=0.18/0.15=1.20\), giving an approximate two-sided \(p\)-value of \(0.230\). This result does not meet a conventional 0.05 significance threshold. Yet the interval also permits a positive effect near the original estimate. The appropriate conclusion is that the replication is inconclusive about the original finding, not that it disproves it. The interval is expressed in log-base-2 expression-contrast units.
Worked Example: A Precise, Smaller Effect Challenges the Original Magnitude
Now suppose an independent hypothetical replication of the same contrast estimates \(\hat{\beta}_R=0.10\) log-base-2 expression units with \(SE_R=0.04\). The original estimate remains \(\hat{\beta}_O=0.50\) with \(SE_O=0.10\). Assume comparable measurement and analysis, independent cohorts, and normal approximations for both estimates.
The replication interval is
The interval is above zero, so the replication supports a positive effect, but it does not include the original estimate of \(0.50\). To quantify the difference, under independence the standard error of \(\hat{\beta}_O-\hat{\beta}_R\) is
The estimated difference is \(0.50-0.10=0.40\), so the standardized difference is \(0.40/0.1077\approx3.71\). This is strong evidence that the two estimated effects differ under the stated assumptions. The replication supports a positive effect but challenges the original magnitude; it does not support the claim that the effect is absent. Both effects and their difference are in log-base-2 expression-contrast units.
Worked Example: A Precise Estimate in the Opposite Direction
Suppose a third hypothetical replication estimates \(\hat{\beta}_R=-0.30\) with \(SE_R=0.08\) log-base-2 expression-contrast units, while the original estimate is \(0.40\) with \(SE_O=0.10\) in the same units. Assume the effect is defined in the same direction in both studies, the cohorts are independent, and the estimates are comparable and approximately normal.
The replication interval is
The entire interval is below zero, so the replication provides evidence for an effect in the opposite direction from the original estimate. The difference between estimates is \(0.40-(-0.30)=0.70\). Its standard error is
The direction reversal and large standardized difference make this a materially different pattern from the wide interval in the first example. Conditional on comparability and the assumptions stated, it is evidence against the original positive-direction finding. It still does not identify the cause of the discrepancy or establish that either study’s result is universally true.
Check Comparability Before Explaining the Discrepancy
Statistical precision is only one part of interpretation. A narrow interval cannot rescue a comparison that does not target the same effect. A careful assessment checks the following, using the locked protocol and cohort information:
- Effect definition and direction: Confirm that the outcome, contrast, coding, and sign convention match. A reversed reference category can flip the sign without any biological change.
- Assay and measurement: Check whether platforms, preprocessing, units, detection limits, and normalization support a valid comparison. A systematic assay shift can alter the estimated contrast, not merely make it less precise.
- Cohort features: Review eligibility, sample availability, and relevant differences in the cohorts. Differences can affect precision or the effect being estimated; they should be described rather than treated as proof of a particular explanation.
- Protocol adherence: Verify the prespecified endpoint, model, covariates, exclusions, and decision rule. A deviation may mean the reported analysis no longer answers the locked replication question.
These checks lead to a useful distinction. If the comparison is sound but the replication interval is wide, the evidence may be inconclusive because the data are not informative enough. If the assay or effect definition is materially different, a disagreement may be uninterpretable as a direct replication test. If the protocol was followed and the interval is narrow yet inconsistent with the original claim, the data provide stronger evidence against that claim. None of these interpretations alone proves why the results differ.
Worked Example: A Protocol Deviation Changes the Interpretation
Consider a hypothetical replication planned to estimate a treatment contrast in normalized protein abundance using a locked assay and prespecified adjustment set. The reported analysis instead uses a different platform, changes the normalization procedure, and adjusts for a different set of variables. Its estimate is \(-0.20\) units with \(SE=0.05\), giving
The calculation shows a precise negative estimate for the analysis actually run. But the planned and reported analyses do not necessarily estimate the same quantity: platform and normalization differ, and the adjustment set changed. The interval therefore cannot, by itself, establish that the locked replication contradicts the original result. The appropriate report separates the numerical result from its limitation: this analysis estimates a negative contrast under the modified measurement and model; comparability to the prespecified replication target is uncertain. The numbers are hypothetical and in assay-specific normalized abundance units.
Why the Distinction Matters
Calling every missed criterion “evidence that the finding is false” confuses a decision rule with the information in the data. A significance threshold asks whether the data are sufficiently inconsistent with a specified null under a model; it does not measure whether a replication is precise enough to assess the original magnitude. Likewise, matching the original study’s \(p\)-value is not a sound general criterion: \(p\)-values depend on both effect estimates and precision, and studies can have different sample sizes.
Alternative summaries answer different questions. Directional agreement asks whether estimates point the same way, but can ignore magnitude and uncertainty. Confidence intervals display uncertainty around each estimate. A formal comparison of independent estimates quantifies their difference under the assumed sampling model. An equivalence analysis, when meaningful bounds were specified in advance, asks whether effects are sufficiently close to a target range; failing to reject a zero-effect null is not a substitute for that analysis. The appropriate choice depends on the prespecified scientific claim.
Diagnostics should include the interval width, estimate direction, and consistency of the analysis with the protocol. Also inspect whether the comparison depends on a small number of observations or unusual exclusions, and whether standard-error assumptions are credible for the estimator and design. If uncertainty is underestimated, a seemingly narrow interval can create unjustified confidence. If the assay or protocol differs, report the deviation explicitly rather than treating a small \(p\)-value as proof that the original finding failed.
A defensible report states the criterion that was missed, gives the replication estimate and uncertainty in context, identifies the direction relative to the original result, and describes any material assay, cohort, or protocol differences. Its conclusion should match the evidence: inconclusive, evidence against a specified magnitude or direction, or not directly interpretable as a comparable replication.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A replication estimate is \(0.12\) units with \(SE=0.10\); the original estimate is \(0.30\) units. Compute a 95% interval and say whether the replication alone clearly contradicts the original estimate.
- A replication interval is narrow, entirely positive, and excludes the original estimate, but includes small positive effects. What does this support, and what does it not establish?
- Why can an estimate with a very small \(p\)-value still be difficult to interpret as a replication if the outcome coding was reversed?
- What additional assumption is needed for the standard error formula for the difference between two estimates used in this tutorial?
Solutions.
1. The interval is \(0.12\mathbin{\pm}1.96(0.10)=0.12\mathbin{\pm}0.196=(-0.076,\ 0.316)\) units. It includes zero and the original estimate of \(0.30\), so this result is inconclusive about whether the original effect is present at that magnitude.
2. It supports a positive replication effect while challenging the original magnitude. Because the interval includes small positive effects, it does not establish that the effect is zero or absent.
3. Reversing outcome coding can reverse the sign of an effect. A precise result may therefore look directionally inconsistent even when it represents the same underlying contrast with the opposite sign convention; coding must be aligned before comparing directions.
4. The two estimates must be independent for \(SE(\hat{\beta}_O-\hat{\beta}_R)=\sqrt{SE_O^2+SE_R^2}\) to follow from adding their variances. If they are dependent, their covariance must also be included.