Confidence Intervals for Replication Effects

Why Confidence Intervals Matter in Replication

A replication estimate is not a single definitive answer: its uncertainty matters. A confidence interval shows a range of effect values reasonably compatible with the observed data under a specified model. It therefore helps distinguish a precise estimate from an uncertain one, and helps assess whether the data are compatible with the discovery estimate or with differences large enough to matter scientifically or clinically.

Building on the earlier material on Replicating Effect Directions and Magnitudes, this tutorial focuses on uncertainty around the replication effect and around its difference from discovery. The interval is not a pass-or-fail replication rule. Its interpretation depends on the estimand, the cohort design, and a suitable standard error.

Definition and Assumptions

Let \(\hat{\theta}_R\) be the replication-cohort estimate of the effect parameter \(\theta_R\), and let \(SE_R\) be its standard error. If the estimate’s sampling distribution is approximately normal, a two-sided 95% Wald confidence interval is

The number 1.96 is the 97.5th percentile of the standard normal distribution. The formula is appropriate when the standard error estimates the sampling variability well and the normal approximation is adequate. In small samples or for a model with a different reference distribution, the appropriate critical value may instead come from a \(t\) distribution or another justified procedure. The interval must be on the effect scale being reported, with units or a clear scale definition.

Before comparing a replication estimate with discovery, check that both estimates target the same quantity: for example, the same outcome contrast, covariate adjustment, feature definition, and effect scale. For the simple difference calculation below, also assume the cohorts are independent and that the estimates’ sampling errors are independent. If those errors are correlated, their covariance must be included.

How to Read an Interval

The interval’s width conveys precision: narrower intervals indicate less sampling uncertainty under the model. Precision is not the same as effect size or statistical significance. A narrow interval can be centered near zero, while a wide interval can include both negligible and large effects. If a two-sided 95% interval excludes zero, that corresponds to rejection of a zero-effect null hypothesis at the 5% level under the same model and test construction. If it includes zero, the data have not ruled out zero at that level; this does not establish that the effect is zero.

A 95% confidence interval does not mean there is a 95% probability that the fixed parameter lies inside this particular interval. Its 95% interpretation concerns the long-run coverage of the procedure. In applied reporting, the interval is useful for seeing which effect values remain compatible with the data and model, while keeping this repeated-sampling interpretation in view.

Comparing Replication and Discovery

A replication interval describes uncertainty in \(\theta_R\). A separate question is whether the replication effect differs from the discovery effect \(\theta_D\). When both estimates target the same parameter and come from independent cohorts, the estimated difference is \(\hat{\theta}_R-\hat{\theta}_D\). Its standard error is obtained from the variances of the independent estimates:

A normal-approximation interval for the difference is this estimated difference plus or minus 1.96 standard errors. It directly addresses the difference, unlike simply looking at whether the two separate intervals overlap. If the estimates are not independent, the general variance formula includes their covariance:

For independent cohorts, the covariance is zero. If the cohorts overlap or share data in a way that correlates estimates, treating them as independent can give an incorrect standard error. The effect scales also need to be comparable; subtracting estimates measured on different scales has no useful interpretation.

Separate confidence intervals are not a substitute for this calculation. Their overlap is not a formal test of equality: the relationship between overlap and a difference test depends on the intervals’ confidence levels and standard errors. Non-overlap can be a signal of a difference, but overlap alone does not establish compatibility or equality. When the scientific question concerns the difference, report an interval for the difference.

Meaningful Effects and Meaningful Differences

Statistical compatibility with zero does not tell us whether an effect is large enough to matter. That judgment needs a scientifically justified threshold, specified in the effect’s units and preferably before examining the replication results. For a positive effect, a threshold might be the smallest effect considered clinically important; for comparing cohorts, a tolerance \(\Delta\) might represent the largest discovery–replication discrepancy considered unimportant.

Compare the whole interval with the threshold, not only its point estimate. If the interval for a positive replication effect lies entirely above a prespecified meaningful-effect threshold, the data support an effect exceeding that threshold under the model. If the interval crosses the threshold, the estimate may be statistically distinguishable from zero while uncertainty remains about whether its magnitude is meaningful. For a difference interval, lying wholly within \([-\Delta,\Delta]\) indicates that differences outside the prespecified tolerance are not supported by that interval at its confidence level. This is stronger than merely including zero, but it does not prove exact equality.

Diagnostics, Alternatives, and Limitations

Before interpreting an interval, check whether the analysis supports it. Relevant diagnostics include the sample size and effective information, the estimated standard error, model fit, influential observations, and whether the reported interval method is suitable for the estimator and data. For a highly skewed estimator, boundary-constrained parameter, or sparse data, a symmetric Wald interval may have poor coverage or extend into impossible values. A justified alternative, such as a profile-likelihood or bootstrap interval, may be preferable; the method and its assumptions should be reported. A standard error that ignores clustering or other dependence can make an interval too narrow.

Cohort comparability is also essential. Differences in outcome definition, assay scale, covariate adjustment, or target population can mean that the estimates do not represent the same estimand. An interval calculation cannot repair that mismatch. Likewise, a wide interval may result from limited information rather than absence of an effect, while a narrow interval only reflects precision conditional on the model and data quality.

A separate interval for each cohort is a convenient descriptive summary; an interval for the difference is the direct comparison when its assumptions hold. These are distinct from pooling estimates across cohorts, which answers a different question and requires additional modeling choices. No single confidence interval settles transportability, clinical importance, or whether a replication design was adequate.