Diagnosing Failed Genetic Replications

Why a Failed Replication Needs a Diagnosis

A replication study can fail to reach statistical significance for several different reasons. The variant may have been coded with the opposite effect allele, the phenotype may not measure the same trait, or the replication genotypes may be imprecise. Differences in ancestry can also change allele frequencies and linkage disequilibrium, while a smaller replication sample may simply have too little power to detect the effect. Finally, a discovery estimate selected for being unusually strong may overstate the effect expected in a new sample.

These possibilities have different implications. A coding mismatch is a technical problem to correct; a low-powered result may be inconclusive; and a sufficiently precise estimate near zero can count as evidence against an effect of the size that was proposed. The first task is therefore not to label a result “replicated” or “not replicated,” but to check whether the two studies estimated comparable quantities and whether the replication was capable of distinguishing the relevant possibilities.

A Diagnostic Sequence

Begin by comparing the studies’ reported estimates, standard errors, allele labels, phenotype definitions, and analysis populations. The following sequence helps separate incompatibilities from statistical uncertainty. The checks are not always independent: for example, ancestry can affect both imputation accuracy and the relationship between a tested marker and a causal variant.

Allele Alignment and Comparable Effects

For an additive genetic association, a coefficient usually describes the expected change in a phenotype for each additional copy of a specified allele, conditional on the model’s covariates. If one study reports the effect of allele A and another reports the effect of the other allele G at the same biallelic variant, their coefficients have opposite signs when expressed on the same phenotype scale. The estimates should be aligned before comparing directions.

For example, suppose a discovery study reports a coefficient of \(0.12\) phenotype units per copy of A. A replication study reports \(-0.11\) units per copy of G at the same variant. Since A and G are the two alleles, the replication estimate expressed per copy of A is \(+0.11\), not \(-0.11\). After this recoding, the two estimates have the same direction. By contrast, a strand complement changes the allele label (for example, A to T) but does not necessarily reverse the coefficient; the key is to establish which biological allele each coefficient counts. Strand-ambiguous variants require particular care because allele labels alone may not resolve orientation.

A frequency check can flag errors: after harmonizing populations and allele definitions, an allele reported as common in one study should not ordinarily appear rare in the other without an explanation. Frequency differences can be real across ancestries, so this check is diagnostic rather than definitive.

Phenotype, Imputation, Ancestry, and Effect Scale

A replication is meaningful only to the extent that it tests a comparable association. Phenotype alignment includes more than matching trait names. The studies should be compared on the operational definition of the outcome: how it was measured, when it was measured, who was eligible, how repeated measurements were combined, and whether transformations or exclusions differ. A diagnosis code and a direct laboratory measurement may capture overlapping but non-identical traits. Such differences can attenuate or otherwise alter an association without implying that one study is erroneous.

The effect scale also matters. For a quantitative trait, a coefficient in raw units is not numerically comparable to one per standard deviation unless the scaling is accounted for. For a binary trait, a logistic-regression coefficient is on the log-odds scale; it is not itself an odds ratio. A coefficient reported as a log odds ratio can be compared with another log-odds coefficient when the allele, outcome, and model are aligned. For a standardized quantitative phenotype, a coefficient is measured in standard deviations per allele copy. These distinctions are central to deciding whether two numerical estimates disagree.

Imputation quality should be checked for the tested variant in each cohort, including whether the variant was directly genotyped or imputed and the cohort-specific quality metric. Poor imputation can increase uncertainty and, depending on the analysis and data quality, may also contribute to attenuation or bias. There is no universal quality cutoff that substitutes for inspecting the metric, its definition, sample size, allele frequency, and analysis pipeline. A high-quality result in one cohort does not certify the variant in another.

Ancestry differences require careful interpretation. Allele frequencies can differ, and a marker can tag a nearby causal variant differently because linkage disequilibrium patterns differ. Thus, different association estimates at the same marker need not mean that the underlying biology has changed. Conversely, ancestry should not be used as an automatic explanation for any disagreement: check sample sizes, frequencies, imputation, phenotype measurement, covariate adjustment, and uncertainty. A result at a tag variant is not necessarily expected to transfer identically if the marker’s relationship to the causal variant differs across populations.

Non-Significance, Power, and the Winner’s Curse

A test statistic and its p-value summarize compatibility with a null value under the model; they do not measure the probability that the null hypothesis is true. For a large-sample two-sided test of a coefficient, a common approximation is \(z=\hat{\beta}/SE\), with the p-value obtained from the standard normal distribution under the null. The confidence interval gives an additional view of precision: a wide interval may include both zero and effects of scientifically meaningful size.

Power is the probability that a specified test rejects the null under a specified alternative, given the assumed effect, standard error, and significance level. It depends on the true effect and the replication design. If the discovery estimate is used as the target without qualification, power may be overstated because discovery estimates selected for strong evidence tend to be exaggerated in magnitude. This selection-related overestimation is known as the winner’s curse. It does not mean every discovery estimate is biased upward by the same amount; it means the selected estimate is not automatically an unbiased planning value.

For a two-sided normal-approximation test at level \(\alpha\), let \(c=z_{1-\alpha/2}\) be the positive critical value. If the true standardized test statistic has mean \(\delta=\beta/SE\) and variance one, the power is:

where \(\Phi\) is the standard normal cumulative distribution function. The first term is the probability of crossing the positive critical value; the second is the probability of crossing the negative critical value. This calculation makes clear why a modest effect with a large standard error can produce a non-significant result even when the association is real.

Technical Failure or Evidence Against the Association?

The distinction depends on whether the replication provides a valid and sufficiently precise estimate of the same association. A technical or comparability problem is plausible when allele orientation is unresolved, the phenotype estimand differs, imputation is poor, or analysis choices do not correspond. In such cases, the immediate conclusion is that the replication is difficult to interpret as a test of the discovery claim. Correct the problem, if possible, and then reassess; do not silently treat incompatible estimates as directly comparable.

A non-significant but imprecise estimate is generally inconclusive, not evidence of no association. More informative evidence against a proposed effect arises when the variant and phenotype are well aligned, data quality is adequate, and the replication estimate is precise enough to exclude effects that would matter. The interval, not just the p-value, is essential. For example, an interval tightly concentrated near zero could conflict with a claimed large effect even if the study was not framed as a formal equivalence test. A wide interval spanning zero and the proposed effect cannot make the same distinction.

Direction also matters, but it must be assessed after harmonization. A negative estimate opposite to discovery is not meaningful evidence of inconsistency if the studies count opposite alleles. Once the allele is aligned, a precise estimate in the opposite direction is more concerning than a noisy estimate with the same sign. In either case, assess whether ancestry, phenotype definition, imputation, or other design differences could plausibly account for the pattern before attributing it to biological heterogeneity.

Common Mistake

Wrong reasoning: “The replication p-value is greater than \(0.05\), so the association has been disproved.” This treats failure to reject a point null as proof that the null is true and ignores the replication’s precision.

Correction: First confirm that the variant, effect allele, phenotype, and effect scale are comparable and that the genotype data are adequate. Then report the aligned estimate and its uncertainty, and consider power for a plausible effect. A non-significant result with a wide interval may be inconclusive; a precise estimate that excludes effects of interest can provide stronger evidence against the proposed association.