Diagnosing Failed Rare-Variant Replication

What Does a Failed Replication Tell Us?

A rare-variant result can fail to replicate for several different reasons. The replication cohort may contain too few informative carriers to detect an effect of the size reported in discovery. Alternatively, the cohorts may not be testing the same association: ancestry, phenotype definition, variant inclusion, or ability to call variants may differ. Technical artifacts can also produce an apparent discovery signal that does not recur in cleaner data.

These explanations should not be collapsed into a single conclusion. A nonsignificant replication result is not, by itself, evidence that the discovery result was false, nor does it show that the association differs between cohorts. The useful question is: which cohort-level differences are compatible with the observed estimates, uncertainty, and data quality?

For a simple carrier analysis, let \(G=1\) indicate that a participant carries at least one qualifying variant in a gene and \(G=0\) otherwise. For a binary phenotype, a cohort-specific logistic model can be written as

where \(Y\) is case status, \(C\) is the set of adjusted covariates, and \(\beta\) is the conditional log odds ratio for carrier status. This model presumes that the phenotype and carrier definition are meaningful and comparable, that the model adequately handles relevant confounding, and that dependence among participants is accounted for when present. For gene-level tests that do not produce this particular carrier coefficient, the same diagnostic logic applies to their specified effect or test statistic.

A useful comparison requires the cohorts to target the same quantity: the same phenotype, comparable participants and covariates, and a sufficiently harmonized gene test and variant definition. If these differ, distinct estimates may reflect distinct estimands rather than successful or unsuccessful measurement of one common effect.

Start With the Estimate and Its Uncertainty

Record the discovery and replication effect estimates, standard errors, confidence intervals, test statistics, and sample sizes. For a coefficient estimate \(\hat{\beta}\) with standard error \(SE\), the familiar standardized statistic is \(Z=\hat{\beta}/SE\). The estimate describes the observed direction and size; the standard error describes its sampling uncertainty under the fitted model. A replication estimate in the discovery direction can be compatible with the same effect even when it does not pass a significance threshold.

Also verify that the replication criterion was set before interpreting the result. A test may be nonsignificant at a chosen \(\alpha\) yet have an effect estimate close to discovery; conversely, a small p-value can accompany an effect too small to be scientifically meaningful. Neither p-values nor direction alone diagnose why results differ.

Separate Low Power From a Different Cohort

Power depends on more than the total sample size. For rare-variant tests, the number of informative carriers, the distribution of variants among them, phenotype prevalence, effect heterogeneity, and the test being used can all matter. A large cohort can still provide little information if few participants carry qualifying variants. Earlier material on power and simulation provides ways to plan such designs; here the task is to check whether the realized replication cohort plausibly had adequate information.

Then examine whether cohorts are comparable in the following dimensions. These checks are diagnostic, not automatic correction procedures.

Ancestry and Phenotype Diagnostics

Ancestry differences are important because rare-allele frequencies can vary substantially across populations. A gene-level carrier frequency contrast can therefore reflect differing ancestry composition, even when carrier status has no association with the phenotype within ancestry groups. Review ancestry proportions among cases and controls, inspect carrier frequencies within relevant strata, and check whether adjustment and matching are adequate for the study design. These comparisons complement, rather than replace, the earlier material on ancestry confounding.

Phenotype differences require a parallel audit. Compare diagnostic criteria, age at assessment, severity thresholds, measurement instruments, and ascertainment routes. For example, a discovery study of severe early-onset disease and a replication study of any adult diagnosis do not necessarily target the same phenotype. A weaker replication effect could reflect a less specific or less severe phenotype, not merely a smaller sample. Check whether control participants were assessed sufficiently to exclude the relevant phenotype; misclassified controls can dilute a contrast.

Compare What the Gene Test Actually Aggregates

A gene-level result is conditional on the set of variants and the rules used to include them. Compare the gene boundaries, functional annotations, frequency thresholds, quality filters, weights, and treatment of singletons. Then compare the variants and carriers that actually contributed in each cohort. A shared gene name does not guarantee a shared test. If cohorts include different mixtures of risk-increasing, neutral, or oppositely acting variants, an aggregate effect can change even when some underlying variant effects are shared.

Callability, Missingness, and Technical Artifacts

A variant that was not callable is not equivalent to a confidently observed reference genotype. Compare callability across cohorts at the variant and sample levels, including whether it differs by case status, ancestry, sequencing batch, or relevant genomic region. For gene tests, inspect the number of callable participants or chromosomes across the included sites, not just the total number sequenced. If missing calls are treated as noncarriers, lower callability can create an artificial frequency difference.

Technical artifacts may be concentrated in a sequencing batch, platform, capture region, or calling pipeline. Tabulate carriers and call-quality measures by batch and case status. A suspicious pattern includes a gene-level signal driven by one batch, especially if batch is imbalanced between cases and controls or carrier calls have atypical quality. Review orthogonal evidence and reprocessing where appropriate. A batch association is a warning requiring investigation, not proof that every call in that batch is false.

A Practical Diagnostic Sequence

The sequence below keeps statistical imprecision separate from comparability and measurement problems. More than one explanation can apply, and a diagnostic pattern should be treated as evidence to weigh rather than a categorical verdict.

Common Pitfall and Why the Diagnosis Matters

Wrong reasoning: “The replication p-value exceeds 0.05, so the discovery association was false.” This treats failure to reject the null as proof of no association and ignores the standard error, power, and comparability of the analyses.

Correction: report the replication estimate and its uncertainty, assess power under an explicit effect assumption, and inspect whether the cohorts measured comparable phenotypes and variant sets with comparable ancestry and callability. If the estimate is imprecise, the evidence may not distinguish no effect from an effect similar to discovery. If the analyses or measurement differ, the two estimates may not answer the same question. Neither situation alone establishes that the discovery result is true or false.

Careful diagnosis matters because different causes call for different next steps. Low power may motivate a larger or more informative replication; ancestry or phenotype mismatch may require a better-defined target population or outcome; differences in variant composition call for harmonized masks; and callability or batch concerns call for technical validation. Reporting the evidence behind the diagnosis is more informative than labeling a finding simply “replicated” or “not replicated.”