Why a Non-Replication Needs Diagnosis
A significant association in one ancestry group and a non-significant result in another do not, by themselves, establish that the association failed to replicate. The second group may have too little power to detect an effect of the size observed in the first. The estimates may refer to different alleles, different variants in linkage disequilibrium with a causal variant, or differently scaled outcomes. Residual confounding can also distort one or both estimates.
A useful diagnosis therefore proceeds in a fixed order: check that the compared estimates refer to the same allele and variant; quantify power in the replication group; consider whether a tag variant captures the same association in both populations; verify that the effect scales match; and examine evidence of residual confounding. These checks do not prove that an association is causal. They identify whether the observed discrepancy is interpretable and what evidence would be needed to resolve it.
Start With Allele Harmonization
An association estimate is tied to a particular effect allele. If one study reports the effect of the A allele and another reports the effect of the G allele at the same biallelic variant, their coefficients have opposite signs under otherwise identical dosage coding. For a quantitative trait, changing the counted allele from A to G changes a coefficient \(\hat{\beta}\) to \(-\hat{\beta}\), while leaving its standard error unchanged. The p-value is unchanged because the test uses the magnitude of the standardized statistic.
Before comparing estimates, confirm the chromosome position, genome build, reference and alternate alleles, effect allele, other allele, and allele frequency. Strand ambiguity matters for A/T and C/G variants: strand complementation does not distinguish the two alleles at these sites. If frequencies and other available information do not resolve the orientation, the variant may need to be excluded rather than assigned a speculative sign. Also check that imputation quality and variant representation are adequate in both data sets.
Worked Example: Correcting an Allele-Orientation Discrepancy
Suppose a discovery analysis reports an effect of \(0.12\) phenotype units per G allele, with \(SE=0.04\). A replication file reports \(-0.10\) units per A allele, also with \(SE=0.04\). Assume both rows have been verified to refer to the same biallelic variant and the same phenotype scale.
The alleles are alternative alleles at the same biallelic site: one dosage counts G and the other counts A. For a biallelic variant, each additional A allele corresponds to one fewer G allele, so the replication estimate must be reversed to express it per G allele:
The discovery estimate is \(0.12\) and the harmonized replication estimate is \(0.10\), both in phenotype units per G allele. Their directions agree; the original negative sign was not evidence of an opposite effect. The replication test statistic is \(0.10/0.04=2.50\), with a two-sided normal-approximation p-value of about \(0.0124\). This example assumes the variant identity and allele coding have actually been verified. Merely seeing opposite signs is not enough to justify flipping an estimate.
Ask Whether the Replication Group Had Enough Power
A non-significant result is weak evidence against an association when the replication group has low power. For a two-sided test of a coefficient equal to zero, suppose the estimated coefficient is approximately normal with standard error \(SE\). If the true coefficient is \(\beta\), the standardized statistic has approximate distribution \(Z\sim N(\beta/SE,1)\). At significance level \(\alpha=0.05\), the usual two-sided rejection rule is \(|Z|>1.96\).
Power is the probability of rejecting the null under the specified alternative. It depends on the effect size used, the standard error, the test, and the significance threshold. It is not the probability that the effect is real. For a two-sided normal-approximation test, writing \(\mu_Z=\beta/SE\), the power is
where \(\Phi\) is the standard normal cumulative distribution function. This calculation is most useful when the effect size is a justified target, such as a harmonized discovery estimate or a prespecified scientifically relevant effect. If the discovery estimate is highly uncertain or inflated by selection, it should not be treated as a known true effect.
Worked Example: A Non-Significant Result With Low Power
Suppose a harmonized discovery effect is \(0.09995\) phenotype units per allele. In a hypothetical replication group, the standard error for this effect is \(0.0700\) units per allele. Use a two-sided test at \(\alpha=0.05\) and the normal approximation.
Under the target effect, the mean of the standardized statistic is
The upper-tail rejection probability is
The lower-tail probability is \(P(Z<-1.96)=\Phi(-1.96-1.427857)=\Phi(-3.387857)\approx 0.00035\). Thus the total power is approximately \(0.2973+0.00035=0.2977\), or about 29.8%. A non-significant result would therefore be unsurprising even if the target effect were correct. It would not establish that the effect is absent; the estimate and its uncertainty should also be examined.
Check Whether the Same Variant Represents the Same Association
Even after harmonizing alleles, two studies may not be testing the same biological signal. A common explanation is that the tested variant is a tag for a causal variant, and the correlation between tag and causal variant differs across populations. Under the simplifying assumptions of one causal variant, standardized genotypes, a linear model, and no confounding, the expected marginal coefficient at a tag variant is
where \(r\) is the correlation between the standardized tag and causal genotypes. With unstandardized genotype dosages, the corresponding expression is \(\beta_{\mathrm{tag}}=\operatorname{Cov}(G_{\mathrm{tag}},G_{\mathrm{causal}})\beta_{\mathrm{causal}}/\operatorname{Var}(G_{\mathrm{tag}})\). A change in linkage disequilibrium can therefore change the size, and in some configurations the direction, of a tag association even when the causal effect is unchanged. This is distinct from a difference in the causal effect itself.
Worked Example: A Weaker Tag Association Despite a Shared Causal Effect
Consider a hypothetical quantitative phenotype and a single causal variant whose effect is \(0.20\) standard deviations per standard-deviation increase in its genotype in both populations. Suppose the tested tag and causal genotypes have correlation \(r=0.80\) in population A and \(r=0.25\) in population B.
Under the stated single-causal-variant and standardized-genotype assumptions, the expected tag effects are
If the standard error in population B is \(0.040\), its expected standardized statistic at this effect is \(0.050/0.040=1.25\), which does not exceed the two-sided 5% critical value \(1.96\). The tag association can thus be much weaker and non-significant in population B without any change in the stipulated causal effect. This explanation is plausible only if the variant is actually a tag, the LD estimates are suitable for the analyzed populations, and the simple single-causal model is a reasonable approximation. Fine-mapping or association at the candidate causal variant can help distinguish this account from a causal-effect difference.
Compare Effect Scales Before Comparing Magnitudes
A coefficient has meaning only with its model and scale. For example, a quantitative-trait coefficient may be expressed in raw units or standard-deviation units; a binary-trait association may be reported as a log odds ratio, odds ratio, or risk difference. Genotype coding also matters: per allele and per standard-deviation genotype effects are not numerically interchangeable. Check the outcome definition, transformation, link function, covariate adjustment, and genotype scale before interpreting a difference in coefficients.
For binary outcomes, the same risk difference need not correspond to the same odds ratio at different baseline risks. Suppose one hypothetical setting has risk \(0.10\) without the allele and \(0.15\) with it. The risk difference is \(0.05\), while the odds ratio is
With risks \(0.40\) and \(0.45\), the risk difference is again \(0.05\), but the odds ratio is
These calculations illustrate why coefficients on different scales cannot be compared as though they were the same target. They do not imply that a genetic association has a fixed risk difference across populations. The outcome model and the estimand must be evaluated in context.
Investigate Residual Confounding
Ancestry adjustment reduces some sources of confounding but does not guarantee that all relevant structure is controlled. Residual confounding can arise when genotype frequencies and phenotype determinants vary together within or across ancestry groups, or when environmental, technical, or recruitment factors remain associated with genotype after adjustment. Calibration checks and the distinction between ancestry confounding and effect heterogeneity are useful earlier material to draw on here; neither an adjusted model nor a well-calibrated genome-wide summary proves that a particular locus is free of confounding.
A simple omitted-variable calculation shows why even modest residual association can matter. Suppose the actual quantitative-trait model is \(Y=\beta G+\gamma U+\varepsilon\), where \(G\) is genotype, \(U\) is an omitted factor, and the error has zero conditional mean. Regressing \(Y\) on \(G\) alone gives, in expectation,
This follows by taking the covariance of \(G\) with both sides of the model and dividing by \(\operatorname{Var}(G)\), assuming \(\operatorname{Cov}(G,\varepsilon)=0\). The expression is a diagnostic illustration, not a complete model for all forms of population structure.
Worked Example: Quantifying a Possible Omitted-Variable Bias
In a hypothetical analysis, let the true per-allele coefficient be \(0.10\) trait units, let an unadjusted environmental factor have effect \(\gamma=0.50\) trait units per unit of that factor, and suppose \(\operatorname{Var}(G)=0.20\) and \(\operatorname{Cov}(G,U)=0.020\). Assume the regression error is uncorrelated with genotype.
The omitted-variable term is
The expected unadjusted coefficient is therefore \(0.10+0.050=0.150\) trait units per allele, rather than \(0.10\). The direction of the bias depends on the signs of \(\gamma\) and \(\operatorname{Cov}(G,U)\). In practice, the omitted factor and its covariance may not be directly observed; this calculation clarifies the mechanism but does not diagnose or correct confounding on its own.
A Sequential Diagnostic Workflow
The checks are most informative when performed in order. If allele orientation is wrong, later comparisons of directions are meaningless. If the replication group is underpowered, the absence of significance is not strong evidence for heterogeneity. If the variants or effect scales differ, a direct coefficient comparison may not answer the intended replication question.
Check variant coordinates, genome build, allele labels, effect allele, strand ambiguity, and frequency consistency. Re-express estimates on the same counted allele.
Report the replication estimate and standard error, and calculate power for a justified target effect. Do not substitute a significance/no-significance comparison for an effect comparison.
Ask whether the studies tested the same variant and whether the tested marker tags the same causal signal in both populations. Examine ancestry-relevant LD and, where possible, the candidate causal variant.
Confirm that outcome units, transformations, link functions, genotype coding, and adjustment sets permit a meaningful comparison.
Review population-structure adjustment, calibration, and technical or environmental differences. State what the available results do and do not distinguish.
Diagnostics, Alternatives, and Limits
Warning signs include allele frequencies incompatible with the claimed allele orientation, a strand-ambiguous variant without reliable resolution, poor imputation in the replication group, a large standard error, and a tag variant whose LD with the candidate causal variant differs substantially across populations. Also examine whether reported effects use different phenotype transformations or genotype standardizations. Calibration diagnostics can flag broad inflation or structure, but a global diagnostic may miss locus-specific confounding; conversely, a locus-level discrepancy does not by itself show genome-wide miscalibration.
The main alternative to declaring a binary replication success or failure is to compare harmonized estimates and their uncertainty, alongside a prespecified replication criterion. A confidence interval for the replication effect indicates the range of values compatible with the data under the model; it is not a probability statement about the fixed effect. A formal heterogeneity test can assess whether estimated effects differ more than expected from sampling error, but low power can make that test inconclusive. Trans-ancestry meta-analysis can summarize information, yet its interpretation still depends on the effect model and comparability of the component estimates.
No single diagnostic can settle every case. LD reference panels may not represent the study participants, causal variants may be unknown, and residual confounding may remain despite careful adjustment. A stable causal effect is also not guaranteed merely because a tag association is explained by LD. The defensible conclusion may be that the replication result is inconclusive, that the marker association is not portable, or that further data are needed to distinguish those possibilities.
Common Mistake
Check Your Understanding
Answer each question before reading the short solutions.
- A replication coefficient is negative while the discovery coefficient is positive. What must be checked before concluding that their directions differ?
- Why does a replication p-value above 0.05 not demonstrate that the discovery effect is absent in the replication population?
- Under a single-causal-variant model, what happens to the expected standardized tag effect if the tag-causal genotype correlation falls from \(0.60\) to \(0.20\), with causal effect \(0.15\) in both populations?
- Why can identical risk differences correspond to different odds ratios?
- In the omitted-variable expression, what happens to the bias term if \(\operatorname{Cov}(G,U)=0\), under the stated assumptions?
Solutions. (1) Verify that the variant is the same and that the effect alleles, strand, and coding are harmonized; a biallelic allele reversal changes the coefficient's sign. (2) The test may have low power, so the data may not distinguish the target effect from zero; inspect the estimate, standard error, and power. (3) The expected tag effects are \(0.60(0.15)=0.090\) and \(0.20(0.15)=0.030\), respectively, despite the same causal effect. (4) Odds are nonlinear functions of risk, so the same absolute risk change at different baseline risks produces different odds ratios. (5) The omitted-variable term is zero, leaving no bias from this particular factor through the displayed covariance mechanism; other sources of bias may remain.