What Does It Mean for a Genetic Effect to Replicate?
A variant can have an association estimate in the same direction in two cohorts and yet be statistically significant in only one. The estimates can also be similar in size while referring to different populations or different phenotype scales. Consequently, “the association replicated” is incomplete unless the intended target effect and the meaning of replication are stated.
The central object is the estimand: the population quantity the analysis aims to estimate. For genetic replication, it is not enough to name a variant and report two p-values. One must specify which allele is counted, which phenotype is analyzed, which population the effect refers to, and what effect scale and model define the parameter. A cohort estimate is evidence about that target, subject to sampling uncertainty and the analysis assumptions.
Specify the Target Effect Before Comparing Cohorts
For a biallelic variant, let \(G\) denote the number of copies of a designated effect allele, so that \(G\) takes values 0, 1, or 2. Under an additive model for a quantitative phenotype \(Y\), a common specification is
where \(C\) denotes the covariates in the model. In this specification, \(\beta\) is the expected difference in the phenotype for each additional copy of the effect allele, conditional on the included covariates. The units of \(\beta\) are the units of \(Y\) per allele. The model assumes the expected phenotype changes linearly with allele dosage on this scale; it does not, by itself, establish that the association is causal.
For a binary phenotype, one possible specification is logistic regression:
Here \(\beta\) is the change in log odds per additional effect-allele copy, conditional on the covariates. Its exponentiation, \(\exp(\beta)\), is the corresponding odds ratio per allele. A coefficient for a quantitative phenotype and a log-odds coefficient are different parameters; their numerical values are not directly comparable.
The population is part of the target. For example, an effect among adults with a particular clinical profile is not automatically the same estimand as an effect among all adults in a community. Likewise, an effect conditional on specified covariates is not automatically the same as a population-averaged effect. Two cohorts can estimate the same target only when their allele definition, phenotype, scale, population target, and model-based parameter are sufficiently aligned.
Worked Example: Naming the Estimand in a Quantitative-Trait Study
Suppose a hypothetical study analyzes a standardized blood biomarker in adults from a defined community. The variant is coded as the number of copies of allele A, and the analysis uses an additive linear model adjusted for age and sex. The estimated coefficient is \(0.12\) standard deviations per allele.
Target effect. The relevant parameter is the conditional mean difference in standardized biomarker level for one additional copy of allele A, among the specified adult population, conditional on age and sex under the additive model. Its units are standard deviations per allele.
Estimate. The value \(0.12\) is the cohort’s estimate of that parameter. It is not a universal property of the variant independent of population, phenotype definition, or model.
If a second cohort instead counts allele G, uses a differently defined biomarker, or reports the phenotype in raw concentration units, its coefficient cannot be compared with \(0.12\) as though the two numbers necessarily measured the same effect. The first step is to establish what each coefficient means; a numerical comparison comes only after the targets are comparable.
Three Distinct Replication Claims
Replication can refer to different features of the evidence. These features should be reported separately rather than treated as interchangeable.
| Claim | Question | What it does not establish by itself |
|---|---|---|
| Direction | Does the estimated effect have the same sign for the same specified allele and phenotype? | That the effects are equally large or statistically significant |
| Magnitude | Are the effects sufficiently close on the same scale for the scientific purpose? | That either estimate is precise or individually significant |
| Statistical significance | Does a test reject the specified null hypothesis at the chosen significance level? | That the effect is large, important, or equal to the discovery estimate |
Directional replication concerns the sign of the effect: positive in both cohorts or negative in both, with the same allele and phenotype coding. A matching sign is a limited claim. It can occur even if estimates differ substantially, and a sign based on a highly uncertain estimate is weak evidence about the underlying direction.
Magnitude replication concerns closeness of effect sizes. “Close” needs a scientific definition on a comparable effect scale. One possible criterion is that the difference between the replication and discovery effects is no greater than a prespecified tolerance \(\delta\):
where \(\beta_D\) and \(\beta_R\) denote the target effects in the discovery and replication settings. The tolerance is expressed in the same units as the effect. Exact equality is rarely something finite samples can establish; estimates and their uncertainty must be considered. Merely observing similar point estimates does not prove the effects are equal, and a nonsignificant test of difference does not prove closeness.
Statistical significance is a decision about a hypothesis test, not a definition of replication. For a two-sided test of \(H_0:\beta=0\), the p-value measures how unusual a test statistic at least as extreme as the observed one would be if the null model were true. A small p-value is not the probability that the null hypothesis is true. Statistical significance also depends on the standard error and the chosen \(\alpha\), not only on effect size.
Worked Example: Same Direction and Similar Estimates, Different Precision
Consider hypothetical estimates of the same quantitative-trait effect, measured in standard deviations per copy of the same allele. In the discovery cohort, \(\hat{\beta}_D=0.20\) and \(SE_D=0.05\). In the replication cohort, \(\hat{\beta}_R=0.16\) and \(SE_R=0.08\). Assume the estimates are approximately normal and use a two-sided test at \(\alpha=0.05\).
For the discovery cohort, the Wald statistic is
The two-sided p-value is approximately \(0.0000633\). A 95% confidence interval is
For the replication cohort,
giving a two-sided p-value of approximately \(0.0455\). Its 95% confidence interval is
At the stated nominal \(\alpha\), both tests reject a zero effect, and both estimates are positive. The point estimates differ by \(0.16-0.20=-0.04\) standard deviations per allele. These results support matching direction and provide estimates that are close, but the wide replication interval indicates substantial uncertainty about its magnitude. The calculation alone does not prove equality of the effects or establish that a particular magnitude tolerance has been met.
Why Direction and Significance Can Disagree
A positive estimate can be consistent with the discovery direction without being statistically significant. Under the approximate normal model for an estimate, the test statistic for a zero-effect null is
For a two-sided test at \(\alpha=0.05\), rejection occurs when \(|Z|>1.96\), approximately. Thus, rejection depends on the estimate relative to its standard error. A smaller cohort, a less precise phenotype measurement, or other sources of uncertainty can increase the standard error and prevent rejection even when the estimated effect remains positive and scientifically similar.
Worked Example: Directional Agreement Without Statistical Significance
In a second hypothetical comparison, suppose the discovery estimate is \(0.30\) phenotype units per allele with \(SE_D=0.06\), while the replication estimate is \(0.24\) units per allele with \(SE_R=0.15\). The phenotype, allele coding, population target, and model are assumed comparable. Use a two-sided test at \(\alpha=0.05\).
The discovery statistic is \(0.30/0.06=5.00\), with a two-sided p-value of approximately \(5.73\times10^{-7}\). The replication statistic is
Its two-sided p-value is approximately \(0.110\), so it does not reject a zero effect at the stated significance level. The replication estimate is nevertheless positive, matching the discovery direction. Its 95% confidence interval is
The interval includes zero and also includes the discovery estimate \(0.30\). Therefore, the replication result is imprecise: it does not establish a nonzero effect at this threshold, but it is also compatible with an effect close to the discovery estimate. It would be incorrect to conclude that the association necessarily failed to replicate solely because this p-value exceeds \(0.05\).
Population Differences Can Change the Target
A difference between cohort estimates is not always a contradiction. If the genetic effect varies across relevant subgroups, cohorts with different subgroup composition can target different population-averaged effects. The population composition is then part of what the estimand means.
For a simple illustration, suppose an effect is defined as a weighted average of two stratum-specific per-allele effects. Let \(\beta_1\) and \(\beta_2\) be effects in two population strata, and let \(w\) be the proportion of the target population in stratum 1. Define the population-averaged effect as
This expression is the chosen average over the two strata; it is not a claim that every regression coefficient in every model must equal this weighted average. Its purpose is to make explicit that changing the target population’s weights can change the target effect when stratum-specific effects differ.
Worked Example: Different Population Averages, Same Stratum Effects
Suppose, hypothetically, that a quantitative phenotype has a per-allele effect of \(0.40\) units in stratum 1 and \(0.00\) units in stratum 2. Define each cohort’s target as the population-weighted average of these stratum-specific effects. Cohort A has 75% of its target population in stratum 1; Cohort B has 25%.
For Cohort A, \(w_A=0.75\), so
phenotype units per allele. For Cohort B, \(w_B=0.25\), so
units per allele. Both target effects are positive, but their magnitudes differ because the target populations have different stratum weights. If the scientific question concerns the effect in each cohort’s own population, these are distinct population-specific estimands. If the question instead concerns a common target population, the comparison must refer to that common target. The arithmetic does not show that population composition is the cause of any real cohort difference; it illustrates how a specified estimand can depend on population weights.
Common Mistakes and a Practical Interpretation
- Calling any positive estimate a successful replication. A matching sign is directional agreement, not proof of a matching magnitude or a statistically supported effect. Report which of these claims the evidence supports.
- Calling a nonsignificant result evidence of no effect. A large p-value can arise from an imprecise estimate. Examine the estimate and interval in context; lack of rejection is not proof that the effect is zero.
- Equating significance with a large effect. A small effect can be statistically significant when estimated precisely, while a larger effect can be nonsignificant when uncertain.
- Comparing coefficients without defining their units or targets. A per-allele effect in one phenotype scale or population is not automatically comparable to a coefficient on another scale or for another population.
- Treating similar point estimates as proof of equal effects. Similarity is a descriptive observation. A claim of magnitude agreement requires a stated scale and a scientific criterion for how close is close enough, alongside uncertainty.
A clear replication summary therefore names the target effect, gives the cohort estimates and their uncertainty, and separates the conclusions about direction, magnitude, and significance. This makes the claim interpretable even when the cohorts differ in precision or population composition.
Check Your Understanding
Answer each question by distinguishing the target effect from the evidence in the sample.
- What information must be specified to interpret a per-allele genetic effect as a particular estimand?
- Two cohorts estimate positive effects for the same allele, but only one p-value is below \(0.05\). Which replication claim is supported by the matching signs, and which claim is not established by them?
- Why does a nonsignificant replication result not by itself show that the genetic effect is zero?
- What additional criterion is needed to make a precise claim that two effect magnitudes are sufficiently close?
- In the population-average example, why did the two cohort targets differ even though the stratum-specific effects were held fixed?