Why Phenotype Differences Can Undermine Replication
A genetic association can be estimated correctly in two cohorts and still appear not to replicate if the cohorts do not define or measure the outcome in the same way. One cohort might identify cases from a clinical examination, while another uses self-report; one might classify mild symptoms as disease, while another requires a severe threshold. These differences can change who counts as a case, the amount of measurement error, and even the scientific question represented by the effect estimate.
The first task is therefore not to explain away a discordant result. It is to ask whether both analyses estimate an effect for the same phenotype, on a comparable scale, in populations selected in comparable ways. A small or imprecise estimate under a different definition is not by itself evidence that the original association is false. Conversely, a difference in definitions cannot automatically excuse a conflicting result.
Define the Target Before Comparing Estimates
A replication claim concerns an estimand: the particular genetic effect the study aims to estimate. For example, the estimand might be the change in a continuous symptom score per additional effect allele in adults meeting a specified diagnostic definition. Changing the diagnostic threshold, age range, or score scale can change that target. Before comparing results, specify at least the outcome construct, operational definition, measurement scale, population, and genetic contrast.
Some differences are straightforward to reconcile. If two instruments measure the same construct and one score is a known positive linear rescaling of the other, estimates can be converted to common units. Other differences are not simple unit conversions. A diagnosis based on clinical records and a broad self-reported symptom measure may capture overlapping but distinct constructs. Forcing these into one variable may hide meaningful differences rather than remove them.
Ascertainment matters as well. It describes how people enter the study or are classified for analysis. A volunteer cohort, a specialty clinic, and a population-based sample can represent different populations, even if they use the same diagnostic wording. Sampling cases and controls according to outcome status can be compatible with estimating an odds ratio under suitable case-control sampling assumptions; that fact does not make all forms of clinic-based selection harmless, nor does it make the resulting population identical to that of another cohort.
Worked Example: Imperfect Case Classification
Hypothetical data. Suppose the target outcome is a true binary disease status. In a cohort, the true disease risks are 0.10 for genotype group \(G=0\) and 0.14 for \(G=1\). A replication cohort instead uses a classification with sensitivity \(Se=0.80\) and specificity \(Sp=0.90\). Assume the classification error is independent of genotype conditional on true disease status. Calculate the risk difference using the observed classification.
Let \(D\) denote true disease status and \(D^*\) the recorded classification. For a person with true disease, the probability of a positive classification is \(Se\); for a person without disease, it is \(1-Sp\). Thus, for genotype group \(g\),
For \(G=0\), the observed positive proportion is \(0.80(0.10)+0.10(0.90)=0.17\). For \(G=1\), it is \(0.80(0.14)+0.10(0.86)=0.198\). The observed risk difference is therefore \(0.198-0.17=0.028\), compared with the true risk difference \(0.14-0.10=0.040\).
Under these assumptions, the recorded risk difference is 0.70 times the true difference: it has been attenuated by 30%. The calculation illustrates one way a genuine effect can look weaker in a replication cohort. It does not establish that a particular replication failure is due to misclassification. In particular, this exact attenuation result is for a risk difference under the stated error model; it should not be applied as an exact multiplier to a logistic-regression odds ratio.
What Harmonization Can and Cannot Do
A defensible harmonization rule should be specified from the phenotype information, not chosen after seeing which rule produces the most favorable genetic result. A useful first distinction is between differences that preserve the construct and differences that change it.
- Potentially reconcilable measurement differences: different labels for the same recorded condition, equivalent units, or score versions with a documented conversion.
- Potentially consequential definition differences: different diagnostic thresholds, different symptom domains, or one definition that includes conditions excluded by another.
- Ascertainment differences: different routes to diagnosis or selection that may change the population represented or the relationship between recorded status and underlying disease.
For a quantitative phenotype, a known transformation can put estimates on a common scale. If a cohort records \(Z=a+bY\), with known \(b>0\), then the per-allele regression coefficient for \(Z\) is \(b\) times the coefficient for \(Y\), and its standard error is also multiplied by \(b\). The additive constant \(a\) changes the intercept but not the per-allele slope. This conversion requires a genuinely linear scale relationship; it does not repair differences in what the instruments measure or in who was assessed.
Standardizing each cohort’s phenotype to its own standard deviation can produce effects expressed in cohort-specific standard-deviation units. Such effects may be useful, but they do not automatically represent the same absolute change in the underlying trait: different score variability can make one standard deviation correspond to different clinical changes. State the scale clearly and do not treat standardization as a substitute for phenotype equivalence.
Worked Example: Converting a Score to Common Units
Hypothetical data. Cohort A measures a symptom score in points on a 0–100 scale and estimates a per-allele effect of \(0.32\) points with \(SE=0.08\). Cohort B uses a documented scale for which its score \(Z\) equals half the Cohort A score for the same person. In Cohort B, the estimated effect is \(0.18\) \(Z\)-points per allele with \(SE=0.07\). Convert Cohort B’s estimate to Cohort A units and compare the estimates.
Because \(Z=Y/2\), the effect in \(Y\)-points is twice the effect in \(Z\)-points. Its standard error is converted by the same factor:
Cohort A estimates \(0.32\) points per allele and Cohort B estimates \(0.36\) points per allele after conversion. Their estimates have the same direction and are close relative to their uncertainty. For example, approximate 95% confidence intervals are \(0.32\pm1.96(0.08)=(0.163,0.477)\) for A and \(0.36\pm1.96(0.14)=(0.086,0.634)\) for B. Each interval is in Cohort A score points per allele.
This conversion is justified only because the example stipulates a documented linear relationship between the scores. If the instruments measured different symptom domains, or if the conversion varied across the score range, multiplying by two would not make the phenotypes equivalent.
Stratify When the Definitions May Represent Different Outcomes
When definitions are not safely interchangeable, retain the distinction in the analysis. Estimate effects separately by phenotype definition or ascertainment method, with the same allele coding and a clearly described effect scale where possible. Stratified estimates reveal whether the association is similar across definitions; they also prevent a single pooled number from concealing that one subgroup measured a materially different outcome.
A difference between estimates should be interpreted with their uncertainty. One estimate being statistically significant and another not significant does not establish that the estimates differ. The relevant question is the size and uncertainty of the contrast between them, alongside the scientific reason for the strata. For independent estimates \(\hat{\beta}_1\) and \(\hat{\beta}_2\), an approximate standard error for their difference is \(\sqrt{SE_1^2+SE_2^2}\). This calculation is only appropriate if the estimates are independent; dependence requires accounting for their covariance.
Worked Example: Preserving Two Phenotype Definitions
Hypothetical data. A replication cohort reports a per-allele risk difference of \(0.060\) with \(SE=0.020\) for clinically confirmed cases, and \(0.025\) with \(SE=0.020\) for a broader self-reported definition. Assume the two estimates are independent for this illustrative comparison. Decide what the estimates support.
The estimated contrast is \(0.060-0.025=0.035\). Its standard error is
The contrast is about \(0.035/0.0283=1.24\) standard errors from zero. This is not strong evidence, on its own, that the effects differ. The estimates nevertheless point to a relevant design concern: the effect estimate is larger under the narrower clinical definition, and the broad definition may capture additional people who do not meet that definition.
The appropriate conclusion is to report both strata and their uncertainty, investigate how the definitions were applied, and avoid claiming that one estimate proves a phenotype-specific genetic effect. If the scientific target is specifically clinically confirmed disease, a replication analysis restricted to that definition may be more interpretable than pooling the two definitions.
When a Narrower Estimand Is Preferable
Harmonization is not always possible. A narrower estimand may be justified when a subset of cohorts shares a sufficiently consistent definition, ascertainment scheme, and measurement scale. The target then becomes explicit—for example, the genetic effect on clinically confirmed disease in cohorts using a specified diagnostic algorithm—rather than an ambiguous average across unlike outcomes.
This choice has costs. Excluding cohorts can reduce precision and limit generalizability. The resulting estimate does not answer the broader question for people identified only by a looser definition. Report which cohorts or participants are included, why the restricted definition is coherent, and how the target population has changed. Do not choose the narrow definition solely because its genetic estimate is larger or more statistically convincing.
A practical sequence is to describe each definition and ascertainment process, identify defensible transformations or common rules, estimate comparable strata when important differences remain, and state the estimand for any restricted analysis. If the differences cannot be reconciled and the strata are too sparse for useful estimates, the honest conclusion is that the available evidence does not support a single directly comparable replication estimate.
Record the definition, threshold, instrument, assessment source, and relevant selection process for each cohort.
Convert units only when a documented transformation supports it; do not equate overlapping but distinct clinical outcomes by assumption.
Harmonize defensible differences, retain meaningful differences as strata, or define a narrower target when only a subset is comparable.
Compare effect sizes and their uncertainty, not just whether each cohort crossed a significance threshold, and state the limits of generalization.
Why Replication Can Fail—and a Common Mistake
Phenotype heterogeneity can affect replication through more than one route. Imperfect classification can reduce contrasts between groups, as in the binary example. A broader definition may include people with milder or different manifestations and therefore target a different effect. A noisy instrument may increase the standard error, reducing the chance of detecting an association even when the expected effect is similar. A different selection process may alter the study population or the relation between recorded phenotype and underlying disease. These mechanisms are not interchangeable, and the phenotype description is needed to assess which are plausible.
Common mistake: “The replication estimate was not significant, so the genetic association did not replicate; alternatively, the phenotype was different, so the failure does not count.” Both conclusions overreach. A non-significant result does not establish a zero effect, and a phenotype difference does not automatically explain the result. Instead, compare the effect estimates and uncertainty, determine whether they target the same outcome and scale, and justify any harmonization, stratification, or restriction using the phenotype information rather than the observed association alone.
Check Your Understanding
Use the phenotype definition and target estimand—not only the p-value—to guide each answer.
- Under what assumptions does the binary misclassification example multiply a risk difference by \(Se+Sp-1\), and why should that result not automatically be applied to an odds ratio?
- A score in one cohort is exactly twice the corresponding score in another. How should the per-allele effect and its standard error be converted to the first cohort’s units?
- Why does one cohort having a significant result and another having a non-significant result fail to demonstrate that their genetic effects differ?
- Give one reason to stratify estimates by phenotype definition rather than pool them, and one limitation of stratification.
- When might a narrower estimand be more defensible than a broad harmonized phenotype? What generalization does the narrower estimate give up?