Design Choices That Undermine GWAS Reproducibility

What Reproducibility Requires in a GWAS

A genome-wide association study (GWAS) result is not just a variant and a p-value. It is an estimate tied to a particular phenotype definition, recruited population, genetic variant or proxy, measurement platform, and analysis. If a later study changes one or more of these, its estimate may differ even when both studies have been conducted carefully. Conversely, repeating the same design does not guarantee identical estimates: estimates vary because each sample is finite.

In this tutorial, reproducibility means that a result can be evaluated in a new study with sufficiently comparable measurements and design, and that the estimates are reasonably consistent in direction and magnitude given their uncertainty. It does not mean that two studies must produce identical estimates or identical p-values. The first diagnostic question is therefore not simply “Did the result replicate?” but “Were the studies estimating and measuring sufficiently similar things?”

Several design choices can make that comparison difficult. Phenotype drift changes the outcome definition. Selective recruitment can change who enters the study and, in some circumstances, create or distort associations. A small sample yields imprecise estimates. Ancestry underrepresentation limits the populations in which the result can be assessed and may change the performance of a marker. Platform limitations can leave a variant unmeasured or poorly represented. These issues can occur together.

Phenotype Drift: Different Definitions, Different Outcomes

A phenotype is an operational definition of the trait or disease used in an analysis. It may come from clinical records, laboratory measurements, medication records, questionnaires, or a combination. Two studies can use the same disease name while applying different rules for who counts as a case. They can also use nominally identical rules but differ in measurement timing, instruments, thresholds, or data availability.

These differences matter because an association estimate refers to the outcome as defined in that study. For example, an association with a carefully adjudicated diagnosis is not automatically the same estimand as an association with self-reported history of that diagnosis. A broader definition may include milder cases or people whose reports have not been clinically confirmed. A stricter definition may exclude some true cases while yielding a more specific case group. Neither choice is universally preferable; the key is to state the definition and assess its comparability.

Phenotype harmonization makes definitions sufficiently comparable for a planned analysis; it does not make every source of measurement error disappear. If studies cannot use identical data, they should document the differences, describe their likely consequences, and avoid presenting the results as though the outcomes were interchangeable.

Recruitment and Selection: Who Enters the Study?

Recruitment determines the population represented by the analyzed sample. A hospital-based study, a volunteer biobank, and a population-based cohort may enroll people with different health profiles, access to care, socioeconomic circumstances, and willingness to participate. A result in a recruited sample does not automatically describe every population from which participants might have come.

Selection is especially concerning when inclusion or participation depends on factors related to both genotype and the phenotype, or to causes of the phenotype. Conditioning on such selection can induce or distort an association. This is one form of selection bias. Recruitment alone does not prove that bias occurred: a case-control design, for example, can be appropriate when its sampling scheme and analysis match the intended comparison. The risk depends on how inclusion relates to the variables involved, not simply on whether the sample is clinic-based or volunteer-based.

A careful interpretation separates two questions: whether an association is present in the analyzed sample, and whether it generalizes to another target population. The second question needs evidence about the recruitment process and population differences; it does not follow automatically from the first.

Sample Size: Inconsistent Estimates Can Be Noise

A small study may estimate an effect imprecisely even when its design and phenotype match a larger study. Under a simple additive linear model for a quantitative trait, let \(G\) be genotype dosage, coded as 0, 1, or 2 copies of an allele, and let \(\beta\) be the mean trait difference per additional allele. If dosage variance is \(\operatorname{Var}(G)\) and the residual standard deviation is \(\sigma\), an approximate standard error under a simple model is

This approximation illustrates why larger samples usually yield more precise estimates when the phenotype, allele-frequency distribution, and residual variability are otherwise comparable. Under Hardy–Weinberg equilibrium for a biallelic variant with allele frequency \(f\), genotype variance is \(2f(1-f)\). Real analyses may include covariates and other complications, so this formula is a diagnostic illustration rather than a substitute for a design-specific calculation.

A reproducibility assessment should compare estimates and their uncertainty, not just whether each study crosses a decision threshold. A small study can yield an estimate in the same direction but with wide uncertainty; it can also yield an estimate in the opposite direction by chance. Conversely, increasing sample size cannot repair a different phenotype definition, systematic selection bias, or a variant that was not measured comparably.

Ancestry Representation and Variant Comparability

Recruitment plans that substantially underrepresent some ancestry groups restrict the populations in which a study can directly evaluate its findings. “Ancestry” in genetic analyses refers to patterns of genetic similarity and shared population history; it should not be treated as interchangeable with a broad social or racial label. Both genetic ancestry and the social conditions that shape recruitment and health can matter, but they are distinct considerations.

A variant associated with a trait may itself be causal, or it may be a marker correlated with a causal variant through linkage disequilibrium. Allele frequencies and linkage disequilibrium patterns can differ among populations. Consequently, a marker can have different precision or represent a causal signal less well in a population not adequately represented in discovery. A difference in the marker’s association estimate does not automatically imply that the underlying biology differs.

Platform Limitations and an Integrated Design Audit

Genotyping arrays, exome sequencing, whole-genome sequencing, and imputation do not provide identical coverage of genetic variation. A variant may be directly measured in one study, inferred with varying quality in another, or absent from the available data. A nearby marker can serve as a proxy only to the extent that it tracks the relevant signal in that population. Thus, “the same locus” does not always mean that studies tested the same variant or an equally informative proxy.

For example, if a discovery study tests a directly genotyped variant but a replication study lacks that variant and uses a proxy, investigators should document the proxy, its imputation or genotyping quality, and its relationship to the discovery variant in the study population. Low-quality measurement can reduce information and make estimates less stable; systematic measurement problems can also affect validity. The precise consequence depends on the platform and data processing, so it should not be inferred from the word “imputed” alone.

A practical audit proceeds by checking comparability in a deliberate order. This does not replace statistical analysis; it helps identify whether two studies are addressing the same question with comparable information.

This audit matters because “not reproduced” can describe several different situations: a genuinely inconsistent estimate, an imprecise estimate, a changed phenotype, a selected sample, a different marker, or a population in which the discovery marker is less informative. These explanations are not interchangeable, and none should be invoked automatically to excuse a disagreement. The study reports and data should determine which are plausible.

Common Mistake

The reverse mistake is equally important: attributing every discrepancy to design differences without evidence. Reproducibility is strengthened by transparent definitions, representative recruitment, adequate precision for the intended comparison, and comparable genetic measurements. If a study differs from its predecessor, report the difference and assess its consequences rather than assuming that it explains the result.