Designing a Confirmatory Genetic Replication

Why a Replication Must Be Designed in Advance

A genetic association result is not confirmed simply because a later analysis produces a small p-value. A confirmatory replication asks whether a specific association can be tested in a new cohort under a plan set before its results are known. The cohort, variant, effect direction, analysis model, decision rule, and power target together determine what evidence the replication can provide.

The plan should make the intended claim testable. If the replication examines a different population, phenotype, variant, or effect scale, it may still provide useful evidence, but it no longer answers precisely the same replication question. A prespecified protocol also limits choices made after seeing results, such as selecting the most favorable variant or changing from a two-sided to a one-sided test.

Definition: A confirmatory genetic replication is a planned test, in a specified replication cohort, of a prespecified genetic association using a defined variant, effect direction, analysis model, and significance criterion. A replication is statistically successful under that plan only if its prespecified decision rule is met. Success is evidence consistent with the association; it does not prove that the association is causal or guarantee that the effect will be identical in every population.

Specify the Replication Question

A useful plan starts with the cohort and the association being tested. State who is eligible, how the phenotype is defined, and whether participants or controls overlap with the discovery analysis. The replication cohort should provide information independent of the discovery result; substantial sample overlap can make the new estimate less independent than the word “replication” suggests. Specify the population to which the result is intended to apply, rather than treating a cohort label as a complete description.

Next name the variant and the allele used to define the effect. For an additive dosage model, let $G_i$ be the number of copies of the prespecified effect allele for participant $i$, typically coded from 0 to 2. Specify which allele is counted and the expected sign of its effect. The coding and identity of the tested variant must be verified before analysis; allele harmonization across association results is a separate technical task.

The estimand—the effect the analysis is intended to describe—should also be stated. For example, a linear model coefficient may represent the mean phenotype difference per additional effect-allele copy, conditional on specified covariates. A logistic model coefficient may represent a change in log odds per copy. These are different effect scales, so “same association” needs a clear meaning when cohorts or outcome models differ.

Choose the Model and Decision Rule

For a quantitative phenotype, a common prespecified model is

Y_i = \beta_0 + \beta_G G_i + \boldsymbol{\gamma}^{T}\mathbf{C}_i + \varepsilon_i,

where $Y_i$ is the phenotype, $G_i$ is effect-allele dosage, $\mathbf{C}_i$ is a vector of covariates, and $\beta_G$ is the association per additional allele copy, conditional on those covariates. The protocol should define the phenotype scale, covariates, sample inclusion and exclusion rules, and treatment of missing data. The model also relies on appropriate outcome and genotype coding, adequate control of relevant confounding and population structure, and standard-error calculations suitable for the data. For a binary phenotype, a logistic regression may instead be specified, with its coefficient interpreted on the log-odds scale.

The null and alternative hypotheses must match the question. If the discovery result motivates a positive effect for the specified allele and that direction is fixed before the replication data are examined, a directional test can be written as follows:

H_0:\beta_G \leq 0 \qquad\text{versus}\qquad H_a:\beta_G > 0.

A two-sided test instead evaluates $H_0:\beta_G=0$ against $H_a:\beta_G\ne 0$. Choose the test and significance level $\alpha$ in advance. A one-sided test is not justified merely because the observed estimate happens to point in the favorable direction. If the estimated effect is in the opposite direction, it does not support the prespecified directional replication claim.

The protocol should say exactly what counts as success: for example, a positive estimated effect and a one-sided p-value below a prespecified $\alpha$. If more than one variant, phenotype, or analysis is eligible to produce the confirmatory claim, define the testing family and the rule for handling multiplicity. A nominal threshold for one test cannot automatically be applied after searching many results and reporting only the most favorable one.

Worked Example: Testing a Prespecified Direction

Suppose a hypothetical replication protocol specifies one variant, counts allele A, and predicts that each additional A allele increases a standardized quantitative phenotype. It specifies an additive linear regression with the same covariates for all participants and a one-sided significance level of $0.05$. In the replication cohort, the estimated coefficient is $0.084$ standard deviations per A allele, with $SE=0.037$.

Check the model and direction. The analysis uses the prespecified dosage and model, and the estimate is positive as predicted. Assume for this illustration that participants are independent, the regression standard error is appropriate, and the normal approximation for the test statistic is adequate.

Calculate the statistic. Under the boundary null value $\beta_G=0$, the test statistic is

z=\frac{\hat{\beta}_G-0}{SE} =\frac{0.084}{0.037} =2.270.

For a one-sided test, $p=1-\Phi(2.270)\approx 0.0116$, where $\Phi$ is the standard normal cumulative distribution function. Since $0.0116<0.05$, the result meets this protocol’s statistical success criterion. It provides evidence in the prespecified positive direction; it does not establish that the allele causes the phenotype.

Power and Sample-Size Planning

Power is the probability that a specified test will reject its null hypothesis when a particular alternative effect is true. It depends on the effect size used for planning, the sample size, the allele frequency, residual variation, the significance threshold, and the analysis. State the target power and the assumptions behind it. A target such as 80% is a design goal, not a guarantee that the study will achieve a significant result.

For a simple additive dosage model with independent participants, Hardy–Weinberg equilibrium, effect-allele frequency $f$, residual standard deviation $\sigma$, and no loss of information from covariate adjustment, genotype dosage has variance $2f(1-f)$. An approximate standard error for the genotype coefficient is therefore

SE(\hat{\beta}_G)\approx \frac{\sigma}{\sqrt{n\,2f(1-f)}}.

For a one-sided test at level $\alpha$, the approximate test statistic under a planned positive effect $\beta_G$ has mean $\beta_G/SE(\hat{\beta}_G)$ and standard deviation 1. The critical value is $z_{1-\alpha}$, so the approximate power is

\text{Power}\approx \Phi\!\left( \frac{\beta_G\sqrt{n\,2f(1-f)}}{\sigma} -z_{1-\alpha} \right).

This approximation makes the design assumptions visible. Relatedness, imputation uncertainty, covariate adjustment, phenotype measurement error, missing genotypes, or a different model can change the effective information. A practical design should account for such features rather than treating the simple formula as an exact sample-size guarantee.

Planning formula: Under the preceding approximation, the sample size for target power $1-\beta$ is $$ n \approx \frac{\left(z_{1-\alpha}+z_{1-\beta}\right)^2\sigma^2} {\beta_G^2\,2f(1-f)}. $$ Here $\beta$ in $1-\beta$ is the type II error probability; $\beta_G$ is the genetic effect coefficient. The two uses have different meanings.

Worked Example: Planning for 80% Power

Consider a hypothetical quantitative-trait replication with a one-sided $\alpha=0.05$, target power $0.80$, effect-allele frequency $f=0.30$, residual standard deviation $\sigma=1$ standard deviation unit, and a planning effect of $0.10$ standard deviations per allele. Assume unrelated participants, Hardy–Weinberg equilibrium, an additive dosage model, and the approximation above.

The dosage variance is $2(0.30)(0.70)=0.42$. Using $z_{0.95}=1.645$ and $z_{0.80}=0.842$, the required sample size is approximately

n \approx \frac{(1.645+0.842)^2(1)^2}{(0.10)^2(0.42)} = \frac{6.185}{0.0042} =1472.6.

Rounding up gives a target of 1,473 analyzable participants. If the investigators expect 10% of recruits to be unusable or missing for the primary analysis, they would recruit at least $\lceil 1473/0.90\rceil=1637$ participants. This inflation addresses the assumed loss; it does not compensate for an effect smaller than the planning value or for a poorly matched model.

How Deviations Change the Claim

Not every departure has the same consequence. A documented change that preserves the original question may be a reasonable implementation decision, but changes that are prompted by observed results weaken the confirmatory interpretation. Record deviations and distinguish the prespecified primary analysis from additional exploratory analyses.

Deviation	Why it matters	Interpretive consequence
Testing a different variant or selecting a proxy after examining results	The tested hypothesis is no longer the single prespecified variant hypothesis	Report the change; treat the result as a different or exploratory test unless the rule was prespecified
Changing the effect direction or test sidedness after seeing the estimate	The direction or rejection rule was selected using the same data	The stated nominal p-value no longer has the planned confirmatory interpretation
Searching many variants or phenotypes and reporting one result	The chance of a small p-value across the search is greater than for one test	Account for the testing family or describe the result as exploratory
Changing cohort eligibility, phenotype definition, or model	The population or estimand may have changed	Interpret the result for the analysis actually performed, not as an exact test of the original plan

Worked Example: Expanding From One Variant to Twenty

Suppose a hypothetical protocol prespecifies one variant and a two-sided $\alpha=0.05$ test. After the primary analysis, an investigator instead tests 20 candidate variants and selects the smallest p-value. The selected result has $p=0.018$.

Compare with the original plan. If $0.018$ had come from the single prespecified test, it would be below $0.05$. But it was selected from 20 tests, so the single-test criterion does not describe the selection procedure.

Illustrate a multiplicity adjustment. A Bonferroni threshold for 20 tests at family-wise level $0.05$ is $0.05/20=0.0025$. Since $0.018>0.0025$, this result does not meet that conservative family-wise criterion. The Bonferroni calculation does not imply that every exploratory analysis must use that exact procedure; it shows why searching and selecting cannot be interpreted as if only one test had been performed. The selected finding is not a confirmatory success under the original single-variant plan.

Common Mistakes and Practical Interpretation

Choosing a one-sided test after seeing the sign. A directional test is defensible when the direction and allele coding are specified before the replication results are examined. Otherwise, use the prespecified two-sided rule or describe the analysis as exploratory.
Calling any significant result a replication. Significance is meaningful relative to a defined hypothesis, model, and decision rule. A result for a different variant, phenotype, or population may answer a related but different question.
Treating power as a probability that the study will succeed. Power is calculated under specified assumptions, including a particular true effect. It is not the probability that the alternative hypothesis is true, and a well-powered study can still produce a nonsignificant result.
Ignoring the effect scale. A coefficient per allele on a standardized phenotype scale is not interchangeable with a log-odds coefficient. Interpret the estimate in the units and model actually prespecified.
Using the discovery estimate uncritically for planning. A power calculation requires a chosen planning effect, but the observed discovery estimate may overstate the underlying effect. State the rationale for the planning value and consider how power changes for smaller effects.

Key takeaway: A strong confirmatory replication claim rests on a question fixed in advance: a defined cohort, variant and effect direction, estimand and model, decision criterion, and power target. The closer the performed analysis remains to that plan, the more directly its result addresses the intended replication question.

Check Your Understanding

Use the definitions and design principles above to answer the following.

What must be fixed in advance to justify a one-sided directional replication test?
In an additive dosage model, how should the genetic coefficient be interpreted for a quantitative phenotype?
Why does a planned power of 80% not mean there is an 80% probability that the genetic association is real?
A study tests 15 variants and reports the one with the smallest p-value. Why is a single-test significance threshold not sufficient to describe that search?
How can a change in phenotype definition or cohort eligibility affect the interpretation of a replication result?