What Counts as Replication?
A pharmacogenomic association concerns a relationship between genetic variation and a drug-related outcome in people exposed to a treatment. A second analysis does not replicate that finding merely because it involves the same gene, drug, or broad clinical domain. The genetic contrast, outcome, treatment context, and statistical claim must be sufficiently aligned to test the original finding.
Replication is a planned attempt to confirm a specified association in new data. It is distinct from repeating the original calculation in the same participants, finding a nearby association after searching many variants, or observing a result that seems biologically plausible. A replication can be informative even when it does not meet a prespecified success criterion: the estimate and its uncertainty still describe what the new study found.
These criteria should be written before examining the replication results. They make explicit what “the same finding” means and reduce the temptation to redefine the allele, outcome, subgroup, or direction after seeing the data. The criteria need not require identical study protocols, but any differences must be assessed for whether they change the question being tested.
Harmonize the Genetic Contrast and Effect Direction
Association estimates are meaningful only in relation to the allele used as the comparison. Reports may code a variant by reference and alternate alleles, by major and minor alleles, or by an explicitly named effect allele. Those labels can differ between datasets. Before comparing estimates, identify the allele whose dosage is modeled and the outcome category or unit to which the reported effect refers.
For a binary outcome, suppose a logistic regression reports an odds ratio for each additional copy of an effect allele. If the replication report instead gives the odds ratio for the other allele at the same biallelic variant, the estimates must be transformed to a common allele coding. Reversing the allele contrast changes the sign of the log odds ratio and takes the reciprocal of the odds ratio. It does not change the underlying association.
Strand orientation also matters. For a single-nucleotide variant, A and T are complements, as are C and G. A report using the opposite DNA strand can therefore label the same allele differently. A/T and C/G variants are especially difficult to align from allele letters alone because their complements are also present in the pair. Allele frequency, genome build, strand conventions, and variant identifiers may help resolve the coding; if the orientation cannot be established reliably, the result should not be called a confirmed replication.
A replication may use a proxy variant rather than the exact discovery variant. A proxy is not automatically equivalent: linkage disequilibrium between the variants can differ across populations, and the proxy may tag the underlying signal less well. State whether the test is an exact-variant replication or a locus-level follow-up, and do not silently treat the two as identical.
Worked Example: Reversing the Reported Allele
In a hypothetical discovery study, the G allele at a biallelic variant is associated with higher odds of an adverse drug reaction. The reported odds ratio per G allele is 1.50. An independent study reports an odds ratio of 0.67 per A allele for the same outcome and model. Determine whether the direction is compatible after allele alignment.
Assumptions for this comparison. The variant is the same, G and A are the two alleles on the same strand, the outcome and exposure definitions are comparable, and both estimates use an additive per-allele logistic model. The reported estimates are hypothetical.
The discovery log odds ratio for G is \(\log(1.50)=0.4055\), which is positive. The replication reports the opposite allele, so convert its estimate to the G-allele scale:
The aligned replication estimate is greater than 1, and its log odds ratio is positive, matching the prespecified direction. The unaligned value, 0.67, is less than 1 only because it describes A rather than G. Directional agreement alone does not establish successful replication: the study must also satisfy the phenotype, exposure, independence, and statistical criteria.
Harmonize the Phenotype and Treatment Context
A drug-response phenotype is not just a label such as “toxicity,” “response,” or “dose.” It includes how the outcome is defined, when it is assessed, how it is measured, and which participants are eligible to have it. For example, an adjudicated severe adverse reaction is not automatically interchangeable with any recorded adverse event. A laboratory threshold at a fixed time is not necessarily the same endpoint as a change from baseline over a variable follow-up period.
Compare the discovery and replication phenotype definitions before comparing effect estimates. Check the event criteria, measurement scale, assessment schedule, follow-up window, handling of treatment discontinuation, and rules for missing or competing outcomes. The earlier material on pharmacogenomic phenotype adjudication is relevant here: reproducible measurement is part of the claim, not a detail to resolve after analysis.
Treatment exposure also determines the association being tested. Record the drug and formulation, indication, starting and titrated doses, duration, adherence, co-medications, and relevant treatment changes. A genotype association with response under one drug or regimen may not be the same association under another regimen. Likewise, a study of treated patients cannot be treated as a direct replication of a study whose phenotype includes people who did not receive the drug.
Comparability does not mean that every protocol detail must match. Instead, ask whether the difference could change the genetic effect or its interpretation. A substantially different dose range, exposure duration, adherence pattern, or co-medication profile may mean that the replication tests a modified question. A simple conversion of dose units cannot establish equivalent exposure if the treatment schedules or clinical circumstances differ.
Worked Example: A Similar Label but a Different Outcome
A hypothetical discovery study defines the outcome as a dose reduction due to an adjudicated drug-related adverse reaction during the first 30 treatment days. Among 100 exposed patients, 12 meet this definition. A second study reports “any adverse event” during 90 days; 9 of its 100 participants have such an event. Decide whether the second outcome directly replicates the first, and calculate the reported proportions.
For each study, the observed event proportion is the number meeting that study’s definition divided by the number assessed:
Thus, the reported proportions are 12% and 9%, respectively. These are descriptive proportions, not genetic association estimates. More importantly, the second study uses a broader event definition and a longer observation window; the numbers cannot be compared as though they measured the same endpoint. Before treating it as a replication, the investigators would need a harmonized analysis using the discovery outcome definition and a compatible assessment window, if the necessary data were collected. If that is not possible, the second study may provide related evidence, but it does not directly test the same phenotype target.
Prespecify Direction and Statistical Criteria
The expected effect direction must be expressed using the same allele coding and outcome coding that will be used in the replication analysis. For example, “the G allele increases risk” is incomplete unless “risk” has a defined endpoint and G has been unambiguously aligned. If the discovery finding predicts a lower response measure, define which allele is expected to have the lower value.
A directional test can be appropriate when the direction is specified before the replication data are examined and an effect in the opposite direction would not count as confirmation of the target claim. For a model coefficient \(\beta\), suppose the target direction is positive and the null is \(H_0:\beta=0\), with \(H_a:\beta>0\). Under the null and the model’s large-sample approximation, the Wald statistic is \(Z=\hat{\beta}/SE\). A positive, sufficiently large \(Z\) provides evidence in the prespecified direction. If the target direction is negative, the sign is reversed in the alternative and the evidence is evaluated in that direction instead.
The significance level and other success criteria should also be prespecified. A study might require a one-sided \(p\)-value below a chosen \(\alpha\), a minimum precision or sample size, and an estimate in the target direction. Some replication plans also set a clinically or scientifically meaningful effect threshold. There is no universal rule that every replication must use one particular threshold; the choice should match the claim, design, and intended strength of evidence.
Worked Example: Testing a Prespecified Direction
In a hypothetical independent replication, the prespecified target is a positive log odds ratio for the G allele on a harmonized adverse-reaction outcome. The planned test uses \(H_0:\beta=0\) versus \(H_a:\beta>0\), with one-sided \(\alpha=0.025\). The replication estimate is \(\hat{\beta}=0.30\), with \(SE=0.12\). Assuming the logistic model is correctly specified, observations are independent, allele coding and phenotype are harmonized, and the Wald normal approximation is adequate, assess the directional test.
The Wald statistic is the estimated coefficient divided by its standard error:
For a standard normal statistic, the one-sided tail probability is \(P(Z\geq 2.50)\approx 0.0062\). Because \(0.0062<0.025\) and the estimate is positive, the result meets the stated directional significance criterion. The estimated odds ratio per G allele is:
The estimate therefore corresponds to about 1.35 times the odds of the defined adverse reaction per additional G allele, under this model and treatment context. This is evidence in the prespecified direction, not proof that the genotype causes the reaction. It counts as a successful replication only if the study also satisfies the other prespecified requirements, including independence and comparable exposure and phenotype definitions.
Independent Confirmation and a Practical Checklist
Independent confirmation requires new information, not merely a new analysis. In the strongest form, the replication uses participants who did not contribute to discovery, with no duplicated individuals across datasets. Reusing discovery participants can make estimates and tests dependent and can exaggerate the appearance of confirmation. If complete separation is impossible, overlap must be identified and addressed rather than described as full independence.
Independence also has an analytic dimension. The replication hypothesis, allele coding, phenotype definition, direction, model, covariates, and success threshold should be fixed before the replication outcomes are inspected. This reduces selective reporting, such as trying several outcomes and highlighting only the one that agrees. A prespecified analysis does not guarantee a correct result, but it makes the evidence easier to evaluate.
A careful replication decision can be organized as a sequence of checks:
Name the variant or allele contrast, treatment exposure, phenotype, measurement window, model, and expected direction.
Resolve allele and strand coding; harmonize outcome definitions, units, timing, and the treatment context as far as possible.
Check participant overlap and establish that the replication test was not selected after inspecting its results.
Evaluate direction, statistical threshold, precision, and any prespecified effect-size requirement; report the estimate and uncertainty.
Common Mistake: Calling Any Similar Result a Replication
A common error is to reason, “The second study used the same gene and drug and found a significant association, so the finding replicated.” That reasoning skips the central questions: Was the same allele contrast tested? Was the same phenotype measured over a comparable period? Were participants exposed to a sufficiently comparable treatment? Was the direction specified in advance? Were the data independent of discovery?
The correction is to treat replication as a prespecified comparison of a defined association, not as a search for any result that resembles the original story. A positive result for a different endpoint, an unaligned allele, or a selected subgroup may motivate further research, but it does not automatically confirm the original claim. Nor does a non-significant result alone settle the matter: its estimate, uncertainty, power, and comparability determine what can reasonably be concluded.
Check Your Understanding
Use the criteria in this tutorial to answer each question.
- A discovery estimate is reported per C allele, but the replication reports the reciprocal odds ratio per G allele at the same variant. What transformation is needed before comparing directions?
- Why might “any adverse event within 90 days” fail to replicate an adjudicated drug-related event within 30 days?
- Name at least four exposure features to compare between the discovery and replication studies.
- When is a one-sided directional test defensible, and why should its direction be specified before examining replication results?
- Why does a second analysis of the discovery participants not provide fully independent confirmation?
- A replication estimate has the expected sign but does not meet its prespecified significance threshold. What should be reported, and what conclusion should be avoided?