Why Replication Criteria Must Be Specified
A gene-level rare-variant finding is a claim about a defined group of variants, a phenotype, and a statistical test in a particular set of participants. Repeating an analysis in another data set is not automatically a replication. If the phenotype, variant set, direction, or test is changed after seeing the new results, an apparently supportive result may answer a different question from the original claim.
A useful replication protocol therefore states in advance what result would count as confirmation. The criteria should preserve the scientific meaning of the discovery while allowing the replication data to provide genuinely new evidence. They also distinguish evidence against the original claim from a study that simply lacks enough information to evaluate it.
What Must Align With the Discovery Claim?
The replication protocol should identify the claim being evaluated, not just the gene name. At minimum, it records the phenotype and its coding, the gene-level variant inclusion rules, the test and covariates, the direction of effect if applicable, and the evidence threshold. These choices should be fixed before the replication results are examined.
Phenotype alignment means that the replication phenotype addresses the same trait and target association as the discovery phenotype. For a case-control study, specify the case definition, control definition, and any ascertainment features that affect interpretation. For a quantitative trait, specify the measurement and scale. A related phenotype can be scientifically informative, but it does not by itself replicate the original phenotype-specific claim.
Variant inclusion alignment means that the rules defining which variants contribute to the gene-level test are stated precisely. For example, a protocol might include qualifying rare protein-truncating variants in a specified gene, using a prespecified frequency threshold and annotation categories. It should also specify the reference gene definition, any quality filters, and the test used to combine variant information. A burden test and a variance-component test can target different patterns of effects, so exchanging one for the other is not automatically an exact replication.
The inclusion rules need not force the replication analysis to use variants that cannot be reliably evaluated in its data. However, any allowed difference must be principled and documented before looking at association results. Otherwise, selecting only the variants that produce the most favorable replication result creates a new analysis rather than a prespecified test.
| Criterion | What to specify | Why it matters |
|---|---|---|
| Phenotype | Trait, coding, case-control or quantitative definition, and relevant ascertainment | Establishes whether the same scientific association is being tested |
| Variant set | Gene definition, frequency threshold, annotation classes, and quality rules | Defines which rare-variant observations contribute to the result |
| Test and model | Gene-level test, effect scale, covariates, and handling of relatedness or structure | Clarifies the estimand and statistical procedure |
| Direction | Expected effect direction, if the test has one | Prevents a result in the opposite direction from being called confirmation |
| Evidence threshold | Significance level and multiplicity rule for the tested candidates | Controls the false-positive rate for the replication decision |
| Samples | Participant overlap rules and any required independent sample split | Protects the replication evidence from discovery-stage reuse |
Worked Example: Writing a Gene-Level Replication Claim
Suppose a hypothetical discovery analysis reports that rare qualifying variants in gene G are associated with disease D in a burden test. The discovery protocol defined qualifying variants as protein-truncating variants with minor-allele frequency below 0.001. Write a replication criterion that preserves the claim.
A suitable criterion would specify the same disease definition and the same gene-level set: qualifying protein-truncating variants in gene G below the stated frequency threshold. It would name the planned burden test, the case-control model and its prespecified covariates, and state whether larger burden is expected to increase or decrease disease odds. It would also set a statistical threshold that accounts for the number of genes being tested in replication.
The criterion should separately identify any permitted deviations. For example, if a variant fails a prespecified quality rule in the replication data, it may be excluded according to that rule. It would not be valid to add a new variant class only after seeing that it strengthens the association. The criterion thus makes clear both what is being replicated and which data-driven changes are not allowed.
Direction of Effect and Statistical Evidence
For a directional burden test, direction is usually expressed on the model’s effect scale. In a case-control logistic model, a positive coefficient for greater qualifying-variant burden corresponds to higher disease odds as burden increases; a negative coefficient corresponds to lower disease odds. The replication protocol must state the expected direction in relation to the phenotype coding. Reversing which outcome is coded as the case reverses the sign, so the coding must be unambiguous.
Not every gene-level test has a single meaningful direction. An omnibus test can detect association under patterns in which some variants increase risk and others decrease it. For such a test, do not invent a single directional criterion for its omnibus statistic. Instead, state what evidence the test evaluates and, if the scientific claim includes a directional interpretation, define separately how the relevant effect estimates will be summarized and assessed.
A replication significance threshold should be chosen before analysis. If one gene is tested at level \(\alpha=0.05\), that is a nominal 5% threshold for that test under its null assumptions. If \(m\) candidate genes are tested and the protocol uses a Bonferroni family-wise threshold, each test uses \(\alpha/m\). Other multiplicity procedures can be prespecified, but the choice should not depend on which genes appear promising in the replication data.
A small p-value is evidence against the null model for the prespecified test; it is not the probability that the null hypothesis is true. Nor does a nonsignificant result establish that the effect is zero. Replication conclusions should consider whether the estimated effect is directionally consistent and whether the data are sufficiently informative to distinguish a meaningful effect from no effect.
Worked Example: Evaluating Direction and a Gene-Level P-Value
In a hypothetical independent case-control replication, 50 of 1,000 cases and 25 of 2,000 controls carry the prespecified qualifying variants in gene G. Assume these counts are from the same variant set and gene-level burden definition used for the discovery claim. Estimate the carrier odds ratio and calculate a two-sided Wald test for no association.
The carrier odds among cases are \(50/950\), and among controls they are \(25/1975\). The estimated odds ratio is
The estimated log odds ratio is \(\log(\widehat{\mathrm{OR}})\approx1.425\). For this two-by-two table, the standard error of the log odds ratio is
Thus the Wald statistic is
The two-sided normal p-value is approximately \(9.3\times10^{-9}\). The estimated direction is positive: in this hypothetical sample, qualifying-variant carriers have higher disease odds. The odds ratio is an estimate of association, not evidence that the variants cause disease. If the discovery claim predicted increased odds and the replication threshold were \(2.5\times10^{-6}\), this result would meet both the direction and statistical-evidence criteria, subject to the model and data assumptions.
The calculation assumes independent case and control observations, correct classification of case status and carrier status, and a valid prespecified gene-level variant definition. In a real design, ancestry, relatedness, and other relevant structure must also be addressed by the sampling and analysis plan; the table alone cannot verify those conditions.
Independence From Discovery
The replication sample must not reuse participants whose outcomes or genotypes contributed to discovery. A participant appearing in both analyses creates dependence between the two results. Even partial overlap can make the replication evidence look more consistent than independent evidence would warrant. The protocol should therefore define how overlap will be prevented or removed, including duplicated records and any participants included through more than one source.
Independence also concerns analysis decisions. The gene, phenotype, variant rules, test, direction, and threshold should not be selected by inspecting the replication results. An independent sample does not rescue an analysis that repeatedly changes its rules until a favorable result appears.
The reason independent replication p-values can be interpreted as new evidence can be stated formally. Let \(D\) denote the discovery data and \(R\) the independent replication data. Suppose that under the replication null hypothesis the prespecified test in \(R\) produces a valid p-value \(P_R\), and that the replication procedure was fixed without using \(R\)'s outcomes. Then for any \(u\) between 0 and 1,
The equality follows because, under the null, the valid replication p-value is uniform (or conservatively distributed) and \(R\) is independent of the discovery data \(D\). Thus selecting a candidate gene in discovery does not by itself make a valid, prespecified replication p-value small under the null. This argument fails if participants overlap, if the replication test is selected after examining its data, or if its p-value is invalid under the model assumptions.
Worked Example: Correcting for Several Replication Candidates
A hypothetical study plans to test three genes selected from an earlier discovery analysis. The replication protocol uses a Bonferroni family-wise level of 0.05. For one gene, the prespecified two-sided test gives an estimated log odds ratio of \(0.180\) with standard error \(0.0720\). The discovery claim predicted a positive effect.
The Bonferroni threshold for each gene is
For the specified gene, the Wald statistic is \(z=0.180/0.0720=2.50\), giving a two-sided p-value of approximately 0.0124. The estimated odds ratio is \(\exp(0.180)\approx1.197\), which is in the predicted positive direction. Because \(0.0124<0.0167\), this result meets the prespecified multiplicity-adjusted evidence criterion as well as the direction criterion.
This conclusion is limited to the defined test and model. It does not show that the other two genes replicated, and it does not establish causation. If the effect estimate had been negative, the p-value might still have met the threshold, but the result would not satisfy this discovery claim’s positive-direction criterion.
Making a Replication Decision
A replication conclusion is strongest when the criteria agree: the phenotype and test address the original claim, the variant set follows the prespecified rules, the direction matches when direction is meaningful, the statistical threshold is met, and the samples are independent. These are separate requirements. A statistically significant p-value cannot compensate for a changed phenotype or a post hoc variant set.
A result that misses the significance threshold is not automatically evidence against the discovery. The estimate may point in the predicted direction but be imprecise, particularly when qualifying variants are rare and carrier counts are small. Conversely, an estimate in the opposite direction may be inconsistent with the original claim even if its p-value is not small. Report the effect estimate, its uncertainty, and the p-value alongside the decision criterion, rather than reducing the conclusion to “replicated” or “not replicated” alone.
Record the phenotype, gene, variant set, test, model, and effect direction claimed at discovery.
Set phenotype alignment, variant inclusion, exclusions, statistical threshold, and multiplicity adjustment before examining replication outcomes.
Prevent participant overlap and avoid using replication outcomes to choose or modify the test.
Check that the analysis was carried out as planned, then report the effect estimate, uncertainty, p-value, direction, and any departures from protocol.
Common Mistakes
- Changing the phenotype and calling it the same replication. A related trait may be useful follow-up evidence, but a different case definition or trait scale can change the claim being tested. State the alignment and describe departures clearly.
- Changing the variant set after seeing results. Adding or removing variant classes because they improve the p-value makes the result data-driven. Apply fixed inclusion rules, and treat a materially changed set as a distinct analysis.
- Calling any significant p-value a replication. A significant result in the opposite direction, or for a different gene-level test, may not support the original claim. Check direction and test alignment as well as statistical evidence.
- Ignoring the number of candidates tested. Applying a 0.05 threshold separately to many genes can inflate the probability of at least one false-positive result. Prespecify a suitable multiplicity procedure.
- Treating nonsignificance as proof of no effect. A large p-value may reflect limited information. Examine the estimate and its uncertainty before deciding what the result can rule out.
- Using overlapping discovery and replication samples. Shared participants undermine independence. Identify and remove overlap or use a genuinely separate sample for the replication test.
Check Your Understanding
Answer each question using the criteria described above.
- A replication analysis changes the case definition but retains the same gene and variant mask. What should be clarified before describing the result as replication?
- Why should variant inclusion rules be specified before examining replication association results?
- A two-sided gene-level test is significant, but its estimated effect direction opposes the prespecified discovery claim. Does it meet a directional replication criterion? Explain.
- Three candidate genes are tested with a Bonferroni family-wise level of 0.05. What per-gene threshold is used?
- How can participant overlap affect the interpretation of a replication p-value?
- Why does failure to meet a replication significance threshold not, by itself, prove that the gene has no effect?