Why Prespecification Matters in Replication
A replication study is informative when its analysis addresses a clearly defined question using data that were not used to choose the result being tested. Before examining replication outcomes, investigators should therefore identify the genetic variant, allele, phenotype, population, covariates and analysis decisions that define the primary test. This is more than administrative detail: it determines what result counts as a replication and makes it possible to distinguish a confirmatory test from a promising follow-up observation.
An association study can generate many plausible analysis paths. Researchers might test several nearby variants, choose among related phenotypes, add or remove covariates, or try alternative transformations. If they report only the most favorable result, its nominal p-value no longer describes the full selection process. Prespecifying the primary analysis limits this freedom. It does not guarantee that the study is unbiased or that the selected test is correct, but it makes the intended test auditable and reduces opportunities for outcome-driven choices.
What the Plan Must Identify
A useful plan is precise enough that another analyst could identify the same test without consulting the results. For the genetic target, specify the variant with a stable identifier and genomic position on a named genome build. Record the reference and alternate alleles and state which allele is counted as the effect allele. For a multi-variant test, define the variant set and its inclusion rules rather than naming only a gene. A nearby marker or a newly selected proxy is not automatically the same target as the originally specified variant.
Define the phenotype operationally: what is measured, how it is coded, the time point or assessment window, and the scale used in the model. “Disease status” may be insufficient if the replication cohort has multiple diagnostic definitions or follow-up windows. For a quantitative outcome, say whether the analysis uses the original scale, a transformation, or a standardized measure. For a binary outcome, define which category is coded as the event.
Also record the target population and the analysis sample rules. Specify relevant ancestry or cohort restrictions, participant exclusions, genotype and phenotype quality-control rules, and how missing genotype, phenotype or covariate values will be handled. State the covariates and their coding. Covariates should follow the scientific design and estimand, not be added or removed just because doing so changes the replication p-value.
Finally, state the model, test statistic or software implementation, the primary significance criterion, and the family of tests to which that criterion applies. If more than one variant or phenotype is confirmatory, the plan must say how multiplicity is addressed. Additional models can be useful sensitivity analyses, but their status and interpretation should be clear in advance.
Worked Example: Turning a Vague Replication Question Into a Testable Plan
Suppose a discovery analysis reported an association between a common variant and a standardized blood-pressure measure. Before opening the replication outcomes, the investigators must turn “replicate the blood-pressure association” into an operational specification.
They define the target as variant rsExample at its position on genome build GRCh38, with the alternate allele T counted as the effect allele. The replication phenotype is systolic blood pressure measured at the baseline visit, analyzed in millimetres of mercury, with the outcome definition and measurement protocol fixed before analysis. The target population is the prespecified ancestry group in the independent cohort. Exclusions are limited to documented sample and measurement quality-control rules; genotype, phenotype and covariate missingness will be handled by complete-case analysis.
The primary model is a linear regression of systolic blood pressure on the dosage of T (0, 1 or 2), age, sex and the cohort’s prespecified ancestry principal components. The coefficient for dosage is the estimated mean difference in blood pressure, in mmHg, per additional T allele, conditional on the listed covariates. The plan identifies this one variant-phenotype pair as the primary test and gives the decision criterion. Any alternative phenotype definition, added covariate set or nearby variant is labelled secondary or exploratory.
This plan fixes the scientific target and the choices most capable of changing the result. It does not assume that the model is automatically adequate: the investigators still need to check data quality and model diagnostics, and they must report any departures from the plan.
Prespecification and the Statistical Model
For a quantitative phenotype, a common primary model is
where \(Y_i\) is the specified phenotype for participant \(i\), \(G_i\) is the dosage of the prespecified effect allele, \(\mathbf{C}_i\) is the vector of prespecified covariates, and \(\beta_G\) is the conditional mean difference in outcome per additional effect allele. For ordinary least-squares inference, the model requires independent observations, an adequately specified conditional mean and suitable error assumptions for the chosen standard errors and p-values. Normal errors support exact small-sample t inference under the usual fixed-design assumptions; with large samples, appropriate robust or model-based procedures may use asymptotic approximations. The plan should name the intended inference procedure rather than treating the regression formula alone as sufficient.
The key prespecification issue is not that one model is always best. It is that the primary model and its covariates are chosen for scientific reasons before replication outcomes are known. If the phenotype is binary, a logistic model may be appropriate, but its coefficient is on the log-odds scale and its assumptions and interpretation differ. The model must match the outcome and estimand that were specified.
Worked Example: Interpreting a Prespecified Association Test
In a hypothetical replication analysis, the prespecified primary model estimates an allele effect of \(\hat{\beta}_G=0.10\) standard-deviation units per effect allele, with \(SE=0.045\) standard-deviation units. Assume the model and large-sample normal approximation are appropriate, the test is two-sided, and the null hypothesis is \(H_0:\beta_G=0\).
The Wald statistic is
The two-sided normal-approximation p-value is
At a prespecified \(\alpha=0.05\) for this single primary test, the result meets the stated statistical criterion. The estimated association is an increase of 0.10 phenotype standard deviations per additional effect allele, conditional on the listed covariates. The p-value is the probability, under the zero-effect null and the model assumptions, of obtaining a test statistic at least this extreme; it is not the probability that the null hypothesis is true. Nor does this calculation alone establish that every other analysis choice was appropriate.
How Selection Changes What a P-Value Says
Prespecification matters because a reported test may be the winner among many tests rather than the sole test that was planned. A simple calculation shows the issue. Suppose \(m\) valid p-values are independent and each follows a uniform distribution on \([0,1]\) when its null hypothesis is true. For one test, the probability of a p-value below \(\alpha\) is \(\alpha\). The probability that all \(m\) p-values are at least \(\alpha\) is \((1-\alpha)^m\). Therefore, the probability of at least one nominally significant result is
This derivation relies on independence and the complete null; correlated tests require a different exact calculation. The union bound gives the generally valid inequality \(P(\text{at least one false positive})\leq m\alpha\), although that bound can be conservative. The important point is that a nominal threshold applied to a selected winner does not have the same error rate as the same threshold applied to one prespecified test.
Worked Example: Choosing the Most Favorable Phenotype
Imagine investigators examine 20 independent phenotype definitions, all null, and report only the smallest p-value. At \(\alpha=0.05\), the probability of at least one p-value below 0.05 is
Thus, under the stated independence and complete-null assumptions, there is about a 64.15% chance of seeing at least one nominal p-value below 0.05 across the 20 tests. This does not mean any particular result has a 64.15% probability of being false. It means that reporting the smallest p-value without accounting for the 20-way search does not preserve a 5% family-wise false-positive rate.
A prespecified single primary phenotype avoids choosing the primary result from those 20 outcomes. If multiple tests are genuinely confirmatory, the analysis plan must define their family and an appropriate multiplicity procedure. The other phenotype analyses can still be reported as exploratory, with their number and selection process made transparent.
Diagnostics, Deviations, and Alternative Approaches
Prespecification does not replace diagnostics. Before interpreting a test, check variant identity and allele dosage, sample counts, phenotype distributions, missingness, covariate coding, convergence where relevant, and whether the model’s assumptions are plausible. For a linear model, inspect residual behavior and influential observations; for a binary model, check convergence and sparse-data problems. If a diagnostic reveals a serious problem, document it and explain the response rather than quietly changing the primary analysis and presenting the revised result as if it had been planned.
The main alternatives are not simply “prespecify” versus “do not analyze.” A locked confirmatory analysis prioritizes a defined hypothesis and controlled interpretation. An exploratory analysis can search broadly for useful patterns, but its findings need appropriate caution and subsequent validation. A protocol may also allow a prespecified adaptation, such as a fallback method if a stated model fails; the trigger and fallback should be defined before outcomes are examined. Adaptive flexibility without a defined rule can turn post hoc judgment into hidden selection.
A common mistake is to treat a result as confirmatory because the variant was selected in a discovery cohort, even though the replication phenotype, covariates or variant proxy were chosen after examining replication outcomes. The correction is to distinguish the independent data used for discovery from the decisions made using replication data. Independent replication data help protect the test from discovery-stage selection, but they do not protect against selection performed within the replication cohort itself.
Another limitation is that prespecification cannot correct a poorly chosen phenotype, an invalid model, population structure, measurement error, or a mismatch between the planned target and the available cohort. It also does not require suppressing scientifically useful follow-up analyses. The safeguard is transparent labeling: report the primary result, disclose planned and unplanned analyses, explain deviations, and avoid presenting an exploratory winner as if it were the sole planned test.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A replication team names a gene but chooses the most significant variant in that gene after examining replication results. Is that variant-level result confirmatory? Explain.
- Why should a plan identify the effect allele as well as a variant identifier?
- For 10 independent null tests at \(\alpha=0.05\), calculate the probability of at least one p-value below 0.05.
- A planned covariate is unavailable for many participants. What should the investigators do to preserve transparent interpretation?
- Why does an independent replication cohort not, by itself, prevent selective reporting?
Solutions. (1) No. Choosing the winning variant using replication outcomes makes the reported variant selected; it is exploratory unless the selection procedure and its multiplicity handling were prespecified. (2) The allele defines the dosage and direction of the estimated effect, so the variant identifier alone does not specify the tested contrast. (3) \(1-(1-0.05)^{10}=1-0.95^{10}\approx 1-0.5987=0.4013\), or about 40.13%, under independence and the complete null. (4) Report the missingness and its consequences, follow any prespecified fallback rule, or transparently label a justified change as a deviation; do not silently redefine the primary analysis. (5) Researchers can still search across replication phenotypes, variants, covariate sets or models and report only favorable results. Independence from discovery does not remove selection based on the replication data themselves.