Why Information Leakage Matters
A prediction model is useful only if its validation performance reflects predictions made for people whose outcomes were not used to build or choose the model. In genetic prediction, the score itself may appear to be a fixed weighted sum of genotypes, but its weights and many supporting decisions can depend on phenotype data. If validation outcomes influence any of those decisions, the reported performance can be optimistic even when the final calculation looks routine.
Leakage is not limited to directly fitting a regression on the validation set. It can enter earlier through a discovery GWAS that includes validation participants, a SNP screen run before the data are split, an imputation model that uses the target phenotype, or repeated score tuning on the same folds later used to report performance. The central question is: could information from a validation outcome have affected the predictions or the choices that produced them?
Write an individual's polygenic score as \(S_i=\sum_{j=1}^{m}G_{ij}\hat{\beta}_j\), where \(G_{ij}\) is the allele dosage for variant \(j\) and \(\hat{\beta}_j\) is its weight. A validation outcome can influence the prediction even if it never appears in the final score formula: it may have affected which variants were retained, how missing dosages were imputed, which weights were chosen, or which score specification was reported.
A validation pipeline is therefore more than a final fitted equation. It includes every data-dependent operation used to get to that equation. Operations fixed in advance using an independent external source can be applied to validation data without refitting them there. Operations estimated from the study cohort generally belong inside the training portion of validation.
Where Leakage Enters a Genetic Prediction Pipeline
| Operation | Potential leakage route | Validation-safe principle |
|---|---|---|
| Discovery weights | A GWAS used to obtain weights includes validation participants and their phenotypes | Use independent weights, or estimate weights using training participants only |
| Feature selection | SNPs are selected using phenotype associations in the full data set | Repeat selection using only the training data in each split |
| Imputation or transformation | A fitted operation uses validation phenotypes, or estimates parameters using validation data under an inductive target | Fit the operation on training data and apply its learned rule to validation predictors |
| Score tuning | Several scores are compared on validation outcomes and the best is reported on those same outcomes | Tune within training data and reserve an outer assessment set or fold |
Some operations need careful classification. Applying a fixed allele-harmonization rule or a transformation specified entirely in advance does not, by itself, use validation outcomes. By contrast, estimating allele frequencies, scaling constants, principal-component directions, or missing-value parameters from a cohort uses that cohort's predictor data. For a standard inductive target—predicting a new person without access to a future batch of genotypes—fit such parameters on training data, then apply them to validation genotypes. A deliberately transductive prediction setting may allow access to a batch of unlabeled predictors, but that is a different prediction protocol and must be stated; it still does not license using validation outcomes.
Leakage Through Sample Overlap
Sample overlap is especially important when polygenic-score weights come from a discovery GWAS. If a validation participant also contributed phenotype and genotype data to that GWAS, the participant's outcome can influence the estimated weights. The resulting score is not independent of that person's outcome, even if the validation analysis treats the published weight file as a fixed input.
Worked Example: A Validation Participant in the Discovery GWAS
Consider a deliberately tiny, hypothetical one-variant illustration. Suppose genotypes and outcomes have already been residualized for an intercept and any specified covariates, so the discovery weight is the through-origin regression estimate \(\hat{\beta}=\sum_i x_i y_i/\sum_i x_i^2\). One discovery participant has \(x=1,y=0\). A person reserved for validation has \(x=1,y=1\).
If the validation person is mistakenly included in discovery, the weight is
That person's score is \(S=1(0.5)=0.5\). If the validation person is excluded, the discovery estimate from the remaining participant is \(\hat{\beta}_{\text{independent}}=(1)(0)/1^2=0\), giving that same validation genotype a score of 0. In the overlapping calculation, the validation outcome contributes directly to its own score weight. These numbers are only an illustration of the dependence mechanism, not a plausible GWAS analysis; the essential point is that overlap can make validation performance look better than performance for genuinely unseen participants.
A useful audit asks whether any participant, phenotype record, or family of derived summary statistics used to create the weights also appears in the validation set. A file labelled “external” is not enough to establish independence: its contributing samples and phenotype definitions matter. When overlap cannot be ruled out, the validation result should not be described as an independent estimate of performance.
Phenotype-Informed Preprocessing, Selection, and Imputation
Preprocessing can leak outcomes before the prediction model is fitted. Examples include choosing phenotype-specific quality-control thresholds after examining the full cohort, selecting SNPs by their association with the outcome before cross-validation, or fitting an imputer that uses the outcome to fill in missing predictors. Even when these operations are called “preprocessing,” they are part of model development if their results depend on the phenotype.
The selection problem has a simple justification. Suppose, just for this derivation, that \(m\) candidate SNPs are independent and all have no association with the outcome. Let each validation association statistic \(Z_j\) have a standard normal distribution, and select the SNP with the largest absolute statistic. For any positive cutoff \(c\), the probability that the selected statistic exceeds \(c\) is
where \(\Phi(c)\) is the standard normal cumulative distribution function. For one pre-specified SNP, the corresponding probability is \(2\{1-\Phi(c)\}\); for \(m>1\), the maximum is more likely to exceed the cutoff. Thus, choosing the apparently strongest SNP using validation outcomes and then evaluating that SNP on those same outcomes makes its apparent association too strong on average. Linkage disequilibrium means SNP statistics are not generally independent, but dependence does not make outcome-informed selection safe.
Worked Example: Selecting a SNP on the Validation Outcomes
Suppose a hypothetical validation set has four people, with outcomes \(y=(0,0,1,1)\). Two candidate SNPs have dosages \(g_A=(0,0,2,2)\) and \(g_B=(0,1,1,2)\). A researcher mistakenly screens both SNPs using these validation outcomes and selects the SNP with the largest difference in mean dosage between outcome groups.
For SNP A, the mean dosage is 0 among the two people with \(y=0\), and 2 among the two people with \(y=1\). The difference is \(2-0=2\) dosage units. For SNP B, those means are \((0+1)/2=0.5\) and \((1+2)/2=1.5\), a difference of \(1.5-0.5=1\) dosage unit. The selection rule therefore chooses SNP A.
In fact, for SNP A these four validation records satisfy \(g_A=2y\) exactly. A linear fit using those same outcomes has intercept 0 and slope \(2/4=0.5\) outcome units per dosage unit, so its fitted values are \(0,0,1,1\) and its validation RMSE is 0 outcome units. That apparent perfect performance is not an honest validation result: the outcomes chose the SNP and also supplied the values used to fit and assess the relationship. The remedy is to perform SNP selection and fitting in training data, leaving validation outcomes untouched until assessment.
Imputation can create the same problem less visibly. A genotype imputer based only on training genotypes or a fixed independent reference panel can be part of a valid pipeline. An imputer that uses the target phenotype to infer a missing genotype cannot be used to predict that phenotype: it inserts the answer, directly or indirectly, into a predictor.
Worked Example: A Phenotype-Informed Genotype Imputer
In a hypothetical training set, three observed genotype–outcome pairs are \((G,Y)=(0,0),(1,2),(2,4)\). A researcher fits a linear imputation rule predicting genotype from outcome. The fitted relationship is \(\widehat{G}=0+0.5Y\), since each training value satisfies \(G=Y/2\). The prediction model fitted to the same training data is \(\widehat{Y}=2G\).
Now consider a validation person whose genotype is missing and whose recorded outcome is \(Y=4\). If the imputer is given this validation outcome, it imputes \(\widehat{G}=0.5(4)=2\). The score model then predicts \(\widehat{Y}=2(2)=4\), with an apparent error of \(4-4=0\) outcome units. But \(Y\) is precisely what the prediction is meant to estimate. The imputation has used it to construct the predictor.
If this were an actual prediction made before the outcome was known, that imputation step could not be carried out. A training-only genotype rule would instead have to be applied using information available at prediction time—for example, a rule fitted from training genotypes. The particular safe imputation method depends on the data and prediction setting; the general requirement is that it not use the held-out target outcome.
Keeping Tuning and Preprocessing Inside Validation
Cross-validation is valid only when the held-out fold is excluded from all data-dependent steps that could influence its predictions. This includes operations that are sometimes treated as separate from fitting: SNP screening, imputation-model fitting, scaling, estimated principal components, choice of covariates, selection among score construction methods, and choice of a score threshold. A fixed external resource may be reused, but choices made after examining a fold's outcomes must not be carried into that fold's evaluation.
In nested cross-validation, the outer fold is reserved for estimating performance. Within each outer training portion, inner validation is used to choose the score specification or tuning parameters. The selected pipeline is then refitted using the outer training portion, and applied once to the untouched outer fold. Every operation that learns from data must be repeated inside the appropriate training portion; doing preprocessing once on the full cohort before nested cross-validation defeats the separation.
Worked Example: Choosing a Score Specification on the Same Folds
Suppose a researcher compares two hypothetical score specifications on three validation folds. Specification A has fold-specific \(R^2\) values of \(0.76, 0.68,\) and \(0.70\); specification B has \(0.72, 0.71,\) and \(0.70\). The researcher selects the specification with the larger mean and reports that same mean as its performance.
The means are
The researcher chooses A because \(0.7133>0.7100\). However, these folds have now served both to choose the specification and to assess it. The reported \(0.7133\) does not account for the advantage A received by being selected from the two candidates; it is not an untouched estimate of the chosen pipeline's performance. To assess the selection procedure, use inner folds to choose between A and B, then evaluate the chosen procedure on outer folds not used for that choice. A final independent test set can also be reserved for one assessment after model development is complete.
A Practical Leakage Audit
Before interpreting validation performance, trace the entire route from raw data to predictions. For each operation, identify its inputs, when it was fitted, and whether validation outcomes could influence its parameters or the decision to use it. Then check that the actual validation design matches the intended prediction setting. A clean separation of outcomes does not by itself guarantee that every other aspect of a study is appropriate, but it is necessary for an honest estimate of predictive performance.
Specify what is to be predicted, for whom, and which predictors would actually be available at prediction time. Do not treat a retrospectively recorded validation outcome as an available predictor.
Check whether discovery GWAS weights, phenotype definitions, or other derived inputs include validation participants. If they do, the validation outcomes may already have influenced the score.
Within each split, fit imputation, selection, transformation, and tuning using only that split's training portion. Apply the resulting rules to its held-out predictors without using held-out outcomes.
Use outer folds or an independent holdout to evaluate the complete model-development procedure, rather than reporting the same validation results used to select the winning pipeline.
Common Mistake and Key Takeaway
A common mistake is to say, “I did not fit the final regression on the test set, so the test set was never used.” That reasoning overlooks earlier decisions: the SNPs may have been chosen using test outcomes, imputation may have used the target phenotype, or GWAS weights may contain those participants. The correction is to audit the entire pipeline, not just the last regression. For a standard inductive validation, every data-dependent operation must be learned without the held-out outcomes, and any upstream phenotype-derived weights must also be independent of the validation participants.
Check Your Understanding
For each situation, identify the potential leakage and state a way to prevent it.
- A researcher calculates SNP–phenotype associations in the full cohort, selects 100 SNPs, and then cross-validates a score using those SNPs. Which step leaked information?
- A validation participant contributed to the GWAS that produced the score weights. Why can this invalidate the validation assessment even if the weight file is treated as fixed?
- A genotype imputation model uses the target phenotype for validation participants. Why is that incompatible with predicting that phenotype?
- Two score thresholds are compared on the same validation folds, and the better-performing threshold is reported with its fold-average performance. What assessment design would avoid this selection optimism?
- A genotype transformation is estimated from the complete cohort without using outcomes. Is it automatically safe for an inductive validation target? Explain the distinction between predictor-only fitting and phenotype leakage.