Why Leakage Changes the Question
A prediction result is useful only if it estimates performance for the intended use. If a model is meant to predict a patient’s outcome at diagnosis, a feature recorded after treatment or after the outcome cannot support that prediction—even if it is strongly associated with the outcome in the data. Likewise, if information from evaluation observations influences model development, the reported score no longer provides a clean assessment of predictions made without that information.
In multi-omics studies, information can enter through many routes: repeated specimens from the same person, labels embedded in clinical records, preprocessing that uses evaluation observations, selection of features using all outcomes, joint construction of molecular representations, or repeated model comparisons on the same test set. The particular route matters because leakage is not one statistical procedure to avoid; it is a mismatch between the information available during evaluation and the information that would be available at prediction time.
Let \(D_{\mathrm{train}}\) denote the data used to develop a prediction rule \(f\), and let \(D_{\mathrm{eval}}\) denote observations used to assess its predictions. A clean evaluation requires that the rule and all choices used to produce it be based only on information permitted by the prediction task and the designated training data. The evaluation observations must represent the intended prediction setting, and their outcomes must not guide fitting or selection. These conditions are not enough by themselves to guarantee transportability to every population, but violations can make a reported score optimistic or otherwise irrelevant.
Before analyzing any feature, specify the prediction target and the time at which the prediction is supposed to be made. Also specify the population and unit to which the prediction will apply. The earlier material on prediction targets and deployment populations is relevant here: the same recorded variable can be legitimate for one prediction time and impermissible for another.
Common Pathways
| Pathway | How information crosses the boundary | Typical warning sign |
|---|---|---|
| Subjects or specimens | Related observations from one subject appear in both development and evaluation data, although deployment concerns new subjects. | Performance drops when assessed on entirely new people or cohorts. |
| Outcomes | A predictor records, encodes, or is caused by an event that occurs after the prediction time or reveals the target directly. | A very strong predictor is created by a care process or record updated after the outcome. |
| Preprocessing | Parameters of a transformation are estimated using evaluation observations, changing the model inputs based on data it should not have seen. | Normalization, imputation, or batch handling uses all observations without considering the evaluation boundary. |
| Feature selection | Evaluation outcomes influence which genes, proteins, metabolites, or other predictors are retained. | Features are selected once using the full labelled data before performance assessment. |
| Integration | Evaluation observations or outcomes influence shared features, aligned representations, or fitted latent components used for prediction. | Joint analysis is performed before separating development from evaluation data, with no justification tied to deployment. |
| Repeated evaluation | Many modelling choices are compared against the same evaluation outcomes, and the best-looking result is reported. | The evaluation set has become an informal tuning resource. |
A data-dependent operation is not automatically leakage in every setting. For example, a transformation estimated from an incoming batch may be legitimate if the real deployment process receives that entire batch before predicting any member of it. It is not legitimate to use that operation to claim performance for one-at-a-time predictions if later observations would not yet be available. The relevant question is always whether the information flow matches the stated use.
Worked Examples
Worked Example: Repeated Specimens From the Same Patients
Suppose a hypothetical study has 40 patients with a binary disease label and two tissue specimens per patient. The intended use is prediction for new patients. The 80 specimen rows are randomly divided, and a classifier is evaluated on 20 test specimens. Eighteen are classified correctly.
The reported specimen-level accuracy is
This is 90% accuracy for the sampled test specimens, but it may not estimate accuracy for new patients. With two specimens per patient, random row allocation can place one specimen from a patient in training and the other in evaluation. If the classifier uses patient-specific molecular signatures, the test specimen is no longer independent of the training information in the way a new patient would be.
For a hypothetical deployment-relevant assessment on 20 specimens from patients not represented in development, suppose 11 are correct. Then \(11/20=0.55\), or 55% accuracy. The two scores answer different questions: performance on additional specimens from a partly familiar set of patients versus performance on specimens from new patients. The arithmetic does not prove that subject overlap caused the entire difference; it shows why the first score cannot, by itself, support the new-patient claim.
Worked Example: A Predictor Recorded After the Outcome
Consider a hypothetical task that predicts by day 0 whether a patient will have remission by day 30. A database also contains a field, “remission documented by day 30,” entered after follow-up. In a sample of 40 patients, this field matches the day-30 outcome for all 40.
Using the field as a predictor gives
The computed value is 100% accuracy in these records. It is not evidence that a day-0 model can predict remission perfectly: the predictor is itself a later record of the outcome. At day 0 it is unavailable. A legitimate day-0 feature set must be restricted to information available by that time, and performance must be assessed for that defined prediction task. A variable can be recorded as a column in a dataset yet still be invalid for a particular prediction target.
Worked Example: Selecting Features Before Evaluation
Suppose 100 hypothetical patients have random binary outcomes and 1,000 candidate molecular features. An analyst calculates each feature’s association with the outcome using all 100 patients, retains the feature with the largest observed association, and only then divides the patients into training and test groups. In the resulting test set of 20 patients, the selected feature agrees with the outcome for 16.
The apparent accuracy is
Although the test labels were not directly used to fit the final classifier, they influenced which feature was chosen: all 100 outcomes were used during selection. Thus this test-set accuracy is not a clean assessment. Searching many noise features and retaining the most outcome-associated one creates a route for chance patterns in the evaluation patients to affect the reported score.
As a scale illustration, if the standardized association scores for null features were approximately independent \(N(0,1)\) variables, the largest absolute score among \(p=1{,}000\) features is roughly \(\sqrt{2\log(2p)}=\sqrt{15.20}\approx3.90\). This approximation is not an exact description of correlated omics features, but it shows why searching many candidates can yield an extreme association even when the features carry no genuine signal. Selection must be treated as part of model development, not as a harmless preliminary step.
Worked Example: Global Standardization and the Deployment Setting
Suppose one molecular feature has training values 0 and 2, and two evaluation values 100 and 102. For illustration, use the population standard deviation formula for the displayed values. If the mean and standard deviation are calculated from the training observations alone, the mean is \((0+2)/2=1\), and the standard deviation is \(\sqrt{[(0-1)^2+(2-1)^2]/2}=1\). The evaluation values, transformed using those training quantities, are 99 and 101.
If all four observations are used to estimate the transformation, the overall mean is \((0+2+100+102)/4=51\), and the population standard deviation is \(\sqrt{[(0-51)^2+(2-51)^2+(100-51)^2+(102-51)^2]/4}=\sqrt{2501}\approx50.01\). The transformed training values are approximately \(-1.020\) and \(-0.980\), while the transformed evaluation values are approximately \(0.980\) and \(1.020\).
The two transformations differ because the evaluation feature values influenced the global mean and standard deviation. Whether that is leakage depends on the task. If predictions are for individuals arriving one at a time and the transformation is meant to be learned from development data, using the evaluation values contaminates the assessment. If deployment explicitly permits estimating a transformation from a complete incoming batch before prediction, that batch-level operation may match the task—but the evaluation must then represent that same setting.
Why Repeated Evaluation Inflates a Winning Score
Even when no individual evaluation outcome is copied into a feature, repeated comparisons can turn a test set into a source of model-development information. Choosing a model because it scored highest on that set selects not only for genuine performance but also for favorable random error.
A short derivation makes the point. Suppose two candidate models have the same true score \(\theta\), and their estimated scores are \(\theta+\epsilon_1\) and \(\theta+\epsilon_2\), where the errors are independent \(N(0,\sigma^2)\). The reported winner has score \(\theta+\max(\epsilon_1,\epsilon_2)\). Since
and \(\epsilon_1-\epsilon_2\sim N(0,2\sigma^2)\), its expected absolute value is \(2\sigma/\sqrt{\pi}\). Therefore,
So the selected score is optimistic on average, even though both candidates have the same true score. For example, if the standard deviation of the score error is \(0.03\), the expected winner’s noise contribution is \(0.03/\sqrt{\pi}\approx0.0169\), or about 1.69 percentage points for an accuracy-scale metric. This calculation assumes two independent, normally distributed errors; the precise amount changes with the number of models, dependence among scores, and evaluation design. The direction of the selection effect remains important.
Detection, Alternatives, and Limits
Leakage diagnosis is primarily an audit of information flow. For each proposed predictor and analysis operation, ask what data it uses, when those data become available, and whether they would be available for the intended prediction. Trace the same questions through the whole workflow: normalization, missing-value handling, feature selection, integration, model fitting, tuning, and metric selection. A surprisingly strong result is a reason to inspect this chain, not proof of leakage on its own.
Useful checks include documenting the prediction time and the timestamp or provenance of candidate features; checking whether patient, specimen, cohort, or batch identifiers connect development and evaluation observations; and comparing performance under evaluation designs that better match the deployment claim. Large drops under a more realistic design can reveal a mismatch in the original estimate, although they do not identify a single cause without further investigation. Implausibly strong individual predictors, abrupt performance gains after adding a data-processing step, and a large gap between internal and external performance also merit review.
The main alternatives are not interchangeable. A single held-out test set can support a final assessment if it is kept separate from development and matches the target setting. Cross-validation can use data more efficiently for internal assessment, but its usefulness depends on keeping the entire development process appropriately separate from each evaluation portion. External or temporal evaluation can test a different, often more deployment-relevant, shift in setting. No evaluation method repairs a target mismatch, an unavailable predictor, or an unrealistic deployment population.
This tutorial identifies the pathways rather than prescribing the detailed design of every resampling or preprocessing procedure. In particular, the correct split unit and how to perform specific operations within resampling require decisions that depend on the data structure and prediction task. Nor does the absence of an obvious leakage route guarantee valid prediction: small samples, population shift, measurement error, and model misspecification can still limit performance.
Common Mistakes
- Wrong reasoning: “The evaluation labels were never used to fit the final model, so the evaluation is independent.” Correction: Labels can still leak through feature screening, model choice, threshold choice, or repeated comparison. Audit every decision informed by evaluation results.
- Wrong reasoning: “All omics variables are measured at baseline, so they are valid predictors.” Correction: Check when each measurement is available and whether it encodes later care, documentation, or the outcome itself.
- Wrong reasoning: “Randomly assigning rows makes the test set independent.” Correction: Rows may share a subject, specimen, or other source of information. Independence must be considered at the unit relevant to the intended prediction.
- Wrong reasoning: “Using evaluation covariates is always leakage,” or, conversely, “using them is always harmless because their labels were hidden.” Correction: Unlabelled evaluation covariates can still influence preprocessing or integration. Whether their use is permissible depends on what is available at deployment and what claim the evaluation supports.
- Wrong reasoning: “A high test score proves there is no leakage.” Correction: A high score may reflect genuine signal, leakage, chance, or a mixture. Performance alone cannot establish the information flow that produced it.
Check Your Understanding
For each question, first identify the information path, then compare it with the stated prediction task. Solutions follow each question.
- A model predicts a new patient’s treatment response, but a molecular feature is measured after the treatment response is recorded. What is the main concern? Solution: The feature is unavailable at the intended prediction time and may be a consequence or record of the outcome. It is outcome leakage for this prediction task.
- An analyst selects 25 genes by their association with outcome using the entire labelled dataset, then evaluates a model on a subset of those same patients. Why is the score not a clean test estimate? Solution: Evaluation outcomes influenced feature selection. The subset was used indirectly during development, even if it was excluded from the final model fit.
- Two specimens from each patient are randomly assigned to training and evaluation rows. The stated goal is prediction for patients not seen during development. What should be checked? Solution: Check whether specimens from the same patient occur on both sides. Such overlap can let subject-specific patterns help evaluation predictions, so row-level separation may not match the new-patient target.
- A researcher evaluates 30 models on one test set and reports the model with the highest score. Explain the likely direction of selection bias. Solution: The selected model tends to have a favorable evaluation error in addition to any real advantage. Therefore its reported winning score is likely optimistic on average; the size depends on score variability and model dependence.
- A pipeline estimates normalization parameters using an entire incoming cohort before predicting any member of that cohort. Is this necessarily leakage? Solution: No. If deployment truly provides the complete cohort before any prediction and the evaluation reproduces that process, use of cohort-wide covariates may be compatible with the target. It would not justify a one-at-a-time prediction claim if future observations would then be unavailable.