Why Leakage Changes the Question
A multi-omics classifier is often described by its final model, but its predictions depend on the entire pipeline: data filtering, normalization, imputation, feature selection, integration, tuning, and classification. An evaluation is informative only if each step uses information that would genuinely be available when predicting an unseen case. If information from the evaluation cases enters any fitted step, the reported score may answer an easier question than the intended one.
This problem is called information leakage. It is particularly easy to overlook in high-dimensional biomarker studies: the number of features may greatly exceed the number of subjects, several omics assays may be processed together, and a subject may contribute multiple specimens or visits. A random split of rows does not necessarily separate subjects, and a transformation can leak information even when it does not use outcome labels.
Let subject \(i\) have outcome \(y_i\), and let \(x_{ij}\) denote the value of feature \(j\) for that subject. If subject \(i\) contributes multiple specimens or measurements, denote the repeated record by \(r\), so the data include \(x_{irj}\) and, where appropriate, \(y_{ir}\). The crucial unit of a split is determined by the prediction target. For prediction in new subjects, all records from a subject must stay together.
Where Leakage Enters a Pipeline
Any operation that estimates quantities from a collection of cases can transmit information across a split. Examples include estimating means and standard deviations for scaling, fitting an imputation rule, choosing features, estimating principal components, and learning batch-correction parameters. If those operations are fitted once using the full dataset before the evaluation split, the test data have already influenced the pipeline.
Not every use of test predictors is automatically leakage. A fixed transformation specified in advance, or a per-sample calculation that uses only that sample and would be available at prediction time, may be legitimate. The question is operational: could this step be performed for a genuinely new case without access to other held-out cases or their outcomes? A transductive deployment that intentionally processes an entire incoming batch together can justify access to batch-level predictor information, but that must match the planned use. It does not justify using the batch's outcomes.
Supervised feature selection is a particularly direct risk. If the test labels help choose which genes, proteins, or metabolites appear in the model, then test performance is no longer an independent assessment of that choice. Unsupervised feature selection can also leak test-distribution information if it is fitted across training and test predictors; whether that is acceptable depends on the deployment setting. In an ordinary new-subject evaluation, fit it using training data only.
Worked Example: Scaling Before the Split
Suppose one omics feature has training values \(0\) and \(2\), and a held-out subject has value \(10\). To illustrate the effect of fitting a standardization rule, use the mean as the centering estimate and divide by the training-set standard deviation computed with denominator \(n\). This small example is hypothetical.
Using training data alone, the training mean is \(\bar{x}_{train}=(0+2)/2=1\). The training standard deviation is \(\sqrt{((0-1)^2+(2-1)^2)/2}=1\). The held-out subject is therefore transformed to \((10-1)/1=9\).
If the transformation is instead fitted to all three subjects, the mean is \((0+2+10)/3=4\), and the standard deviation is \(\sqrt{((0-4)^2+(2-4)^2+(10-4)^2)/3}=\sqrt{56/3}\approx4.320\). The held-out value becomes \((10-4)/4.320\approx1.389\).
The test subject has changed the scale used for the training data and for its own transformed value. This example does not prove that the resulting accuracy must increase; leakage can distort performance in either direction. It shows why the transformation must be fitted on training data when the intended task is prediction for an independently arriving subject.
Feature Selection and Repeated Subjects
A clean evaluation protects the test outcomes from every development choice. If a researcher ranks thousands of features by their association with the outcome across the full dataset, retains the top features, and only then cross-validates a classifier, each validation fold's outcomes have already influenced which features are considered. Cross-validation of the classifier cannot undo that information transfer.
Worked Example: A Test Set Used to Choose a Feature
Consider four training subjects with outcomes \(0,0,1,1\), and two test subjects with outcomes \(0,1\). Two candidate features have training values \(0,0,1,1\). In the test set, feature A has values \(0,1\), while feature B has values \(1,0\). A classifier trained on either feature predicts outcome 0 for feature value 0 and outcome 1 for value 1.
If feature A is chosen using the test outcomes, its test predictions are \(0,1\), giving \(2/2=1.000\), or 100% accuracy. Feature B would give \(0/2=0\) correct. Choosing A after inspecting the test labels makes the test set part of model selection; the 100% is not an honest estimate for a prespecified pipeline.
The remedy is to choose the feature using training data only. If feature selection and tuning are repeated during cross-validation, they must be repeated inside each training fold, without using the held-out fold's outcomes. An outer fold then evaluates the complete selection-and-fitting procedure rather than a classifier whose inputs were chosen using the answer key.
Repeated records create a different route for leakage. If a patient's specimens are divided between training and test sets, the model can exploit stable subject-specific patterns, such as a molecular fingerprint, instead of learning patterns that generalize to new patients. The relevant split unit is the subject—not the specimen—when the goal is prediction for new subjects.
Worked Example: Two Measurements From Each Subject
Imagine four hypothetical subjects, A and B with outcome 0 and C and D with outcome 1. Each contributes two measurements. A row-wise split puts the first measurement from every subject in training and the second in testing. Suppose a feature records a stable subject fingerprint, and a fitted nearest-neighbor rule assigns a test measurement the label of the training measurement with the same fingerprint.
Each of the four test measurements matches a training measurement from the same subject. The predictions are therefore \(0,0,1,1\), matching the four outcomes, and the row-wise test accuracy is \(4/4=1.000\). This apparent 100% performance reflects recognition of subjects already represented in training, not prediction for unseen subjects.
For a new-subject target, place both records from each subject in the same partition. For instance, training on A and B and testing on C and D creates no subject overlap; the fingerprint-matching rule can no longer find the same person in training. With only four subjects, this example cannot provide a precise performance estimate, but it correctly illustrates the split unit.
How to Restructure and Audit the Evaluation
The evaluation should reproduce the information flow of the intended prediction task. For a held-out test set, first assign subjects to training and test partitions. Then fit each data-dependent step using training data only, apply the fitted operation to test predictors, and score test predictions only after the pipeline is fixed. If cross-validation is used for development, all preprocessing and feature selection must be repeated within each training fold. When tuning choices are made by cross-validation, an outer evaluation fold should remain untouched by those choices.
State whether the target is a new subject, a new specimen from a known subject, or a future measurement. Specify which predictors and batch information would be available at prediction time.
For new-subject prediction, assign subjects—not rows—to partitions. Keep all of one subject's measurements together.
Estimate imputation, scaling, filtering, feature selection, integration, and tuning using only the training portion relevant to that fit.
Transform test predictors using training-fitted rules, generate predictions, and calculate the prespecified performance summaries. Do not revise the pipeline after seeing test results and continue to call that same set a test set.
A useful justification for keeping feature selection inside training folds follows from how selection uses the outcome. Suppose \(S_j\) is the test score for candidate pipeline \(j\) when that pipeline is fixed without the test outcomes. If the test set is used to choose the pipeline with the largest score, the reported score is \(\max_j S_j\), not the score of a prespecified pipeline. For any fixed candidate \(k\), \(\max_j S_j\geq S_k\) for every possible test sample. Taking expectations gives \(E[\max_j S_j]\geq E[S_k]\). Thus, when candidates have equal true performance, selecting the largest noisy test score tends to produce an optimistic reported score. This argument does not say every contaminated score must be high; it explains why test-based selection invalidates its role as an independent evaluation.
Audit Clues, Alternatives, and Limits
An audit should trace data flow, not just inspect the classifier code. Ask when the split occurred; which observations were used to estimate each transformation; whether outcome labels were used in filtering or ranking; whether any subject appears in more than one partition; and whether the final pipeline was changed after test performance was examined. Check that cross-validation code refits the entire pipeline for each fold rather than passing a once-preprocessed matrix to the folds.
Warning signs include unusually strong performance in a small cohort, near-perfect discrimination after screening many features, very different results under subject-grouped versus row-wise splits, and preprocessing performed on a merged train-test table. These are diagnostic clues, not proof of leakage: small samples can produce unstable results, and legitimate signal can be strong. Conversely, a plausible score does not establish that leakage is absent. Code, split identifiers, and a record of when each choice was made are needed to audit the procedure.
| Approach | What it protects against | Key limitation |
|---|---|---|
| Random row split | Separates individual records | Does not protect against repeated-subject overlap |
| Subject-grouped split | Keeps one subject's records together | Does not by itself prevent preprocessing or feature-selection leakage |
| Nested cross-validation | Separates inner tuning from outer evaluation when the full pipeline is nested | Can be highly variable in small cohorts; does not repair a wrong split unit |
| Independent test set | Evaluates a pipeline fixed before test outcomes are inspected | Becomes a development set if used repeatedly to revise the model |
These approaches address different risks and are not interchangeable. A subject-grouped split can still leak if scaling or feature selection was fitted globally. Nested cross-validation can still be invalid for new-subject prediction if its folds split repeated records from the same subject. An independent test set is not a permanent source of unbiased estimates after repeated model choices are based on its results. Finally, a clean internal evaluation estimates performance under its sampling and measurement conditions; it does not establish performance in a different population or setting.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A researcher standardizes each protein using means and standard deviations computed from the full dataset, then performs five-fold cross-validation. What has gone wrong, and what should change?
- A patient contributes three biopsies, and the goal is prediction for a new patient. What must be kept together during splitting?
- Why is a test score used to choose the best of many feature sets not an independent estimate of performance?
- Would per-sample normalization necessarily be leakage if it uses only measurements from the new sample? State the condition that matters.
Solutions. (1) Each validation fold contributed to the scaling estimates, so its predictors influenced the transformation applied during evaluation. Fit the scaling rule separately within each training fold and apply that fitted rule to the held-out fold. (2) Keep all three biopsies from each patient in a single partition; otherwise the evaluation can reward recognition of a patient already seen in training. (3) The selected feature set is chosen partly because of its test score, so the score contains selection noise; the maximum of candidate scores is at least each individual candidate score, and selection tends to inflate the reported result. Make feature selection inside training data and evaluate the fixed procedure on untouched data or outer folds. (4) Not necessarily. It is appropriate only if the operation uses no information beyond that sample and is available in the intended prediction setting; using other held-out samples or their outcomes would change that conclusion.