Why Audit the Whole Pipeline?
A performance estimate is only informative if the evaluation reproduces the information available when predictions will actually be made. In multi-omics work, a model may combine measurements from several assays, and information can cross the evaluation boundary through more than the final classifier. A duplicated subject, a feature filter fitted using all labels, or an integration choice made after inspecting test results can make evaluation subjects influence the prediction procedure before its score is reported.
A leakage audit therefore follows information through the entire pipeline, from subject identifiers and assay processing to model selection and final scoring. It asks not only whether the test labels entered the classifier, but also whether any decision affecting predictions or the reported estimate used information that would not be available under the intended prediction setting. The earlier material on detecting post-outcome predictors, tuning without optimism, and protecting a held-out test set provides the relevant foundations; this tutorial organizes those concerns into an audit that can be applied to a complete multi-omics analysis.
The audit must begin by stating the prediction task. Specify the prediction unit (for example, a person rather than an assay aliquot), when prediction would occur, what data would then be available, and the population to which performance is meant to apply. These choices determine which information is permissible. For example, access to a patient's baseline transcriptomic assay may be realistic for a baseline prediction task, while a measurement collected after the outcome is known is not. A split that is valid for predicting new samples from already observed subjects may be invalid for predicting entirely new subjects.
A Stepwise Audit
For each pipeline operation, record the data it uses, the time at which it is performed, and whether the operation is repeated using only the training portion of each evaluation split. In the notation below, \(X^{(m)}\) denotes the measurements from omics modality \(m\), and \(Y\) denotes the outcome. The audit does not depend on a particular classifier or performance metric.
State who or what receives a prediction, the prediction time, the intended deployment population, and the measurements available then. Check that outcome definitions and inclusion rules do not use future information.
Link records across modalities using subject, specimen, aliquot, and replicate identifiers. Check whether one person or one biological sample appears on both sides of a split, and whether relatives, sites, or batches require grouping to match the intended generalization target.
For each imputation rule, normalization, scaling, batch adjustment, dimensionality reduction, and feature filter, ask which samples and outcomes were used to estimate it. For ordinary inductive evaluation, learn these quantities from training data and apply the learned operation to held-out data.
Record whether modality inclusion, feature matching, latent representation, weights, or other integration choices were fixed in advance or chosen from evaluation results. Any choice informed by validation outcomes belongs inside the model-selection process, not in a supposedly untouched final test.
Check where hyperparameters and classification thresholds were chosen, who could inspect test outcomes, and whether a test result prompted a change to the pipeline. The earlier material on hyperparameter tuning and test-set protection explains why the final test must not guide these decisions.
Use dated code, split assignments, data-access logs, and analysis records to determine what happened. Classify each issue as documented and harmless under the target, a correctable implementation error, or a breach that makes the reported estimate unsuitable as an untouched evaluation.
This process is an information-flow audit, not simply a search for suspiciously high scores. A low score does not prove that leakage was absent, and a high score alone does not prove leakage. The evidence is in the procedure: what data each operation saw, what was decided after seeing outcomes, and whether that matches the intended prediction setting.
Worked Examples of Audit Decisions
Worked Example: Detecting Subject Overlap Across Modalities
Suppose a hypothetical study has 12 people, each with a DNA-methylation profile and a protein assay. A file contains 24 rows, one for each assay record. The analyst randomly assigns 18 rows to training and 6 to testing, then reports test performance for predicting a person's disease status. An identifier audit finds that 4 of the 6 test rows belong to people who also have a training row.
The fraction of test records with a training record from the same person is \(\frac{4}{6}=0.667\), or 66.7%. This is not evidence that exactly 66.7% of the reported score is inflated; it quantifies the overlap in this audit. Because the stated prediction unit is a new person, the split does not test that target. A model can exploit person-specific signals learned from a person's training assay when predicting that person's held-out assay.
The remedy is to assign all records from each person to the same partition and evaluate on people absent from training. If the intended target were instead a new assay from a known person, that would be a different prediction task and would need to be stated explicitly. Merely removing the identifier column from the model would not remove the overlap in biological information.
Worked Example: Checking a Normalization Step
In a hypothetical single-feature illustration, three training values are 2, 4, and 6 expression units, and a held-out value is 20 expression units. The pipeline standardizes values by subtracting a mean and dividing by a standard deviation. For this example, use the population-form standard deviation solely to make the arithmetic explicit; the audit issue is which observations estimate the transformation.
The training mean is \(\bar{x}_{\mathrm{train}}=(2+4+6)/3=4\) units. The training population-form standard deviation is \(\sqrt{[(2-4)^2+(4-4)^2+(6-4)^2]/3}=\sqrt{8/3}=1.633\) units. Thus the held-out value transformed using the training estimates is \((20-4)/1.633=9.798\).
If the analyst instead estimates the mean using all four values, the mean is \((2+4+6+20)/4=8\) units. That transformation has used a held-out predictor to determine how training values are represented. Under an inductive target in which the held-out sample is not available during model fitting, this is a leakage pathway even though its outcome was not used. Whether such unsupervised access is permissible depends on the deployment protocol; for example, a prespecified transductive task may allow access to a batch of unlabeled future predictors. The analysis must match and disclose that protocol.
Worked Example: Feature Selection Before the Split
In a hypothetical classification analysis, the analyst ranks 5,000 genes by their association with disease status using all 100 subjects, selects the top 50, and then makes an 80-subject training set and a 20-subject test set. A classifier is fitted on the training subjects, and its test AUC is reported as 0.91.
The ranking used the outcomes of the 20 test subjects. Consequently, the test labels influenced which 50 genes entered the classifier, even though the classifier coefficients were estimated using only the 80 training subjects. The test set was therefore not untouched: its labels had already affected the prediction procedure. The AUC of 0.91 cannot be interpreted as an unbiased held-out estimate for this selection procedure.
For a valid split-based evaluation, the gene ranking and selection must be performed using training data only, and then the selected genes and fitted model are applied to the held-out subjects. If the reported result has already guided further choices, simply rerunning the final score on the same test set does not restore its independence. A newly reserved evaluation sample, or an evaluation design that contains all selection steps within its training process, is needed.
Worked Example: Auditing Test-Set Reuse During Model Choice
Suppose an analyst evaluates three candidate integration pipelines on one test set and obtains AUC values of 0.78, 0.81, and 0.76. The analyst chooses the second pipeline because it has the largest test AUC, changes a preprocessing choice, and reports the revised pipeline's test AUC of 0.84.
The arithmetic comparison is \(0.81>0.78\) and \(0.81>0.76\), so the second candidate did have the largest of the first three observed values. But that does not make 0.81 an independent estimate for the selected pipeline: the test outcomes helped select it. The later change and report of 0.84 make the problem clearer, because the same test information has also guided a modification.
The correct audit finding is test-set reuse, not proof that the true AUC is below 0.84 by a known amount. The estimate is no longer a clean final evaluation of the selected procedure. The analyst can report the sequence transparently as exploratory development, then evaluate the finalized pipeline using data not used for these decisions.
Why Selection Creates Optimism
A short argument explains why choosing the best-looking result on evaluation data is hazardous. Suppose \(K\) candidate pipelines have the same true performance \(\theta\), and their observed test scores are \(\theta+\varepsilon_k\), where each error has mean zero. The selected score is the maximum. Since the maximum of numbers is at least their average, taking expectations gives:
The inequality follows pointwise: \(\max_k \varepsilon_k \geq K^{-1}\sum_k \varepsilon_k\). The final equality follows from \(E(\varepsilon_k)=0\) for every candidate. With non-identical observed errors, the same argument still holds under the stated equal-true-performance setup; dependence among errors does not invalidate the inequality. If candidates have different true performances, the selected score's bias is more complicated, but test-guided selection still means the reported test result is not an independent assessment of the selected pipeline.
This derivation does not supply a correction for an individual analysis. The magnitude depends on the number of choices, their dependence, sample size, and score variability. It justifies the audit rule: document selection and keep the final evaluation separate from decisions it is meant to assess.
Diagnostics, Alternatives, and Limits
Useful audit evidence includes a subject-to-sample crosswalk, split files created before modeling, code showing that transformations are fitted within training partitions, and records of when outcomes or test summaries were inspected. Warning signs include identical or near-identical assay records on both sides of a split; a feature list generated from all subjects; normalization or batch correction fitted once before resampling; unexplained changes after test scoring; and reported performance that cannot be reproduced from the documented pipeline. These are prompts to investigate, not automatic proof of a particular amount of bias.
The choice of evaluation design is the main alternative decision, and it should follow the prediction target. A random subject-level holdout can address prediction for new subjects drawn from a similar population. Grouping all records by subject prevents within-person overlap; grouping by cohort or site may be needed when the target is performance in a new cohort or site. Cross-validation can use data efficiently, but every data-dependent operation must be repeated inside the relevant training portions. A final held-out test set can give a separate assessment only if it is not used to choose the pipeline. These options answer different questions; none repairs an outcome definition or information set that is inconsistent with the intended clinical use.
Not every use of all predictors is automatically leakage. For example, a deployment system might receive an unlabeled batch of test predictors before fitting a prespecified normalization, while never receiving their outcomes. That is a transductive protocol and must be evaluated as such. In contrast, a procedure intended to predict one new patient at a time cannot generally use that patient's value to estimate a population transformation unless the deployment protocol explicitly permits it. State the boundary rather than relying on the label “unsupervised.”
| Audit question | Potential concern | Evidence to inspect |
|---|---|---|
| Are the same people represented in training and testing? | Subject-specific information can cross the split. | Identifiers linked across assays, aliquots, replicates, and partitions. |
| Was a transformation learned using held-out samples? | Evaluation information may influence the fitted representation. | Code and stored parameters for normalization, imputation, scaling, and batch adjustment. |
| Did test outcomes influence feature or integration choices? | The test set was used for selection, not just evaluation. | Feature lists, analysis logs, scripts, and dated decisions. |
| Was the test result inspected before the pipeline was finalized? | Repeated use can make the reported score optimistic. | Version history, access records, and a record of model comparisons. |
A leakage audit cannot establish that a model will transport to a different population, that measurements are clinically useful, or that the outcome is measured without error. It also cannot recover a truly independent estimate from a test set whose results have already guided development. In those cases, the appropriate conclusion is limited: identify what can still be learned from the data, describe the breach, and obtain an evaluation set or design suited to the claim being made.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A study splits assay rows at random, and some subjects contribute two rows. What must be checked before interpreting test performance for new subjects?
- Under an inductive prediction target, may a mean used for scaling be estimated from both training and held-out predictors? Explain.
- A feature screen uses all outcomes before cross-validation, while model coefficients are refitted in each fold. Is the evaluation protected from leakage?
- Why does choosing the candidate with the highest test score tend to produce an optimistic selected score when candidates have equal true performance and zero-mean score errors?
- What should an analyst do if test outcomes have already guided a pipeline change?
Solutions. (1) Link all records to subjects and check for subject overlap; if the target is new subjects, assign each person's records to one partition. (2) No: the held-out predictors influence the transformation, contrary to the stated inductive protocol; estimate the mean in training data and apply it to held-out data. (3) No: the held-out outcomes influenced feature selection before the folds were evaluated, so selection must be repeated using each training portion. (4) The maximum is at least the average of the candidate errors, whose expectation is zero, so the expected selected error is nonnegative and is generally positive when candidates' errors vary. (5) Treat the test results as development information, document the reuse, and evaluate the finalized pipeline on data not used for those choices.