Why an Information Audit Matters
A genomic prediction result is only meaningful if its evaluation matches the intended use. If the goal is to predict a diagnosis for a new person, the evaluation must represent prediction for people whose outcome and sample information were unavailable while the model was developed. A workflow can appear to separate training and validation samples while still allowing information to cross that boundary through preprocessing, feature selection, repeated samples, or analyst decisions.
The central audit question is not merely, “Were the rows split?” It is: Could any information unavailable at the intended prediction time have influenced the fitted model, the selected features, the chosen analysis, or the reported score? Leakage can be direct, as when a predictor encodes the outcome, or indirect, as when validation results are repeatedly used to refine a workflow.
First state the prediction target and the information boundary. Specify what counts as a new prediction unit: a new aliquot, a new specimen from a known person, a new person, or a new cohort. These are different tasks. A workflow that predicts another specimen from a previously observed person may legitimately use that person's earlier information if deployment will do so; it does not establish performance on entirely new people.
A Practical Audit Checklist
Trace the workflow from raw files to reported performance. For every operation, record its inputs, when it is fitted, whether it uses outcomes, and which samples are allowed to contribute. The checklist below is designed to expose information paths that a simple review of the final train/test split may miss.
Write down the prediction unit, the prediction time, and what information will be available then. Decide whether the evaluation is meant to represent new people, new specimens, or a future cohort.
Check person identifiers, aliquots, technical replicates, family relationships where relevant, and repeated measurements. Confirm that related samples are assigned to splits in a way consistent with the target.
List scaling, imputation, normalization, batch adjustment, missingness filters, variance filters, and feature selection. For each operation, ask whether its parameters were estimated using only the training partition or an allowed external reference.
Check whether a predictor is calculated from the outcome, recorded after the prediction time, or acts as an identifier or near-identifier. Document whether it could actually be obtained at deployment.
Record every time validation performance, labels, or error patterns were examined. If those results guided feature choices, model changes, thresholds, or stopping decisions, the set has become part of development.
Refit learned transformations within training partitions, group related samples before splitting, remove unavailable predictors, or obtain a fresh evaluation set after development decisions are fixed.
An operation fitted on training data can then be applied to held-out data using the training-fitted parameters. For example, estimate an imputation value or standardization mean within a training partition, and apply that same value to its held-out partition. A transformation fixed in advance using a genuinely external reference may be permissible if it does not use evaluation outcomes or information unavailable in the intended setting. The key issue is not whether a transformation is called “preprocessing”; it is what data informed it.
Worked Example: Auditing a Standardization Step
Suppose a hypothetical training partition has measurements for one genomic feature of \(2, 4,\) and \(6\) units, and the held-out sample has a value of \(100\) units. The intended use is prediction for a new sample, so the held-out feature value must not determine the training transformation.
The training mean is
If standardization uses the training sample standard deviation, then
The held-out value is transformed using those training estimates:
If the analyst instead standardizes all four values together, the mean becomes \((2+4+6+100)/4=28\). The held-out sample has now influenced the transformation applied to the training data. This is not outcome leakage, but it violates a strict inductive evaluation: the fitted pipeline has used information from the sample it is meant to treat as unseen. Recompute the transformation within each training partition and apply its estimates to that partition's held-out samples. If deployment explicitly allows joint access to an unlabeled batch, a transductive evaluation may be appropriate, but it answers a different question and must be stated as such.
Filtering, Repeated Samples, and Outcome-Derived Variables
Feature filtering is especially easy to overlook because it often occurs before model fitting. A filter based on variance, missingness, association with the outcome, or a ranking score is part of the learned analysis if its result depends on the data. For an honest held-out evaluation, estimate such a filter using only the relevant training data, then apply the selected feature set to held-out samples. This applies whether the filter is supervised or unsupervised for an inductive target: even without outcomes, using held-out feature distributions lets those samples affect the fitted pipeline.
Repeated measurements and technical replicates create a different route for information to cross a split. If two aliquots from one person appear on opposite sides, the model may recognize person-specific genomic signal rather than learn a pattern that generalizes to new people. The appropriate grouping unit depends on the target. For performance on new people, keep all samples from each person together; for performance on future samples from known people, the split and claim must reflect that setting. The earlier material on grouping repeated samples addresses this split-design issue; an audit should still verify the identifiers and actual assignments in the data.
Finally, inspect the timing and construction of each predictor. A variable derived from the outcome itself, a post-diagnosis treatment, or a code entered only after the clinical decision can reveal the label or unavailable future information. Such a predictor may yield excellent apparent performance while being unusable for the stated prediction task. A strong association with the outcome is not by itself proof of leakage: the relevant checks are how the variable was generated and whether it exists at prediction time.
Worked Example: A Technical Replicate Crosses the Split
Consider a hypothetical dataset of four people, each with two technical aliquots, and a binary outcome. There are eight rows. For a new-person evaluation, suppose a random row-level split places one aliquot from every person in training and the other aliquot in validation.
A nearest-neighbor classifier can match each validation aliquot to its nearly identical training aliquot from the same person. If, in this constructed example, all four validation aliquots are correctly assigned by that match, the validation accuracy is \(4/4=1.00\), or 100%. This calculation describes performance on four aliquots whose people were already represented in training; it does not estimate performance on new people.
The audit repair is to split by person before model fitting, putting both aliquots from each person in the same partition. With four people, the evaluation is necessarily small and uncertain, but it now tests the intended unit. Before interpreting the score, verify that duplicate detection and person identifiers are reliable: mislabeled or missing identifiers can leave the leak in place even when the splitting code requests grouped folds.
Validation Reuse and Selection Optimism
A validation set stops being an untouched evaluation set when its results influence development. Repeatedly trying features, model specifications, or thresholds and keeping the version with the best validation score selects partly for favorable random error. Nested cross-validation and the separation of tuning from evaluation were established in the earlier material on nested cross-validation; in an audit, the practical task is to establish what was tried and which results guided choices.
A short calculation shows why reuse is risky. Suppose two models have the same true performance \(\theta\), and their observed validation scores are \(\theta+\varepsilon_1\) and \(\theta+\varepsilon_2\), where the errors are independent, have mean zero, and are normally distributed with variance \(\sigma^2\). If the model with the larger observed score is selected and that same score is reported, the expected selected score is above \(\theta\):
For the last equality, \(\varepsilon_1-\varepsilon_2\sim N(0,2\sigma^2)\), and a zero-mean normal variable with standard deviation \(\sqrt{2}\sigma\) has expected absolute value \(2\sigma/\sqrt{\pi}\). The calculation assumes equal true performance and independent normal score errors; it illustrates the direction of selection bias, not a universal correction for real validation scores.
Worked Example: Detecting Reuse of a Validation Set
Imagine a hypothetical analysis that tries two equally good classifiers. Each has true accuracy \(0.80\), and suppose each validation accuracy has an approximately normal error with standard deviation \(\sigma=0.04\), independently of the other. The analyst chooses the higher validation accuracy and reports it as the performance estimate.
Under the assumptions above, the expected optimism from selecting the higher of two scores is
Thus, under this simplified model, the selected score averages about 2.26 percentage points above the shared true accuracy, giving an expected selected score of about \(0.8226\), or 82.26%. It does not mean every selected score will be that high, nor does it quantify the bias in a particular study. It demonstrates why an audit must ask whether validation results were used to choose between alternatives. If they were, treat that set as development data and use a genuinely fresh evaluation or an appropriate resampling design for the final estimate.
Diagnostics, Repairs, and Common Pitfalls
Warning signs include preprocessing code run once on the complete matrix before splitting; a feature list selected using all labels; unexpectedly perfect performance; similar-looking sample identifiers appearing across partitions; and many undocumented iterations guided by a validation score. These signs do not prove a particular bias, but they identify paths that need investigation. Keep a reproducible record of sample IDs by partition, transformation parameters, selected features, model decisions, and when validation results were viewed.
A common mistake is to assume that a split is honest because no validation outcome was passed directly to the classifier. That reasoning misses other routes: validation features can affect a global transformation, related samples can carry person-level information across folds, or analysts can choose a model after inspecting validation results. The correction is to audit the full information path, including code run before fitting and decisions made after scores are seen.
| Audit finding | Why it matters | Practical response |
|---|---|---|
| Scaling or imputation fitted on all samples | Held-out features influence the fitted pipeline in an inductive evaluation | Fit within training partitions; apply those estimates to held-out samples |
| Feature filter run before splitting | Selected features may reflect held-out data or labels | Repeat the filter using training data only |
| Related aliquots in different partitions | Evaluation may reward recognition of a known person or specimen | Group by the unit that must be new at prediction time |
| Post-outcome or outcome-derived predictor | It reveals information unavailable at the intended prediction time | Remove it or redefine and justify the prediction task |
| Validation set used to refine the workflow | Selection can make the reported score optimistic | Use a fresh evaluation set or a design that accounts for tuning |
No checklist can establish honesty from a performance score alone. The audit depends on provenance, timing, and the deployment question. Some data access may be legitimate under a clearly stated transductive or known-person target, but it should not be presented as evidence for a different, more demanding target. Likewise, removing a suspicious variable or rerunning a filter does not fix repeated validation reuse; each leak requires a repair that blocks its own information path.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A researcher selects the 500 markers most associated with outcome using all samples, then runs cross-validation. What is the audit concern, and what is the repair?
- The intended target is a new patient. Two aliquots from each patient are randomly divided into training and validation rows. Is that split suitable? Explain.
- A normalization method uses held-out feature values but not held-out outcomes. Is it automatically acceptable for prediction on a new sample?
- After examining validation results, an analyst changes the classification threshold and reports accuracy on that same set. What has happened to the validation set?
- A predictor is strongly associated with diagnosis. What information is needed before deciding whether it leaks the outcome?
Solutions. (1) The outcome-based marker filter used held-out labels, so cross-validation no longer evaluates the full selection procedure honestly. Refit the filter separately within each training partition and apply its selected markers to that partition's held-out samples. (2) No. The same patient is represented on both sides, so the result does not test prediction for new patients; assign both aliquots to the same partition. (3) No. For an inductive new-sample target, held-out features have influenced the fitted preprocessing. Fit normalization within training data and apply its parameters to held-out data, unless the stated deployment target explicitly allows joint unlabeled-batch processing. (4) The validation set has influenced development and is no longer an untouched evaluation set. Evaluate the fixed threshold on a fresh set or use a design that accounts for selection. (5) Establish how the predictor was generated and when it becomes available. Association alone does not establish leakage; a post-outcome or outcome-derived variable unavailable at prediction time is not a valid predictor for that task.