Building Leakage-Safe Analysis Pipelines

Why Treat the Analysis as One Procedure?

A genomic prediction analysis often combines several operations: filling in missing predictor values, scaling predictors, selecting features, and fitting a prediction model. Each operation can learn something from the observed data. An imputer learns quantities such as feature means; a scaler learns centers and scales; a supervised selection rule learns which features appear related to the outcome; and the model learns its coefficients. If any of these steps uses validation observations while being fitted, the validation results no longer represent predictions from a procedure trained without those observations.

The safe approach is to define the sequence as one pipeline. For a particular training split, fit every data-dependent step using that split alone. Then apply the fitted steps in order to the held-out observations, without updating them. Repeat the whole fit-and-apply process afresh for each split. This is more than a software convenience: it specifies which information is allowed to influence each prediction.

Let \(X_{\mathrm{tr}}\) and \(y_{\mathrm{tr}}\) denote the predictor matrix and outcome vector in a training fold, and let \(X_{\mathrm{val}}\) denote the predictors in its validation fold. A pipeline can be represented as a sequence of fitted maps: an imputation map \(I\), a scaling map \(S\), a feature-selection map \(F\), and a prediction model \(M\). The maps are fitted in sequence on training data; validation predictors pass through those already-fitted maps.

The order shown is illustrative: some legitimate pipelines use a different order, and some omit steps. What matters is that each learned quantity is estimated from the training fold, and validation data receive the corresponding transformation without influencing its estimate. If feature selection is supervised, it uses \(y_{\mathrm{tr}}\), never validation outcomes. A fixed operation that does not estimate anything from the data does not need fitting, but it must still be applied consistently.

One Split, One Complete Fit

The pipeline is refitted from the beginning at every training split. The imputation values, scaling constants, selected features, and model parameters may all change from fold to fold. Cross-validation estimates performance for this repeated procedure; it does not create one universal set of preprocessing quantities that can be estimated once from the full data set.

Worked Fold With Imputation, Scaling, Selection and Prediction

The following small hypothetical example makes the information flow explicit. It is illustrative rather than a genomic study. Suppose the training fold contains four samples, two predictors \(A\) and \(B\), and a continuous outcome. Predictor \(A\) is missing for one training sample.

Why the Whole Pipeline Must Be Refit

Suppose two folds have different training samples. Even if both folds use the same named operations, their fitted maps are generally different. For example, a feature's training mean, its scale, and its association with the outcome depend on which observations are in that fold. Reusing estimates from another fold is not equivalent to fitting the stated procedure on the current training fold.

A short justification follows directly from the definition of the prediction procedure. Conditional on the training fold, its fitted pipeline is determined by \(X_{\mathrm{tr}}\) and \(y_{\mathrm{tr}}\). A valid held-out prediction is then a function of that fitted pipeline and the held-out predictors. If an imputer, scaler, or selector is instead fitted using the validation observations, the fitted map also depends on validation data. The resulting score evaluates a procedure that had access to information unavailable in the intended training setting.

The distinction matters even when validation outcomes are never used. Using validation predictors to estimate scaling or imputation still changes the learning procedure and allows information from the evaluation sample into its fitted representation. Whether that constitutes a practically large bias depends on the setting, but it is not the same validation design as training-only preprocessing.

Assumptions, Diagnostics, and Boundaries

Fold awareness prevents a particular information-flow error; it does not guarantee that the prediction model is appropriate or that the evaluation split answers the scientific question. The split must respect the target population and dependence structure. In genomic data, for example, related individuals or samples from a shared site may require grouped or site-aware splits when the intended prediction is to new families or sites. The pipeline cannot repair a split that places related observations on both sides when that split is incompatible with the prediction target.

The individual operations also need explicit rules for their edge cases. A feature can be entirely missing in a training fold, so its training mean is undefined. A feature can be constant in a fold, so standardization by its sample standard deviation is undefined. The analysis must specify a consistent response, such as dropping that feature in that fold or using a prespecified fallback rule; the validation data must not be used to decide the response. Likewise, a feature selector may choose no features, or a model may fail to converge. These are pipeline failures to report and handle, not reasons to silently substitute validation information.

Useful diagnostics include checking, for every fold, the number of missing values before and after imputation, the estimated centers and scales, the number and identity of selected features, and whether model fitting completed. Check that the validation transformation has the same feature names and order expected by the fitted model. Investigate unexpectedly identical preprocessing values across folds when the training subsets differ, since that can indicate accidental reuse of globally fitted transformations. Also verify that no validation outcomes were passed to any step that selects features or estimates model parameters.

A common mistake is to scale or select features once on the complete data set and then run cross-validation on the resulting matrix. The reasoning is often, “The validation outcomes were hidden, so there can be no leakage.” The correction is that predictors alone can affect estimated imputation and scaling values, and supervised selection can directly use outcomes if it is done globally. All data-adaptive steps belong inside the fold-specific pipeline. The alternative—preprocessing once before splitting—is only appropriate for operations that are genuinely fixed in advance and do not estimate their behavior from the data being evaluated.

A second alternative is to write separate preprocessing code for every fold. That can be valid if it faithfully refits every learned step and applies the fitted versions consistently, but it is error-prone: one global scaling call or a feature-name mismatch can break the information boundary. A pipeline implementation packages the ordered operations together and reduces this risk. It does not decide the right split design, imputation strategy, feature selector, or model; those choices still require scientific justification. Nor does this tutorial decide how to choose among competing hyperparameter settings. The essential point here is that whichever procedure is evaluated must be refitted in its entirety inside the relevant training split.