Why the Split Must Come First
A train-test split is meant to mimic a practical prediction task: fit a procedure using the training data, then assess it on observations that played no part in fitting. That separation fails if information from the test set influences an imputation model or any other data-dependent preprocessing step before evaluation. The test observations may still look “held out” in the final model-fitting code, but the workflow has already allowed them to shape the inputs.
This is data leakage. It can make test performance look better, worse, or simply different from what the same procedure would achieve on genuinely new data. The direction is not guaranteed; the fundamental problem is that the reported test result no longer estimates performance under the intended separation. Imputation deserves particular care because its fitted conditional models, variable summaries, and choices of predictors can all depend on the data supplied to them.
The principle is not “never use outcomes in imputation.” In an inferential analysis of an observed dataset, an outcome may appropriately be included in an imputation model for missing covariates. The issue here is the prediction setting: a held-out outcome is unknown when predictions are made. Using it to impute a held-out predictor would give the evaluated procedure information unavailable in deployment. The permitted information therefore depends on the task and on what will be available at prediction time.
What Must Be Isolated
Split the observations before estimating anything from the analysis data that will be used in preprocessing. This includes, for example, means used for imputation, regression coefficients in chained equations, scaling constants, category groupings inferred from observed values, and predictor selections based on associations in the data. A rule fixed independently of the analysis data—such as a prespecified clinically meaningful category definition—does not become leakage merely because it is applied to both sets.
For a strict inductive prediction workflow, training data determine the imputation procedure. Test records can then be processed using that fitted procedure, without allowing their values to update its parameters. When a test predictor is missing, its value may be generated using the imputation model fitted on training data and the available predictors for that test record. Test outcomes remain unavailable to this step. In a chained-equations procedure, this means the conditional models and other learned quantities come from training, while the test row supplies only the covariate information legitimately available for its imputation.
This distinction can be technically demanding: some MICE implementations are designed to impute a dataset by estimating a set of conditional models within that dataset, rather than to expose a simple fit-on-training, transform-on-test interface. Running MICE once on the combined train and test rows is not a substitute for that interface; it allows the test data to affect the fitted procedure. The workflow must preserve the separation even if that requires implementing or selecting a procedure that can apply training-fitted models to new records.
Specify which variables are available when a prediction is made and which variable is the outcome to be predicted.
Assign observations to training and test sets before fitting data-dependent imputation or preprocessing steps.
Estimate the imputation models and any other preprocessing quantities using training data and only permitted variables.
Use the training-fitted procedure on test predictors. Do not use test outcomes or update fitted preprocessing parameters from test records.
Compare predictions with the held-out outcomes only after the prediction procedure has been applied.
Worked Examples
Worked Example: A Test Value Changes the Training Imputation
Consider a hypothetical prediction dataset with three training records and one test record. The predictor \(X\) is measured in arbitrary units. Training values are \(2\), \(4\), and missing; the test value is \(100\). Suppose, just for this calculation, that missing values are replaced by the mean of the observed values used to fit the imputation rule.
If the split is respected, the training-only mean is
The missing training value is therefore imputed as \(3\). If the mean is instead calculated before splitting, the test value enters the calculation:
The missing training value would then be imputed as \(35.33\), rather than \(3\). The imputation rule fitted before the split has learned from the held-out predictor value, even though no test outcome was used. A mean imputer is not MICE, but the calculation makes the leakage mechanism explicit: a data-derived quantity used for training was estimated partly from test data.
The appropriate value for this strict inductive workflow is the training-only estimate, \(3\). The test value can be processed after fitting, but it must not change that estimate. This example assumes the mean is an appropriate imputation rule solely to isolate the leakage issue; it does not claim that mean imputation is generally adequate for MICE analyses.
Worked Example: A Held-Out Outcome Must Not Impute a Test Predictor
Suppose a hypothetical prediction task uses an available predictor \(X\) to predict a binary outcome \(Y\). In the training data, a fitted imputation model for \(X\) uses other available predictors, and the prediction model is then fitted using the completed training predictors and \(Y_{\mathrm{train}}\). A test record has a missing \(X\), an observed auxiliary predictor \(Z=2\), and a held-out outcome \(Y=1\). Assume the training-fitted imputation rule for this illustration is \(\widehat{X}=2Z\), and the fitted prediction rule classifies \(Y=1\) when \(X>3\).
The valid test-time imputation uses the available predictor:
The prediction is therefore \(Y=1\), because \(4>3\). The held-out outcome is used afterward to score that prediction. The prediction is correct, but the single example does not establish a performance rate.
Now suppose someone uses the held-out \(Y\) in an alternative imputation rule, \(\widehat{X}=2Z+10Y\). For this record, that gives \(2(2)+10(1)=14\). This uses the answer being predicted to construct a predictor, so it is leakage—even if the final classification happens to remain correct. The calculation illustrates why an outcome can be appropriate in some training imputation models but must be unavailable when imputing test predictors for prediction.
The example’s numeric rules are hypothetical and chosen for transparent arithmetic. In an actual workflow, the imputation rule must be estimated from training data, and its inputs must match what is available for each future prediction.
Worked Example: Leakage Inside Cross-Validation
Suppose a hypothetical training set is used for five-fold cross-validation to choose between two prediction models. A practitioner first estimates an imputation rule using all observations in that training set, fills its missing predictor values, and then divides the completed data into five folds. In each round, four folds are used to fit a prediction model and the remaining fold is used for validation.
Although each prediction model is fitted on four folds only, the imputation rule was estimated using the held-out fold as well. For example, if the rule is the observed mean, every validation-fold observation with an observed predictor value can affect the mean used to impute missing values in the four fitting folds and in the validation fold. Thus, the validation fold did not remain isolated during preprocessing.
The correction is to repeat the full fitting process within each round. For each fold, estimate the imputation models from the four fitting folds, apply those fitted models to both the fitting and validation predictors without updating them from the validation fold, then fit and score the prediction model. Once a model choice is made, estimate the final preprocessing and prediction procedure using the complete training set and evaluate it on the untouched test set.
This example assumes the folds are being used to estimate performance or compare models under an inductive prediction target. If the intended task explicitly permits using the entire batch of unlabeled test predictors before predicting its outcomes, that is a different, transductive target and must be described as such.
Justification: Why Fitting Within the Split Matters
A test-set performance estimate is intended to describe predictions from a procedure trained without access to the test observations. The procedure includes more than the final regression or classification model: it also includes the fitted preprocessing. Write \(A_{\mathrm{train}}\) for the information in training observations and \(A_{\mathrm{test}}\) for the information in test observations. In an inductive workflow, the fitted procedure has the form \(g(A_{\mathrm{train}})\). The test prediction for a record uses that fitted procedure and its available predictors \(X_{\mathrm{test}}\), but not its outcome \(Y_{\mathrm{test}}\):
If preprocessing is estimated on the combined data, the actual procedure instead has the form \(g(A_{\mathrm{train}},A_{\mathrm{test}})\). The test observations have then influenced the predictions before performance is calculated. The held-out comparison no longer evaluates a procedure fitted independently of those observations. Splitting first and fitting each data-dependent step only on training data restores the intended information flow.
This argument does not require test predictors to be statistically independent of training predictors in the population. It concerns which realized test records were used to construct the fitted procedure. It also does not imply that every test-set use is forbidden: the test outcomes must eventually be used to calculate an evaluation metric. The key is that they are used for evaluation, not for choosing or fitting the evaluated pipeline.
Diagnostics, Alternatives, and Limitations
A practical audit asks whether any quantity used to process a training record was estimated using test records, and whether any test record was processed using information unavailable at prediction time. Inspect the order of operations, not only the final prediction-model call. Warning signs include imputation performed on the full dataset before the split; one set of imputed values reused across cross-validation folds; imputation of test predictors using test outcomes; and feature selection or category handling performed globally before validation.
A useful comparison is between three possible workflows:
| Workflow | What informs imputation? | Interpretation |
|---|---|---|
| Training-fitted, inductive | Training data fit the imputation models; test outcomes are withheld | Evaluates prediction on new records using a fixed fitted procedure |
| Impute before splitting | Training and test records jointly inform the imputation procedure | Contaminates a strict held-out evaluation |
| Transductive processing | Training data and permitted unlabeled test predictors may inform processing | Can be valid only when this access matches the stated prediction task |
The transductive option is not automatically invalid. For example, an application may genuinely make a batch of unlabeled predictor records available before any outcomes are known. A procedure designed to use that batch can be evaluated for that setting, provided test outcomes remain unavailable and the evaluation is described honestly. It does not estimate the same target as a procedure intended to predict one new record using a model fixed from training data.
Multiple imputation adds further choices: imputations are stochastic, and predictions or performance summaries may be combined across completed datasets in ways appropriate to the analysis. Those choices do not remove the separation requirement. For strict evaluation, each training-fitted imputation procedure must be estimated without the held-out fold or test outcomes. A procedure that estimates fresh conditional models from the combined data is not made leakage-free merely by producing multiple completed datasets.
Finally, leakage prevention does not guarantee that the imputation model is adequate, that missingness assumptions are plausible, or that the evaluation sample represents the intended population. Nor does it by itself resolve how uncertainty from imputation should enter a particular prediction metric. These are distinct questions. The immediate diagnostic is narrower: did any information unavailable at prediction time influence the fitted pipeline?
Common Mistake
Wrong reasoning: “I did not include the test outcomes in the final prediction-model fit, so the test set is untouched.” This overlooks preprocessing. If the imputation model, means, or selected predictors were learned before the split using test observations, those observations already influenced the pipeline.
Correction: Split first. Within each training split or cross-validation fold, fit the imputation and preprocessing steps using only the fitting portion. Apply the fitted steps to the held-out predictors without using held-out outcomes or updating fitted quantities from the held-out records. Keep the final test outcomes for scoring.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A practitioner estimates predictor means on the full dataset, imputes missing values, and then makes a train-test split. Is this a strict inductive evaluation? Why?
- A test record has missing predictors and an observed outcome that will be used to assess prediction. May that outcome be used to impute the record’s predictors before predicting it?
- In five-fold cross-validation, what data should be used to fit the imputation models for one fold’s validation predictions?
- When can using unlabeled test predictors during preprocessing be defensible?
Solutions. (1) No. The full-data means incorporate test observations, so preprocessing was influenced by data meant to be held out. Split first and estimate the means or imputation models using training data only. (2) No. The outcome is the answer being predicted and is unavailable at prediction time; use only permitted predictors and training-fitted imputation models, then use the outcome for scoring. (3) Use only the four fitting folds for that round to fit the imputation models. Apply those fitted models to the validation predictors without refitting from the validation fold. (4) It can be defensible when the intended prediction task genuinely permits access to the whole batch of unlabeled predictors before outcomes are known. This is a transductive target, not the strict inductive target, and should be evaluated and reported accordingly.