Why Preprocessing Must Respect the Split
A validation set is meant to stand in for observations the model has not seen during development. That protection is lost if the validation data help determine how the inputs are prepared. For example, calculating a mean or standard deviation from the full dataset allows validation predictors to affect the scaling applied before the model is evaluated. The model may never use validation outcomes, yet information has still crossed the boundary.
This issue is called preprocessing leakage. In the usual predictive setting, the goal is to estimate performance on future observations whose feature values are not available when the model is developed. Therefore, quantities learned from the data during preprocessing must be estimated from the training set alone. The fitted preprocessing rule is then applied, unchanged, to validation and test data.
A preprocessing operation can be applied to a held-out record without leakage. The key distinction is between fitting the operation, which estimates a rule or its parameters, and applying it, which uses those already-fitted parameters on new records. Calculate a training mean, for instance, then use that mean to transform training, validation, and test values. Do not recalculate the mean using each held-out set.
Fit on Training Data, Then Apply
Suppose a training set contains observed values \(x_1,\ldots,x_n\) for a numerical predictor. Under the population-form convention for a standardizer, its training mean and standard deviation are
The training values and any later value \(x\) are standardized using those training estimates:
Some software uses a sample standard deviation with divisor \(n-1\) rather than the population-form standard deviation above. Either convention can be used if specified and applied consistently; the leakage rule is the same. In the examples here, the population-form standard deviation is used. If \(s_{\mathrm{train}}=0\), the formula is undefined, so a constant predictor must be handled separately, for example by removing it according to a rule established using the training data.
Worked Example: Standardizing a Validation Value
A hypothetical insurer is developing a model using a numerical predictor measured in dollars. The training values, in thousands of dollars, are \(2,4,6\); a validation record has value \(10\). Using the population-form standard deviation, find the training standardization parameters and the properly standardized validation value. Also calculate the value that would result from fitting the standardizer on all four observations.
Method. Fit the mean and standard deviation on training observations only, then apply those estimates to the validation value.
Training mean. The sum is \(2+4+6=12\), so \[ \bar{x}_{\mathrm{train}}=12/3=4. \] The squared deviations are \((2-4)^2=4\), \((4-4)^2=0\), and \((6-4)^2=4\). Their sum is \(8\), giving \[ s_{\mathrm{train}}=\sqrt{8/3}\approx1.633 \] to three decimal places.
Correct validation transformation. Apply the training estimates: \[ z_{\mathrm{val}}=\frac{10-4}{\sqrt{8/3}} =\frac{6}{\sqrt{8/3}} \approx3.674 \] to three decimal places.
Leaky calculation. If all four values are used to fit the standardizer, their mean is \((2+4+6+10)/4=5.5\). Their squared deviations from \(5.5\) sum to \(12.25+2.25+0.25+20.25=35\), so the full-data population standard deviation is \(\sqrt{35/4}=\sqrt{8.75}\). The validation value then becomes \[ \frac{10-5.5}{\sqrt{8.75}}=\frac{9}{\sqrt{35}}\approx1.521 \] to three decimal places.
The two transformed values differ because the validation predictor helped determine the full-data mean and standard deviation. That is leakage in the standard held-out evaluation setting, even though the validation outcome was not used. The common mistake is to treat “no validation outcomes used” as sufficient protection; here, the validation feature value itself affected preprocessing.
Imputation and Data-Dependent Transformations
The same boundary applies when replacing missing values. If the training-set mean is the chosen imputation value, estimate it from observed training values only. Use that fitted value for missing entries in both training and held-out sets. A held-out set may have a different missingness pattern or a different distribution; that is part of what evaluation should reveal, not a reason to refit the imputer on the held-out data.
Worked Example: Fitting a Mean Imputer Without Validation Information
A hypothetical benefit model uses a measured account balance. Its training records have values \(12,15,\) and \(21\), along with one missing value. The validation records have values \(30\) and one missing value. Find the proper mean imputation for the missing entries, and compare it with the result of incorrectly estimating the mean from all observed records.
Method. Estimate the mean from observed training values and carry that estimate over to the validation set.
The observed training values sum to \(12+15+21=48\), with \(3\) observed values. Thus the training imputation value is \(48/3=16\). The training missing entry and the validation missing entry are each imputed as \(16\).
If all observed records are pooled, the observed sum is \(12+15+21+30=78\), with \(4\) observed values. The resulting mean is \(78/4=19.5\). Using \(19.5\) for the validation missing value lets the validation observation \(30\) change the imputation rule.
The correct validation imputation is \(16\), not \(19.5\). The common mistake is to regard an unsupervised estimate such as a predictor mean as harmless to calculate globally. For an inductive evaluation, it still uses held-out feature information.
A transformation can also have parameters estimated from data. For example, a winsorization rule may cap values above a chosen training quantile. The rule itself can be fixed in advance, but its cutoff is data-dependent and must be fitted using training values alone. The empirical quantile convention should be stated because quantile definitions can differ; the next example uses the nearest-rank definition.
Worked Example: Estimating a Winsorization Cutoff
A hypothetical risk score is capped at its empirical 75th percentile before model fitting. Use the nearest-rank definition: for \(n\) sorted observations and probability \(p\), the selected rank is \(\lceil np\rceil\). Training values are \(2,4,6,8\), and a validation value is \(100\). Find the training-based cutoff and transformed validation value, then compare with the cutoff obtained by incorrectly pooling the validation value with training data.
Method. Estimate the cutoff using training data, then cap each value at that cutoff.
The ordered training values are \(2,4,6,8\). The selected rank is \(\lceil4(0.75)\rceil=\lceil3\rceil=3\), so the training 75th-percentile cutoff is \(6\). Capping the validation value gives \(\min(100,6)=6\).
If the validation value is included, the ordered pooled values are \(2,4,6,8,100\). The selected rank is \(\lceil5(0.75)\rceil=\lceil3.75\rceil=4\), giving a cutoff of \(8\). The validation value would then be capped at \(8\).
The leakage-safe transformed validation value is \(6\). The full-data calculation produces \(8\) because the validation record changed the estimated cutoff. The common mistake is to think that a predetermined capping formula makes its fitted cutoff predetermined; the formula is fixed, but the cutoff is estimated from data.
Why the Standardization Formula Works
The training-based standardization has a useful property: it makes the training values have mean zero and population-form variance one, provided the training standard deviation is positive. This follows directly from the definitions. For each training observation, set \(z_i=(x_i-\bar{x}_{\mathrm{train}})/s_{\mathrm{train}}\). Then
because the deviations from the training mean sum to zero. The average squared standardized value is
by the definition of \(s_{\mathrm{train}}^2\). This result is about the training data used to fit the standardizer. It does not say that validation or test values will have mean zero or variance one. Forcing each held-out set to have those properties would estimate new parameters from evaluation data and change the evaluation procedure.
A Leakage-Safe Workflow
Keep the split boundary explicit throughout development. The order below applies whether the preprocessing consists of one operation or several. It is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Create training, validation, and, when used, test sets before estimating data-dependent preprocessing quantities.
Estimate imputation values, scaling parameters, and transformation cutoffs using training observations only. Record the fitted values and rules.
Transform training data using the fitted steps, and apply those same steps without refitting to validation data. Apply them unchanged to test data for final evaluation.
Do not use validation or test values to revise preprocessing estimates during the evaluation being reported.
When cross-validation is used, the same principle applies within each fold: fit preprocessing on that fold’s training portion and apply it to that fold’s held-out portion. Fitting a scaler once on all observations before forming the folds lets every held-out fold influence its own preprocessing. Any preprocessing learned from outcomes must also be restricted to the training portion.
Common Mistakes and the Scope of the Rule
- Splitting after fitting. Computing means, standard deviations, or cutoffs on the full dataset before making a split lets held-out observations influence the fitted transformation.
- Refitting on validation data. Applying a training-fitted imputer is appropriate; recomputing its imputation value from validation records is not appropriate for the standard held-out evaluation.
- Assuming predictors cannot leak. A feature value can affect an imputation or scaling estimate even when its outcome is never examined.
- Confusing fixed rules with fitted parameters. A rule such as “cap at the training 75th percentile” still requires estimating that percentile from training observations.
- Ignoring the evaluation design. In a different, explicitly transductive task, unlabeled future predictors may be available before predictions are made. Using them is defensible only if the intended deployment and evaluation procedure both allow that information. It is not the usual assumption for a held-out estimate of performance on future, unseen observations.
Check Your Understanding
For each problem, assume the population-form standard deviation when standardizing and use the nearest-rank empirical quantile when specified.
- A training predictor has values \(1,3,5\), and a validation value is \(9\). Find the training mean, population-form standard deviation, and correctly standardized validation value. (Answer: mean \(3\), standard deviation \(\sqrt{8/3}\approx1.633\), validation value \(6/\sqrt{8/3}\approx3.674\), rounded to three decimals.)
- Observed training values for a predictor are \(4\) and \(8\), with one training value missing. Validation has one observed value \(20\) and one missing value. What value should be used to impute each missing entry, and what value would result from incorrectly pooling all observed records? (Answer: use \(6\) for both missing entries; pooling gives \(32/3\approx10.667\).)
- Training values are \(3,6,9,12\), and a validation value is \(30\). Find the training-based standardized validation value using the population-form standard deviation. (Answer: training mean \(7.5\), standard deviation \(\sqrt{11.25}\approx3.354\), standardized value \(22.5/\sqrt{11.25}\approx6.708\), rounded to three decimals.)
- Use nearest rank to cap values at the training 75th percentile. Training values are \(1,5,7,9\), and validation is \(50\). What is the cutoff and capped validation value? What cutoff results if the validation value is pooled in? (Answer: training cutoff \(7\) and capped value \(7\); pooled cutoff \(9\).)
- A team applies a logarithm chosen in advance, but estimates its centering constant by calculating the median of all predictors before splitting. Is the transformation leakage-safe for a standard held-out evaluation? State the correction. (Answer: no; estimate the median from training predictors only, then use that same centering constant for training, validation, and test data.)