Why Preprocessing Must Respect the Split
A model evaluation is intended to estimate how well a model would perform on observations that were not used to develop it. Preprocessing is part of that development process whenever it learns something from the data. If an evaluation partition influences an imputation value, a scale parameter, a category map, or a cutpoint, information has crossed the partition boundary—even if the outcome column was never used.
The key distinction is between fitting a preprocessing rule and applying it. Fitting estimates the rule’s parameters from training observations. Applying uses those already-estimated parameters to transform training or evaluation features. The evaluation features do not get to change the rule.
Some preprocessing rules are fixed without estimating anything from the data. For example, a rule that converts a dollar amount to thousands by dividing by 1,000 is fixed by the chosen units. By contrast, the mean used for imputation, the mean and standard deviation used for standardization, or a median used as a bin boundary are learned parameters. A transformation such as taking a logarithm can be fixed in advance; choosing a power transformation after inspecting the data is a learned decision.
What It Means to Fit and Then Apply
Use the established split to identify which rows are available for model development and which rows are reserved to assess predictions.
For example, calculate a training mean using observed training values, or calculate a training standard deviation using the training feature.
The training observations and the evaluation observations must use the same imputation value, scale, category map, or cutpoint.
The evaluation outcomes are used to calculate performance, not to revise preprocessing or the fitted model.
This procedure does not mean that training and evaluation data must have identical distributions. They may differ, and that difference can be important evidence about how the model generalizes. The point is to measure performance under a rule developed without access to the evaluation observations, rather than allowing the evaluation data to help prepare itself.
Worked Examples
Worked Example: Mean Imputation Without Evaluation Leakage
A hypothetical insurer is developing a model using a numeric feature, the number of service requests recorded per policy. In the training partition the values are \(10, 12, \text{missing}, 14\). In the evaluation partition they are \(18, \text{missing}, 30\). Use the mean of observed training values to impute missing values. Compare this with incorrectly calculating one mean from all observed values in both partitions.
Method. Mean imputation estimates the replacement value from the observed training values only. It avoids allowing evaluation features to determine the imputation rule.
Step 1: calculate the training mean. The observed training values sum to \(10+12+14=36\), and there are 3 observed values. Thus the training mean is \(36/3=12\). The missing training value is replaced by 12.
Step 2: apply that value to the evaluation partition. The evaluation value 18 stays 18, the missing value is replaced by 12, and 30 stays 30. The transformed evaluation values are therefore \(18,12,30\). The observed evaluation values do not alter the training mean.
Step 3: calculate the incorrect pooled value. If the evaluation observations are included when estimating the mean, the five observed values are \(10,12,14,18,30\). Their sum is 84, so the pooled mean is \(84/5=16.8\). That incorrect procedure would replace the missing evaluation value by 16.8 rather than 12, and would also replace the missing training value by 16.8.
Check. The training-only replacement is 12; the pooled replacement is 16.8. The difference arises because the evaluation value 30 helped determine the pooled mean. The evaluation row has therefore influenced the preprocessing used to assess the model.
Worked Example: Standardizing With Training Statistics
Suppose a numeric feature has training values \(2,4,6\), and an evaluation observation has value 10. Standardize using the training mean and the training sample standard deviation, with denominator \(n-1\). Find the correct standardized evaluation value and compare it with the result from incorrectly standardizing all four values together.
Method. Estimate the center and scale from the training observations, then apply that fixed standardization to the evaluation value. This avoids giving evaluation observations influence over the coordinate system used by the model.
Step 1: training mean. The mean is \((2+4+6)/3=4\).
Step 2: training sample standard deviation. The deviations from 4 are \(-2,0,2\), and their squared values are \(4,0,4\). The sum of squared deviations is 8. With 3 observations, the sample variance is \(8/(3-1)=4\), so the sample standard deviation is \(\sqrt{4}=2\).
Step 3: transform the evaluation value. Subtract the training mean and divide by the training standard deviation: \((10-4)/2=3\). The evaluation feature is 3 training standard deviations above the training mean.
Step 4: show the incorrect calculation. If all four values \(2,4,6,10\) are used, their mean is \(22/4=5.5\). Their deviations are \(-3.5,-1.5,0.5,4.5\); the squared deviations sum to \(12.25+2.25+0.25+20.25=35\). The pooled sample standard deviation is \(\sqrt{35/3}\), approximately 3.4157. The resulting evaluation value is \((10-5.5)/3.4157\), approximately 1.317.
Check. The correct answer, using the stipulated sample standard deviation, is 3.000. The pooled calculation produces approximately 1.317 because the evaluation value altered both the center and scale. A zero training standard deviation would make this standardization undefined; such a feature needs a separate, prespecified handling rule rather than an evaluation-based scale.
Worked Example: Encoding a Category Not Seen in Training
A hypothetical claims model uses a service channel with categories “standard” and “priority” in training. Use “standard” as the reference category and create one indicator column for “priority.” An evaluation observation has category “priority,” and another has a previously unseen category “mobile.” Apply a policy, fixed in advance, that maps an unseen category to the reference category.
Method. Learn the category levels and indicator columns from training, then apply the same encoding to evaluation data. This avoids inventing a new model input column because a category appeared in the evaluation partition.
Step 1: define the training encoding. The training levels are “standard” and “priority.” With “standard” as the reference, the single indicator is 0 for “standard” and 1 for “priority.”
Step 2: encode the evaluation observations using that map. The “priority” observation receives 1. The “mobile” observation is not one of the training levels, so the stipulated unseen-category policy maps it to the reference category and it receives 0.
Step 3: check the model inputs. Both evaluation observations have exactly the one input column available to the model: “priority.” No “mobile” coefficient can be estimated from training because “mobile” did not occur there. Creating a new indicator from evaluation data would not solve that problem; its coefficient would still have no training information supporting an estimate.
Check. The encoded values are 1 for “priority” and 0 for “mobile” under the stated policy. The answer depends on that policy: mapping unknown categories to the reference is not universally appropriate, so the policy must be selected without inspecting evaluation outcomes and documented. In a practical setting, other policies may be suitable, but they must also be fixed for evaluation.
Worked Example: Keeping a Data-Derived Bin Boundary Fixed
A hypothetical model groups a numeric feature into two bins using the median of the training feature values. The training values are \(5,9,13,17\), and the evaluation values are \(0,1,2,8\). Define the lower bin as values less than or equal to the training median and the upper bin as values above it. Find the correct evaluation bin assignments, then show what goes wrong if the median is recalculated after pooling the partitions.
Method. Estimate the median cutpoint from training values only and use it unchanged. This avoids letting the evaluation feature distribution move the boundary.
Step 1: calculate the training median. The two middle training values are 9 and 13, so the median is \((9+13)/2=11\).
Step 2: apply the training cutpoint. The evaluation values \(0,1,2,8\) are all less than or equal to 11, so all four are assigned to the lower bin.
Step 3: calculate the incorrect pooled median. The sorted pooled values are \(0,1,2,5,8,9,13,17\). The middle two values are 5 and 8, so the pooled median is \((5+8)/2=6.5\). With that incorrectly revised cutpoint, the evaluation value 8 would be assigned to the upper bin, while \(0,1,2\) would remain in the lower bin.
Check. The valid evaluation assignments are lower, lower, lower, lower, based on the training cutpoint 11. The pooled cutpoint 6.5 changes the assignment of 8 because evaluation observations helped move the boundary.
Why the Training Mean Is the Relevant Mean
The mean-imputation and standardization examples illustrate a general justification: preprocessing parameters describe the rule available to the model at the time it is fitted. For example, the training mean is the value that minimizes the sum of squared deviations among training observations. To see this, let the observed training values be \(x_1,\ldots,x_n\), and consider choosing a constant \(c\) to minimize their total squared distance.
Differentiate with respect to \(c\) and set the derivative to zero. The second derivative is \(2n>0\), so the stationary point is the minimum.
If evaluation values are added to this objective, its minimizer becomes the mean of the combined data. That may be a valid descriptive summary of the combined values, but it is not the preprocessing rule estimated using training observations alone. The assessment would then reflect both the model and a data-dependent transformation partly learned from the observations used to assess it.
Common Exam Traps and Practical Checks
A frequent mistake is to equate “unsupervised” with “safe to fit anywhere.” Imputation, scaling, and binning may not use the outcome, but their estimated parameters still depend on feature values. If evaluation features help estimate those parameters, the evaluation partition has influenced the model-development process. This can make the assessment less representative of the intended procedure.
Another trap is to transform training data with training statistics but transform evaluation data with evaluation statistics. That gives the two partitions different definitions of the same feature. For example, a standardized value of 1 in training means one training standard deviation above the training mean; if evaluation is standardized using its own mean and standard deviation, the same value no longer has that interpretation. The model must receive evaluation features transformed according to the training rule.
- Imputation: calculate numeric replacements, category replacements, or missingness-related learned quantities from training observations only.
- Scaling: estimate means, standard deviations, minima, maxima, or other scale parameters from training observations only. Check for zero or otherwise unusable training scales.
- Categories: define levels and any rare-level grouping using training data. Specify in advance how categories absent from training will be handled.
- Binning and transformations: estimate cutpoints or transformation parameters on training data; do not recalculate them for evaluation.
- Model development decisions: if a choice is made after comparing alternatives using evaluation performance, that evaluation result has influenced development. Keep the intended evaluation separate from such decisions.
A useful audit question is: Could this preprocessing value or decision change if the evaluation rows were different? If the answer is yes, it is learned from data and should be estimated using training observations, then held fixed. The same principle applies whenever a model is assessed on a partition reserved for evaluation.
Check Your Understanding
For each problem, identify the training-only preprocessing rule and avoid using evaluation observations to revise it.
- Training values for a numeric feature are \(4,8,\text{missing},12\); evaluation values are \(20,\text{missing}\). Use the observed training mean for imputation. What value replaces each missing entry? (Answer: 8 in both partitions.)
- Training values are \(1,3,5\), and an evaluation value is 9. Standardize using the training mean and sample standard deviation. What is the standardized evaluation value? (Answer: 3.000.)
- Training categories are “web” and “phone.” Use “web” as reference and a “phone” indicator; map an unseen evaluation category “agent” to the reference by a prespecified rule. What are the indicator values for evaluation categories “phone” and “agent”? (Answer: 1 and 0, respectively.)
- A two-bin feature uses the training median as its cutpoint. Training values are \(2,6,10,14\), and evaluation values are \(0,8,20\). With values at or below the cutpoint in the lower bin, what are the evaluation assignments? (Answer: The training median is 8; the assignments are lower, lower, upper.)
- Training values are \(3,7,11\), and the evaluation value is 15. A colleague calculates the mean and sample standard deviation from all four values before standardizing. Explain why this is inappropriate and give the correct standardized evaluation value using training statistics. (Answer: Evaluation data must not determine the fitted scale; the training mean is 7, the training sample standard deviation is 4, and the correct standardized value is 2.000.)