Why Scaling Can Leak Information
A held-out test set is meant to represent observations that were unavailable when the predictive model and its preprocessing were developed. A common mistake is to split the data, then calculate a predictor's mean and standard deviation using both the training and test observations. The outcome values in the test set may remain hidden, but its predictor values have still influenced the fitted preprocessing.
This is a form of test-set information leakage. It can make a test evaluation less representative of the intended prediction task: the test set has helped determine how the inputs are represented before the model is assessed. The issue is subtle because scaling is an unsupervised calculation: it does not use the outcome. But withholding test outcomes alone does not withhold test information if test predictors are used to estimate preprocessing parameters.
This tutorial focuses on the information flow caused by scaling. It does not mean that every use of unlabeled test predictors is automatically invalid in every possible setting. For example, a deliberately transductive task may allow the predictor values of the specific cases to be predicted. That is a different prediction setting and must be stated explicitly. In a conventional held-out evaluation, use training-only scaling so the test set remains an honest check of the fitted workflow. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Standardization: The Test Mean Enters the Training Transformation
For a training predictor \(x\), let \(\bar{x}_{tr}\) be its training-sample mean and \(s_{tr}\) its training-sample standard deviation. The training-fitted z-score is \((x-\bar{x}_{tr})/s_{tr}\). Use the same two quantities to transform every test observation. Here, as in R's sd() and scale() defaults, the standard deviation is the sample standard deviation, with denominator \(n-1\).
If instead the mean is calculated over all observations, it combines the training and test means. Writing \(n_{tr}\) and \(n_{te}\) for the two sample sizes, and \(\bar{x}_{te}\) for the test-predictor mean, gives
This identity follows directly by adding the training values and test values, then dividing their combined sum by their combined count. Unless the test mean happens to equal the training mean, the full-data mean differs from the training mean. The full-data standard deviation also depends on test values, so using both full-data statistics lets the test set affect the transformed training observations as well as its own transformed observations.
Worked Example: Quantifying Leakage in Z-Score Scaling
A hypothetical insurer splits six policy records into four training records with predictor values \(0, 2, 4, 6\), and two test records with values \(8, 10\). Calculate the training-only z-scores for the test records, then compare them with z-scores obtained by mistakenly scaling all six values together. Use sample standard deviations and round z-scores to three decimals.
Method. First calculate the training mean and sample standard deviation. Then apply those fixed values to the test observations. For the mistake, recalculate both quantities using all six predictors.
Correct training mean. The training sum is \(0+2+4+6=12\), so \(\bar{x}_{tr}=12/4=3\). The squared deviations from 3 are \(9,1,1,9\), with sum 20. Thus \(s_{tr}=\sqrt{20/(4-1)}=\sqrt{20/3}=2.582\) to three decimals.
Correct test scores. For the test value 8, \(z=(8-3)/\sqrt{20/3}=1.936\). For the test value 10, \(z=(10-3)/\sqrt{20/3}=2.711\). The training observations themselves have scores \(-1.162,-0.387,0.387,1.162\), using the same training mean and standard deviation.
Leaking calculation. The six-value sum is 30, so \(\bar{x}_{all}=30/6=5\). The squared deviations from 5 are \(25,9,1,1,9,25\), summing to 70. Hence \(s_{all}=\sqrt{70/(6-1)}=\sqrt{14}=3.742\).
Using the leaking statistics, the test scores become \((8-5)/\sqrt{14}=0.802\) and \((10-5)/\sqrt{14}=1.336\). The training values are also changed: their full-data scores are \(-1.336,-0.802,-0.267,0.267\), rather than the training-only scores above.
Check. The test mean is \((8+10)/2=9\), and the weighted-mean identity gives \((4\cdot3+2\cdot9)/6=5\), agreeing with the full-data mean. The test values have pulled the center upward and increased the estimated spread. The mistake is not merely using a different scale for test observations; it changes how the training observations are represented too.
Min-Max Scaling Has the Same Information-Flow Problem
Min-max scaling maps a value using a fitted minimum and maximum. If training observations range from \(a_{tr}\) to \(b_{tr}\), the training-fitted transformation is \((x-a_{tr})/(b_{tr}-a_{tr})\). Values outside the training range can legitimately produce results below 0 or above 1. Clipping them to the interval \([0,1]\) is an additional modelling choice, not part of the scaling formula.
The condition \(b_{tr}>a_{tr}\) is necessary because otherwise the denominator is zero; a constant predictor needs separate handling. If test values are allowed to set the minimum or maximum, the training transformation changes to accommodate observations that were supposed to be held out.
Worked Example: A Held-Out Maximum Changes Min-Max Scaling
A hypothetical claim-severity predictor has training values \(10,20,30\) and a test value of \(50\). Calculate the training-fitted min-max values, then show what happens if the test value is included when estimating the maximum. Round to three decimals.
Training-fitted method. The training minimum is 10 and maximum is 30, so the training range is \(30-10=20\). The training values map to \((10-10)/20=0\), \((20-10)/20=0.5\), and \((30-10)/20=1\). The test value maps to \((50-10)/20=2\).
Full-data mistake. Including 50 changes the maximum to 50 and the range to \(50-10=40\). The training values now map to \(0\), \(10/40=0.25\), and \(20/40=0.5\). The test value maps to \(40/40=1\).
Check. The test observation has changed the scale applied to every training value: the training maximum maps to 1 under the correct method but to 0.5 under the leaking method. A test value above the training range is not an arithmetic error; it is evidence that the held-out case lies beyond the range seen during fitting. The common mistake is to force that value into the training range by using it to reset the maximum.
A Reproducible Training-Only Workflow
The safe workflow has a simple boundary: estimate each scaler from the training predictors, save the fitted parameters, and apply those same parameters to other data. For standardization, save the training mean and standard deviation. For min-max scaling, save the training minimum and maximum. Do not recompute them when transforming the test set.
Separate training and test records before estimating any scaling parameters. Keep test outcomes aside for the final evaluation.
Calculate the training mean and sample standard deviation, or the training minimum and maximum, for each predictor requiring that scaling.
Apply the training-fitted formula to training and test predictors without recalculating its parameters.
Use the same stored parameters for later observations that are processed by the deployed workflow.
For example, R's scale() can fit the standardization on the training vector, but applying scale() separately to the test vector would estimate new test means and standard deviations. Instead, apply the saved training values explicitly:
These expressions assume the training predictor has nonzero standard deviation. If \(s_{tr}=0\), the standardization formula is undefined and the predictor is constant in training; do not divide by zero. The appropriate handling depends on the analysis, but it must not be decided by consulting the test distribution. Likewise, a missing training value requires a defined missing-value procedure before the mean and standard deviation are calculated.
Worked Example: Applying Saved Training Statistics to New Records
A hypothetical training set has predictor values \(1,3,5,7\). Two later records have values \(9\) and \(11\). Calculate the training mean and sample standard deviation, transform the two later values using those training statistics, and contrast the result with refitting on all six values. Round z-scores to three decimals.
Fit on training data. The training mean is \((1+3+5+7)/4=4\). The squared deviations are \(9,1,1,9\), summing to 20, so the sample standard deviation is \(\sqrt{20/3}=2.582\).
Transform later records without refitting. The value 9 receives z-score \((9-4)/\sqrt{20/3}=1.936\). The value 11 receives z-score \((11-4)/\sqrt{20/3}=2.711\). The saved parameters remain \(\bar{x}_{tr}=4\) and \(s_{tr}=\sqrt{20/3}\).
Compare with refitting. The sum of all six values is 36, so the full-data mean is 6. Their squared deviations from 6 are \(25,9,1,1,9,25\), summing to 70; the full-data sample standard deviation is \(\sqrt{70/5}=\sqrt{14}=3.742\). The later values would then score \((9-6)/\sqrt{14}=0.802\) and \((11-6)/\sqrt{14}=1.336\).
Check. The two procedures produce different coordinates for the same later records because the second procedure uses their values to redefine the center and spread. The correct method is to retain the training-fitted parameters. This example also shows why a stored preprocessing step is part of the predictive workflow, not just a temporary calculation.
Common Exam Traps
- “The test outcomes were not used, so there is no leakage.” Incorrect. Test predictors can still influence preprocessing parameters. For a conventional held-out evaluation, fit the scaler on training predictors only.
- “Scale training and test separately.” This avoids combining the sets, but it gives them different coordinate systems. A test value then does not have the same interpretation as a training value. Apply the training-fitted parameters to both.
- “Every scaled test value must lie between 0 and 1.” This is false for training-fitted min-max scaling. A test value beyond the training range can map outside that interval.
- “A larger test set only changes test scores.” If its statistics are included, it can change the transformation of training observations too. In standardization, both center and spread may change.
The core check is to ask, for every preprocessing parameter: which observations were used to estimate it? If a held-out test observation helped determine a parameter used before evaluation, the test set has influenced the fitted workflow. A clean train-test comparison requires that the answer for scaling parameters be “the training data only.”
Check Your Understanding
For each problem, state which scaling parameters should be used and check whether the proposed calculation lets test data influence preprocessing.
- Training values are \(4,6,8\), and the test value is 10. Using sample standard deviations, find the training-only mean and standard deviation, the training-fitted z-score of 10, and the full-data z-score of 10. Round to three decimals. (Answer: Training mean \(6\), training SD \(\sqrt{4}=2\), training-fitted test z-score \(2.000\); full-data mean \(7\), full-data SD \(\sqrt{20/3}=2.582\), full-data test z-score \(1.162\).)
- Training values are \(5,15,25\), and the test value is 35. Find the test min-max value using training limits and then using the full-data limits. (Answer: Training limits give \(1.500\); full-data limits give \(1.000\).)
- A training set has four observations with predictor mean 12. A test set has two observations with predictor mean 24. Find the full-data mean and the amount by which it exceeds the training mean. (Answer: Full-data mean \(16\); it exceeds the training mean by \(4\).)
- Training values are \(1,3,5,7\), and the test value is 9. Using sample standard deviations, calculate the test z-score using training statistics and using full-data statistics. Round to three decimals. (Answer: Training-fitted score \(1.936\); full-data score \(0.802\).)
- A workflow estimates the training mean and standard deviation, then estimates a second mean and standard deviation from the test predictors before transforming the test records. Is this a valid conventional held-out scaling procedure? State the correction. (Answer: No. Apply the saved training mean and standard deviation to test predictors without refitting.)