Why Preprocessing Parameters Need a Training Set
A transformation rule and the parameters used by that rule are not always the same thing. The formula \(z=(x-c)/s\) specifies how to standardize a predictor, but it does not specify the center \(c\) or scale \(s\). If those values are calculated from observed data, the observations used to calculate them matter.
In a predictive analysis, the training data are used to fit the model and any data-dependent preprocessing. Validation data can be used for a stated model-selection purpose, and test data provide a final assessment, but neither should quietly change the preprocessing that was fitted on the training data. Otherwise, the procedure being assessed has made use of information outside its permitted fitting data.
Some transformations have no data-estimated parameters. For example, \(g(x)=\log(1+x)\) has a fixed formula once the input domain is appropriate. By contrast, standardization requires a center and scale, and a fitted power transformation may require an estimated power. Distinguishing these cases helps identify exactly what must be learned from training data.
When the fitting procedure uses an outcome, such as selecting a predictor transformation by its relationship to the response, the training outcomes may be used. Validation and test outcomes must not be used to estimate that transformation. For an unsupervised operation such as calculating a predictor mean, the training predictor values supply the fit; no outcome is needed.
Fit Once, Then Apply the Stored Values
A reliable process separates parameter estimation from transformation application. First specify which observations are training observations. Next estimate each data-dependent parameter from those observations and record it. Finally apply the recorded values to the training, validation, and test observations as appropriate. The validation and test values may be transformed, but they do not alter the fitted parameters.
Identify the training observations before estimating centers, scales, ranges, or power parameters.
Calculate each parameter using only the data allowed by the preprocessing method. Record the values and the transformation rule.
Use the stored parameters to transform new observations. Do not calculate a new center, scale, range, or power from the validation or test data.
For example, if the training mean is \(\bar{x}_{\text{train}}\) and the training sample standard deviation is \(s_{\text{train}}\), standardize every observation with \(z=(x-\bar{x}_{\text{train}})/s_{\text{train}}\). A validation observation can have a standardized value well above 1 or below -1. That does not, by itself, signal an error: the standardization is anchored to the training distribution.
Worked Example: Standardizing With the Training Mean and Scale
A hypothetical pricing analysis uses a positive predictor measured in tens of units. Its training values are 2, 4, and 6. A validation observation has value 8. Calculate the training mean and sample standard deviation, then standardize all four values using those training estimates. Round only the final standardized values to three decimals.
Method. Fit the standardization parameters on the three training observations, then apply the same formula to the validation observation.
The training mean is \( \bar{x}_{\text{train}}=(2+4+6)/3=4 \). The squared deviations from 4 are \( (2-4)^2=4 \), \( (4-4)^2=0 \), and \( (6-4)^2=4 \). The sample variance is \( (4+0+4)/(3-1)=4 \), so \(s_{\text{train}}=\sqrt{4}=2\).
Using \(z=(x-4)/2\), the training values become \((2-4)/2=-1\), \((4-4)/2=0\), and \((6-4)/2=1\). The validation value becomes \((8-4)/2=2\).
Check. The transformed training values have mean \((-1+0+1)/3=0\) and sample standard deviation \(\sqrt{(1+0+1)/2}=1\). The validation value is 2 because it is two training standard deviations above the training mean. The method avoids the mistake of recalculating the mean and scale after the validation value arrives.
What the Standardization Formula Guarantees
The training-set properties of standardization follow directly from the definition. Suppose the training values are \(x_1,\ldots,x_n\), with \(n>1\), training mean \(\bar{x}\), and positive sample standard deviation \(s\). Define \(z_j=(x_j-\bar{x})/s\) for each training observation.
The last equality holds because \(\sum_{j=1}^{n}x_j=n\bar{x}\). For the sample variance, substitute the definition of \(z_j\):
This follows from the sample variance definition \(s^2=\sum_{j=1}^{n}(x_j-\bar{x})^2/(n-1)\), provided \(s>0\). These results apply to the training observations used to calculate \(\bar{x}\) and \(s\); they are not guarantees about validation or test values. If \(s=0\), this standardization formula is undefined, so the predictor needs an explicit handling rule rather than division by zero.
Other Parameters Must Be Stored Too
The same fit-then-apply logic applies to transformations beyond z-scores. A min-max transformation, for example, can use the training minimum \(a\) and maximum \(b\) to map \(x\) to \((x-a)/(b-a)\), provided \(b>a\). The rule maps training values into the interval from 0 to 1. It does not guarantee that new values fall in that interval. Do not silently clip a new value unless clipping was specified as part of the method.
Worked Example: Applying Training Range Endpoints
A hypothetical claims model uses a continuous predictor. Its training values are 10, 20, 30, and 40. A validation observation has value 50. Fit a min-max transformation using the training range and calculate the transformed values. Round only the final results to three decimals.
Method. Store the training minimum \(a=10\) and maximum \(b=40\), then apply \(g(x)=(x-a)/(b-a)\) to each value.
The denominator is \(40-10=30\). The training values transform to \((10-10)/30=0\), \((20-10)/30=1/3\), \((30-10)/30=2/3\), and \((40-10)/30=1\). To three decimals, these are 0.000, 0.333, 0.667, and 1.000.
The validation value transforms to \((50-10)/30=40/30=4/3\), or 1.333. It is above 1 because 50 exceeds the training maximum. The stored training range remains 10 to 40; replacing the maximum by 50 would change the transformation after fitting. The method avoids treating the target interval as a clipping rule.
Fitting a Power Parameter
A power transformation can also have a data-estimated parameter. For positive \(x\), the Box-Cox transformation is \(g_{\lambda}(x)=(x^{\lambda}-1)/\lambda\) when \(\lambda\ne0\), and \(g_0(x)=\ln(x)\). The value of \(\lambda\) is not supplied by the formula; it must be fixed in advance or estimated by a stated fitting procedure. Once estimated, it is stored and applied unchanged to new positive observations.
Different fitting procedures can use different criteria. The following example deliberately uses a small candidate set and training regression residual sum of squares to illustrate the fit/apply boundary. It is not a claim that this candidate-grid rule is the general Box-Cox estimation procedure. The key requirement is that the stated criterion uses training data only.
Worked Example: Selecting a Power From Training Data
A hypothetical service team models an outcome \(y\) from a positive predictor \(x\). The four training pairs are \((1,1),(2,2),(4,3),(8,4)\). Compare the Box-Cox candidates \(\lambda=0\) and \(\lambda=1\), fitting a simple linear regression with an intercept for each candidate. Select the candidate with lower training residual sum of squares. Then transform a new predictor value \(x=16\) using the selected power. Give the new transformed value to four decimals.
Method. Transform the training predictor under each candidate, fit the intercept-and-slope regression, and compare training residual sums of squares. Do not use the new predictor to choose \(\lambda\).
For \(\lambda=1\), \(g_1(x)=x-1\), so the transformed training predictors are 0, 1, 3, and 7. Because this is \(x\) shifted by a constant, a regression with an intercept has the same fitted values and residual sum of squares as regression on \(x=(1,2,4,8)\). For that regression, \(\bar{x}=15/4=3.75\) and \(\bar{y}=10/4=2.5\). The sum of squared deviations in \(x\) is \(85-4(3.75)^2=28.75\), and the cross-product sum is \(49-4(3.75)(2.5)=11.5\). Thus the slope is \(11.5/28.75=0.4\) and the intercept is \(2.5-(0.4)(3.75)=1\). Predictions are 1.4, 1.8, 2.6, and 4.2, giving residuals \(-0.4,0.2,0.4,-0.2\). The training residual sum of squares is \(0.16+0.04+0.16+0.04=0.40\).
For \(\lambda=0\), \(g_0(x)=\ln(x)\). The training values satisfy \(y=1+\ln(x)/\ln(2)\) exactly: this gives 1 at \(x=1\), 2 at \(x=2\), 3 at \(x=4\), and 4 at \(x=8\). A regression with an intercept therefore has zero residuals and training residual sum of squares 0. Since \(0<0.40\), the stated rule selects \(\lambda=0\).
Apply the stored power to \(x=16\): \(g_0(16)=\ln(16)=4\ln(2)\), approximately 2.7726. Check. The selected transformation is the one with the smaller training criterion, and the new value was transformed using that selected \(\lambda\), not used to reselect it. This example illustrates a training-only selection rule; its perfect fit on four observations does not establish that the transformation will generalize well.
Common Mistakes and the Evaluation Boundary
A common mistake is to pool training and validation observations before calculating a preprocessing parameter, on the grounds that more observations should give a more stable estimate. That changes the fitted preprocessing rule using information from the validation set. A related mistake is to calculate one set of parameters for training data and a different set for test data. In that case, the model receives predictors on different scales at fitting and scoring time.
For the training values 2, 4, and 6 in the first example, the correct training mean and sample standard deviation are 4 and 2. The validation value is 8, so its correct standardized value is \((8-4)/2=2\). The incorrect pooled calculation treats all four values as a single fitting set: its mean is \((2+4+6+8)/4=5\), and its sample standard deviation is \(\sqrt{(9+1+1+9)/3}=\sqrt{20/3}\), approximately 2.582. The resulting value for 8 is \((8-5)/2.582\), approximately 1.162, not 2. The error is not an arithmetic slip; it is using a different, validation-informed fit.
Cross-validation requires the same discipline. Within each fold, estimate preprocessing parameters from that fold's training portion and apply them to that fold's held-out portion. If parameters are first estimated using the entire data set and then cross-validation is performed, the held-out observations have already influenced the inputs being evaluated. The reported assessment then does not represent the full procedure being tested. After choosing a final procedure, fit preprocessing parameters on the designated full training set and store them for later scoring.
The operational record should make the fitted rule reproducible: include the transformation, each estimated parameter, the training data or fitting split used, and any explicit behavior for values outside an observed range or for an undefined case. This does not require validation and test values to resemble training values. It requires the transformation applied to each value to be the one that was actually fitted.
Check Your Understanding
For each problem, identify the fitting data and apply the stated rule without refitting on held-out observations.
- A training predictor has values 3, 5, and 7, and a validation value is 9. Using the training sample mean and sample standard deviation, what is the standardized validation value? (Answer: 2)
- A min-max transformation is fitted to training values 4, 8, and 12. What transformed value results for a test observation of 16, using the training endpoints and no clipping? (Answer: 1.500)
- A positive predictor has training values 1, 3, and 9. A stored Box-Cox power is \(\lambda=1\). What is the transformed value for a new observation of 10? (Answer: 9)
- A four-observation training set has mean 12 and sample standard deviation 3. A held-out observation is 21. Calculate its standardized value using the stored training parameters. What is the incorrect source of an answer obtained after recomputing the mean and standard deviation from all five observations? (Answer: 3; the incorrect answer uses held-out data to refit the scaler.)
- In a five-fold cross-validation procedure, a predictor's median is calculated once from all observations before the folds are formed. Is that median fit restricted to each fold's training portion? What should be done instead? (Answer: No; calculate and store the median separately within each fold's training portion, then apply it to that fold's held-out portion.)