Why a Transformation Pipeline Must Be Training-Only
Scaling, clustering, and principal component analysis (PCA) can turn raw predictors into features on a different scale or representation. But a predictive model is evaluated on observations that were not used to fit it. If a transformation is estimated using those evaluation observations, the evaluated procedure has indirectly used information from data it was meant to hold out.
The solution is to distinguish fitting a transformation from applying it. Fitting estimates and stores the quantities needed to define the transformation using training data. Applying uses those stored quantities, unchanged, to transform training, validation, test, or future records. A new record can affect its own transformed value, but it must not change the fitted parameters used to calculate that value.
This distinction matters even when a transformation does not use the response variable. For example, using the predictor values in a test set to calculate a mean is still a change to the training procedure. It may seem harmless because no test responses were used, but it gives the pipeline information about the test population’s predictor distribution. In a standard held-out evaluation, that information is unavailable when the model is trained. The procedure being evaluated should therefore fit transformations from training data alone.
Specify the Fit and Apply Stages
A useful specification names the input variables, the operations and their order, the data used to estimate each operation, and the parameters that must be retained. The fitted objects are part of the predictive model: saving only the final regression or classification model is not enough to reproduce its predictions.
Specify the columns used, the method, and any choices such as a retained number of components or a chosen number of clusters. Make these choices using training data or a training-only validation process.
Estimate the transformation parameters from that portion only. Keep a record of the parameters and the order of operations.
Use the saved parameters to transform observations outside the training portion. Do not recalculate them from validation, test, or incoming records.
Supply the transformed training features to the predictive model during its fit, and supply transformed validation or future features to that same fitted model at prediction time.
The sequence itself is part of the specification. If variables are first centered and scaled and then passed to PCA, that order must be repeated when new observations are processed. Likewise, the same input columns, units, and column order must be used. These controls make the pipeline reproducible; they do not guarantee that a chosen transformation improves prediction.
Scaling: Save the Training Location and Scale
For a quantitative variable \(X\), suppose scaling uses the training sample mean \(\bar{x}_{\text{train}}\) and sample standard deviation \(s_{\text{train}}\). The transformed value for an observation \(x\) is
Both quantities are calculated from training data, then saved. In particular, the standard deviation for a new observation is not recalculated from that observation or from the set of observations arriving with it. The same rule applies if a specified procedure uses a different scale statistic: save the training estimate and use it as fitted.
Worked Example: Applying a Training-Set Standardization
A hypothetical pricing model uses one predictor, the number of annual service visits. Its three training observations are 2, 4, and 6 visits. The sample mean and sample standard deviation are used for standardization. A held-out observation has 10 visits. Find its standardized value, and show why recalculating the transformation with the held-out value is a mistake.
Method. Fit the standardization on the three training values, then apply the stored training mean and standard deviation to the held-out value.
The training mean is \(\bar{x}_{\text{train}}=(2+4+6)/3=4\). The sample variance is \(((2-4)^2+(4-4)^2+(6-4)^2)/(3-1)=(4+0+4)/2=4\), so \(s_{\text{train}}=\sqrt{4}=2\). The fitted transformation is therefore \(z=(x-4)/2\).
For the held-out observation, \(z=(10-4)/2=6/2=3\). The correct standardized value is 3.
Common mistake and correction. If the held-out observation is incorrectly included when estimating the scaling parameters, the four-value mean becomes \((2+4+6+10)/4=5.5\). The sum of squared deviations is \(12.25+2.25+0.25+20.25=35\), so the four-value sample standard deviation is \(\sqrt{35/3}\approx3.416\). This gives \((10-5.5)/3.416\approx1.317\), not 3. That calculation has changed the fitted transformation using evaluation data. The correction is to retain the training values 4 and 2 and apply them unchanged.
As a check, applying the fitted transformation to the training values gives \(-1,0,1\); the held-out value 10 is three training standard deviations above the training mean. The training-based result is internally consistent.
Clustering: Keep the Fitted Centers Fixed
A clustering method fitted to training predictors produces a representation of the training data, such as a set of centers. If a later step assigns new observations to their nearest center, the centers used for that assignment must be the training-fitted centers. Re-running clustering on a combined training and validation set is not the same operation: the centers may move, and the meaning of the resulting assignments can change.
For a nearest-center assignment with predictor vector \(x\) and fitted centers \(c_1,\ldots,c_K\), the assigned cluster is the index \(j\) minimizing the chosen distance \(d(x,c_j)\). Under Euclidean distance, it is equivalent to minimize squared Euclidean distance because squaring is an increasing function for nonnegative distances. This assignment uses the fixed centers; it does not estimate new ones.
Worked Example: Assigning New Records to Fitted Centers
A hypothetical equipment-maintenance analysis uses two training predictors: hours of use and months since the last inspection. A clustering procedure fitted on the training records has produced two centers, \(c_1=(1,0)\) and \(c_2=(5,0)\), in the specified predictor scale. Assign new record \(A=(4,2)\) using Euclidean distance. Then explain what to do with a second record \(B=(2,1)\).
Method. Keep both fitted centers fixed and compare each new record’s squared Euclidean distance to them.
For record \(A\), the squared distance to \(c_1\) is \((4-1)^2+(2-0)^2=9+4=13\). Its squared distance to \(c_2\) is \((4-5)^2+(2-0)^2=1+4=5\). Since \(5<13\), record \(A\) is assigned to the cluster with center \(c_2\).
For record \(B\), the squared distance to \(c_1\) is \((2-1)^2+(1-0)^2=1+1=2\). Its squared distance to \(c_2\) is \((2-5)^2+(1-0)^2=9+1=10\). Since \(2<10\), record \(B\) is assigned to the cluster with center \(c_1\).
The checks are direct: each record is assigned to the nearer of the two stored centers. The common mistake would be to refit the clustering procedure after adding \(A\) and \(B\), then use potentially shifted centers. That would answer a different question and would not apply the training-fitted transformation.
Cluster names such as “cluster 1” and “cluster 2” are labels, not inherent descriptions of the groups. For reproducibility, preserve the fitted center ordering or another fixed mapping from centers to labels. A later tutorial considers how cluster assignments may be encoded as predictors; the point here is only that their assignments must be generated by the same fitted clustering transformation.
PCA: Save the Centering, Scaling, and Loadings
PCA represents observations using directions estimated from a set of training predictors. If the training procedure centers the variables, let \(\bar{x}_{\text{train}}\) be the vector of training means. If it also scales them, let \(s_{\text{train}}\) be the vector of training standard deviations. For a new row vector \(x\), the transformed vector before projection is \(z\), with entries obtained by centering and, when specified, scaling using those training quantities. If \(V_q\) is the matrix of the selected \(q\) loading vectors, its component scores are
The loading vectors and the rules for centering and scaling are fitted once on training data. A validation or future observation is projected onto those same directions. Recomputing PCA on the new data can rotate or reorder the directions, so its scores would no longer be produced by the training-fitted transformation.
Worked Example: Projecting a New Record onto a Fixed Component
A hypothetical training set has two unscaled predictors, \(X_1\) and \(X_2\), with training means 2 and 2. A one-component PCA fitted on the training data has first loading vector \(v_1=(1/\sqrt{2},1/\sqrt{2})\). The specified procedure centers but does not scale the predictors. Find the first-component score for a new record \(x=(4,2)\).
Method. Subtract the saved training means from the new record, then take the dot product with the saved loading vector.
The centered record is \(z=(4-2,2-2)=(2,0)\). Its score is
The calculation uses no values from other validation records. As a check, the score is the projection of the centered point \((2,0)\) onto a unit direction at equal angles to the two coordinate axes; the result \(\sqrt{2}\) is consistent with that projection.
Common mistake and correction. Centering the new record using a mean recalculated from a validation batch would change \(z\), and hence its score. The correct score for this fitted pipeline uses the training means 2 and 2 and the stored loading vector, even if the validation batch has a different average.
Why the Transformation Must Stay Fixed
The reason can be stated directly. Let \(g_{\hat{\theta}}(x)\) denote a transformation with fitted parameter \(\hat{\theta}\). Training produces \(\hat{\theta}_{\text{train}}\), so the prediction procedure for a new observation uses \(g_{\hat{\theta}_{\text{train}}}(x_{\text{new}})\). If evaluation observations are included in fitting, the result instead uses \(g_{\hat{\theta}_{\text{combined}}}(x_{\text{new}})\). In general, \(\hat{\theta}_{\text{combined}}\ne\hat{\theta}_{\text{train}}\), and therefore the transformed inputs need not be equal. This is not merely a different numerical implementation; it is a different fitted procedure.
For standardization, the difference is explicit:
The equality can happen in a special data set, but that does not justify refitting. The procedure must be specified before observing evaluation results and applied consistently, rather than relying on accidental equality in one sample.
In cross-validation, this rule applies separately inside each fold. Fit the transformation on that fold’s analysis portion; apply it to the fold’s assessment portion; then evaluate predictions. If transformation parameters are estimated once from all observations before cross-validation, each assessment fold has influenced its own transformation. That makes the validation procedure optimistic or otherwise unrepresentative of the intended train-then-predict process. Choices such as the number of retained components or clusters also need to be made without using the final test set.
In R, the conceptual separation is important even when a function automates it. For example, fitting PCA with prcomp on the training predictors creates a fitted object containing the centering, any scaling, and rotation information; applying predict with that fitted object projects new rows using those stored quantities. The key is to fit the object on training data and pass new data to its application method, rather than calling the fitting procedure again on new data. Similarly, save a scaling vector or a clustering fit’s centers along with the predictive model.
Reproducibility and Exam Pitfalls
A saved pipeline should make it possible to reconstruct exactly how a row became a model input. Record the predictor names and order, units, transformation order, fitted parameters, selected component count or cluster count, and the software method used. Also retain the training-derived object rather than only a report of its output. If records are processed later, the same stored parameters should reproduce the same transformed values.
- Refitting on the test set: a test set is for evaluating the fitted procedure, not for recalculating its transformation parameters.
- Mixing parameter sources: using training means but validation standard deviations does not apply one fitted standardization.
- Changing input order: a PCA loading for one column is not interchangeable with a loading for another column.
- Fitting outside cross-validation folds: even an unsupervised transformation can carry information from an assessment fold into the fitted pipeline.
- Confusing transformation with prediction: a transformation creates inputs; it does not by itself establish that those inputs improve predictive performance.
Check Your Understanding
For each problem, distinguish the parameters fitted on training data from the calculations applied to new observations.
- A training predictor has values 1, 3, and 5. Standardize using the training sample mean and sample standard deviation. What standardized value is assigned to a new observation of 7? (Answer: 2)
- A transformation uses the training sample mean and sample standard deviation. Training values are 10 and 20; a new value is 25. What is its standardized value, rounded to four decimals? (Answer: 1.4142)
- A fitted two-center clustering transformation has centers \((0,2)\) and \((4,2)\). Using Euclidean distance, assign the new point \((3,4)\) to the nearer center. (Answer: The second center, since the squared distances are 13 and 5.)
- A centered, unscaled PCA has training means \((1,2)\) and a saved loading vector \((0.6,0.8)\). What is the first-component score for new point \((4,6)\)? (Answer: 5.0)
- In a cross-validation fold, a candidate procedure calculates scaling parameters once from the entire data set and then fits its predictive model on the fold’s analysis portion. State the correction. (Answer: Fit the scaling parameters on the fold’s analysis portion only, apply them unchanged to that fold’s assessment portion, and repeat within each fold.)