Transforming Validation and Test Records

Why Held-Out Records Need a Fixed Transformation

A predictive model is fitted using a particular set of predictor definitions. If training records were standardized, assigned to clusters, or projected onto principal components, then the model learned relationships with those transformed predictors—not with an arbitrary alternative version of them. Validation and test records must therefore be transformed using the quantities estimated from the training data.

This is an application step, not a new fitting step. For a held-out record, use the training-fitted imputation values, centers, scales, cluster rule, and PCA loadings. Do not recalculate those quantities from the validation or test predictors. Doing so can change the meaning and scale of the derived features, even when the response variable is not used.

Definition: A training-fitted transformation is a rule whose parameters are estimated from the training predictors and then held fixed when applied to other records. Examples include an imputer with training-estimated replacement values, a standardization rule with training means and standard deviations, a clustering rule with fixed cluster centers, and a PCA projection with fixed centering and loading vectors.

In a validation workflow, fit the transformations using the analysis portion of the training data, apply them to the assessment portion, and then evaluate the predictive model. For a final test set, fit the transformations using the designated training data and apply them to the test predictors. In either case, the held-out records receive the transformation; they do not determine it.

This distinction is especially important for unsupervised features. Clustering and PCA do not require the response variable, but their fitted centers, scales, or directions still depend on the predictor records used to estimate them. The earlier material on foldwise clustering and foldwise PCA established the corresponding rule within resampling: each held-out fold is transformed using its training-fold fit.

The Transformation Is a Sequence of Fixed Operations

A derived-feature pipeline can include several operations in a specified order. For example, missing values may first be imputed, the resulting predictors may then be standardized, and the standardized values may be used both for cluster assignment and for PCA scores. Each operation must use the version and parameters established during training.

Check the input schema.
Confirm that the held-out record has the required predictors in the expected units and formats. Apply the training-defined treatment for missing values and factor levels.

Apply training-fitted preprocessing.
Use training-estimated imputation values, means, and scales. Do not estimate replacements or scaling parameters from the held-out records.

Create the derived features.
Assign clusters using the fitted clustering rule and calculate PCA scores using the fitted centering and loading vectors.

Match the model's input columns.
Supply the downstream model with the same feature definitions, names, and order used during fitting, then obtain its prediction.

The order is substantive. If clustering was fitted on standardized predictors, the record must be standardized with the corresponding training means and scales before its cluster is assigned. If PCA was fitted after imputation and standardization, its scores must be calculated from predictors processed in that same way. A different order generally produces different inputs.

Applying a Training-Fitted PCA Projection

Let a record have predictor vector \(x\). Suppose the training procedure stored the centering vector \(m_{\mathrm{PCA}}\) and a matrix \(V\) whose columns are the retained PCA loading vectors. If scaling was part of the training preprocessing, let \(\mu_{\mathrm{tr}}\) and \(s_{\mathrm{tr}}\) be the training means and standard deviations used for each predictor. The held-out record is standardized using those training quantities, then centered as required by the PCA fit.

z_j=\frac{x_j-\mu_{\mathrm{tr},j}}{s_{\mathrm{tr},j}}, \qquad t=(z-m_{\mathrm{PCA}})^\mathsf{T}V

Here, \(t\) is the row of retained component scores. If PCA was fitted directly to unscaled, centered predictors, the appropriate training-fitted transformation is instead \(t=(x-\mu_{\mathrm{PCA}})^\mathsf{T}V\), with no standardization step. The formula must match the actual training pipeline. For example, when PCA is fitted to standardized training predictors, those predictors are already centered, so \(m_{\mathrm{PCA}}\) is usually the zero vector, up to numerical rounding.

Why the PCA score formula works: A principal component is a projection of a centered predictor vector onto a loading direction. For a unit loading vector \(a\), the variance of the projected training values is \(a^\mathsf{T}Sa\), where \(S\) is the training covariance matrix. Maximizing this variance subject to \(a^\mathsf{T}a=1\) gives the eigenvector equation \(Sa=\lambda a\). Once that direction \(a\) has been estimated, the score for any new centered record is its projection, \((z-m_{\mathrm{PCA}})^\mathsf{T}a\). The new record is projected onto the existing direction; it does not estimate a new direction.

Worked Example: Calculating PCA Scores for a Validation Record

A training set contains two predictors, measured in their original units. The training means are \(\mu_{\mathrm{tr}}=(10,20)\), and the training standard deviations are \(s_{\mathrm{tr}}=(2,4)\). PCA was fitted to the standardized training predictors, whose PCA center is \(m_{\mathrm{PCA}}=(0,0)\). The retained loading vectors are \(v_1=(0.8,0.6)\) and \(v_2=(-0.6,0.8)\). A validation record has \(x=(14,16)\). Calculate its two scores. The method is to apply the saved training standardization and project onto the saved loadings.

Step 1: Standardize using the training values. For the first predictor, \(z_1=(14-10)/2=4/2=2\). For the second, \(z_2=(16-20)/4=-4/4=-1\). Thus \(z=(2,-1)\).

Step 2: Project onto the first loading. The first score is \(t_1=2(0.8)+(-1)(0.6)=1.6-0.6=1.0\).

Step 3: Project onto the second loading. The second score is \(t_2=2(-0.6)+(-1)(0.8)=-1.2-0.8=-2.0\). The validation record's retained score vector is therefore \((1.0,-2.0)\).

Check. The loading vectors are orthogonal because \(0.8(-0.6)+0.6(0.8)=0\), and each has squared length \(0.8^2+0.6^2=1\). Since both components are retained, reconstructing the standardized record gives \(1.0(0.8,0.6)+(-2.0)(-0.6,0.8)=(2,-1)\), as required. This check confirms the projection arithmetic. A common mistake is to calculate means or standard deviations from validation records instead; that would use a different coordinate system from the one used to fit the PCA and downstream model.

Assigning a Held-Out Record to Existing Clusters

For a clustering method that assigns a record to its nearest fitted center, assignment uses the predictor representation and distance rule established during training. In standardized Euclidean clustering, for example, first transform the held-out predictors with the training-fitted means and scales, then calculate its distance to each stored center. The assigned cluster is the center with the smallest distance. The centers are not recalculated to accommodate the new record.

d_k^2(z)=\sum_{j=1}^{p}(z_j-c_{kj})^2, \qquad \widehat{k}(z)=\underset{k}{\operatorname{arg\,min}}\ d_k^2(z)

Here, \(c_k\) is the stored center for cluster \(k\), expressed in the same predictor coordinates as \(z\). If training used a different distance measure or a different representation, use that fitted rule instead. A tie requires a predetermined tie-handling rule; it should not be resolved by refitting the clustering on held-out records.

Worked Example: Assigning a Validation Record to a Fitted Cluster

A hypothetical training procedure standardized two predictors using training means \((50,40)\) and standard deviations \((10,5)\), then fitted a three-center clustering rule. Its stored centers in standardized coordinates are \(c_1=(0,0)\), \(c_2=(2,1)\), and \(c_3=(-1,2)\). A validation record has original measurements \((65,45)\). Assign it to a cluster using squared Euclidean distance. The method is to use the training scaling and compare distances to the fixed centers.

Step 1: Standardize. The record becomes \(z_1=(65-50)/10=1.5\) and \(z_2=(45-40)/5=1\), so \(z=(1.5,1)\).

Step 2: Calculate each squared distance. To \(c_1\), \(d_1^2=(1.5-0)^2+(1-0)^2=2.25+1=3.25\). To \(c_2\), \(d_2^2=(1.5-2)^2+(1-1)^2=0.25+0=0.25\). To \(c_3\), \(d_3^2=(1.5-(-1))^2+(1-2)^2=2.5^2+(-1)^2=6.25+1=7.25\).

Step 3: Select the nearest center. The smallest squared distance is \(0.25\), so the record is assigned to cluster 2. If the predictive model was trained using one indicator per cluster, the corresponding feature vector is \((0,1,0)\), in the established order for clusters 1, 2, and 3.

Check. The unsquared distances are \(1.8028\), \(0.5000\), and \(2.6926\), respectively, so they give the same nearest center. Squared and unsquared Euclidean distances have the same ordering because squaring is increasing for nonnegative distances. A common mistake is to recalculate the centers using validation records; that changes the fitted partition rather than assigning records to it.

Keeping the Whole Feature Pipeline Consistent

The fitted transformation includes more than a PCA loading matrix or a set of cluster centers. If training used an imputation rule, scaling, or a particular encoding, those steps are part of the map from raw records to model inputs. Likewise, a cluster label used as a categorical predictor needs consistent indicator columns, and a PCA score needs the component definition and sign used during model fitting. Keep the output columns aligned with the downstream model's training inputs.

The following example combines imputation, standardization, one PCA score, a cluster assignment, and a prediction calculation. It illustrates how a missing value is handled by the training-defined rule before either derived feature is calculated.

Worked Example: Applying a Combined Transformation

A hypothetical training procedure uses two predictors, \(x_1\) and \(x_2\). It stores imputation values \(8\) and \(4\), respectively; after applying its training preprocessing, it standardizes with means \((8,4)\) and standard deviations \((2,1)\). A PCA score uses loading \(v_1=(0.6,0.8)\) and center \((0,0)\). Separately, the clustering rule uses the same standardized coordinates and has centers \(c_A=(0,0)\) and \(c_B=(1,0)\). The fitted downstream linear prediction is \(\widehat{y}=2+0.5t_1+1.2I_B\), where \(I_B=1\) for cluster B and \(0\) otherwise. A validation record has \(x_1=10\) and a missing \(x_2\). Calculate its derived features and prediction.

Step 1: Impute with the training values. Replace the missing \(x_2\) with \(4\). The completed record is \((10,4)\). The imputation value is not estimated from this validation record or from other validation records.

Step 2: Standardize. The coordinates are \(z_1=(10-8)/2=1\) and \(z_2=(4-4)/1=0\). Thus \(z=(1,0)\).

Step 3: Calculate the PCA score. \(t_1=(1-0)(0.6)+(0-0)(0.8)=0.6+0=0.6\).

Step 4: Assign the cluster. The squared distance to A is \((1-0)^2+(0-0)^2=1\). The squared distance to B is \((1-1)^2+(0-0)^2=0\). Therefore the record is assigned to B and \(I_B=1\).

Step 5: Calculate the downstream prediction. \(\widehat{y}=2+0.5(0.6)+1.2(1)=2+0.3+1.2=3.5\).

Check. The transformed input consists of a PCA score of \(0.6\) and a cluster-B indicator of \(1\); substituting both into the stated fitted equation gives \(3.5\). A common mistake is to omit the imputation step and allow the missing value to produce an undefined standardized value, PCA score, or distance. The correction is to apply the specified training-fitted replacement before those calculations.

Common Errors and Their Corrections

The central check is whether every held-out feature is expressed in the same coordinate system and with the same definition as its training counterpart. The following errors can violate that requirement:

Re-estimating the center or scale. Use training means and standard deviations, not values calculated from the validation or test records.
Re-fitting PCA. Use the fitted center and loadings. A new PCA fit can rotate, reorder, or change the signs of components, so its scores need not match the training features.
Re-fitting the clusters. Assign records to the existing centers under the fitted distance rule; do not move centers to include held-out records.
Changing the order of operations. Apply imputation, scaling, and projection in the same order used during training.
Changing the model-input schema. Keep indicator columns, component names, and column order consistent. A cluster absent from one held-out batch still needs its established column, with the appropriate value for each record.

A quick diagnostic is to inspect a few records from the training data after applying the stored transformation. When the identical training preprocessing is applied, their derived features should reproduce the training features, subject only to expected numerical rounding. This is a useful implementation check; it does not replace the rule that held-out records must use the training-fitted parameters.

Key takeaway: Fit the transformation on training data, then apply that fixed transformation to validation or test predictors. PCA scores are projections onto stored loading vectors, and cluster features come from applying a stored assignment rule. The derived columns supplied to the predictive model must retain the definitions and layout used when that model was fitted.

Check Your Understanding

For each problem, apply only the stated training-fitted quantities to the held-out record.

A training transformation has mean \(12\) and standard deviation \(3\) for one predictor. A test value is \(18\). What standardized value should be supplied to a fitted PCA? (Answer: \(2\))
A centered two-predictor PCA has loading \(v=(0.6,0.8)\). After training-fitted preprocessing, a validation record is \(z=(1,-2)\). What is its score on this component? (Answer: \(0.6(1)+0.8(-2)=-1.0\))
A clustering rule has fixed centers \(c_1=(0,0)\) and \(c_2=(2,0)\). A held-out record, already in the fitted coordinate system, is \(z=(1.6,1)\). Calculate the squared distances and its assignment. (Answer: \(d_1^2=1.6^2+1^2=3.56\), \(d_2^2=(-0.4)^2+1^2=1.16\); assign to cluster 2.)
A fitted pipeline imputes \(x_1\) with \(5\), then standardizes using mean \(5\) and standard deviation \(2\). A record has missing \(x_1\). What value enters the next step of the pipeline? (Answer: Impute to \(5\), then standardize to \((5-5)/2=0\).)
A fitted model predicts \(\widehat{y}=1+2t+0.5I_B\). A held-out record has PCA score \(t=-0.5\) and is assigned to cluster B. What is its prediction? (Answer: \(1+2(-0.5)+0.5(1)=0.5\).)

What You'll Learn