Why Cluster Features Need Fold-Safe Construction
Clustering can summarize a complicated set of predictors into a segment label, a set of indicators, or distances from cluster centers. A supervised model can then use those features to predict an outcome, such as whether a hypothetical insurance application will require additional review. But the cluster representation is itself learned from data. If validation observations help determine that representation, validation is no longer a clean assessment of a model built without them.
The key rule is to treat scaling and clustering as fitted parts of the predictive procedure. For each split, fit them on the training portion, then apply the fitted transformations to the held-out portion. The held-out outcomes must not be used to construct features, and its predictors must not determine the scaler or the clusters when the intended use is prediction for new observations.
Constructing Cluster Features in a Training Fold
For a training fold, let \(x_{ij}\) be the value of predictor \(j\) for observation \(i\). If predictors are to be standardized, calculate each predictor's mean and standard deviation from the training fold only. For definiteness, the examples here use the population-form standard deviation within that fold:
The same training-fold mean and standard deviation are then applied to held-out observations. For example, a held-out value \(x_{hj}\) is standardized as \((x_{hj}-\bar{x}_{j,\mathrm{train}})/s_{j,\mathrm{train}}\), not by recalculating either quantity using the held-out data.
For \(k\)-means, fit the centers using the standardized training observations. A new observation can be assigned to its closest fitted center using the same distance measure used by the clustering method. Its cluster label can be represented by one indicator for each cluster, or its distances to the centers can be used as predictors. Whatever representation is chosen must be applied consistently to training and held-out observations.
Worked Example: Standardize and Assign a Held-Out Risk Score
A hypothetical training fold contains four applicants, with a single numeric predictor \(x\) representing a risk score: \(0, 2, 8, 10\). A \(k\)-means model with \(k=2\) is fitted after standardization. A held-out applicant has score \(x=4\). Find the training-fold scaling values, the two fitted centers, and the held-out applicant's cluster.
Method and training mean. Use only the four training scores. Their mean is \((0+2+8+10)/4=5\).
Training standard deviation. The squared deviations from 5 are \(25,9,9,25\). Their sum is \(68\), so the population-form variance is \(68/4=17\), and \(s_{\mathrm{train}}=\sqrt{17}\approx4.1231\).
Standardized training scores. The four values become \(-5/\sqrt{17},-3/\sqrt{17},3/\sqrt{17},5/\sqrt{17}\), approximately \(-1.2127,-0.7276,0.7276,1.2127\). The two-cluster solution groups the first two and last two values. Their centers are respectively \((-5/\sqrt{17}-3/\sqrt{17})/2=-4/\sqrt{17}\) and \((3/\sqrt{17}+5/\sqrt{17})/2=4/\sqrt{17}\).
Transform and assign the held-out applicant. Apply the training-fold scaling: \(z_h=(4-5)/\sqrt{17}=-1/\sqrt{17}\). Its distance from the first center is \(|-1/\sqrt{17}-(-4/\sqrt{17})|=3/\sqrt{17}\). Its distance from the second is \(|-1/\sqrt{17}-4/\sqrt{17}|=5/\sqrt{17}\). Since \(3/\sqrt{17}<5/\sqrt{17}\), assign the applicant to the first cluster. A two-indicator feature is therefore \((1,0)\), if the first cluster is listed first.
Check and common mistake. The applicant's score did not affect the mean, standard deviation, or centers. Recalculating the scaler after adding the score would allow the held-out predictor to influence its own representation, which is not the intended inductive evaluation.
Why the Validation Data Must Not Fit the Clusters
A validation set is intended to mimic observations not used to build the predictive procedure. If its predictors are included while scaling or clustering, the resulting representation depends on the validation sample. Even if the outcome values are hidden, the validation predictors have influenced the model-building process. In particular, a validation point can shift a mean or standard deviation, move a center, or change which training observations share a cluster.
This contamination does not guarantee that every individual validation score will improve. Its direction can vary in a particular split. The problem is that the validation score no longer measures the specified procedure—one fitted without those observations—and can be optimistic because the representation has adapted to the evaluation sample. The risk is greater if validation results are also used to choose the number of clusters, scaling method, or feature representation.
Worked Example: Quantifying Full-Data Scaling Leakage
Suppose a training fold has predictor values \(0,2,8,10\), and a held-out observation has value \(20\). The predictor is one coordinate of a multivariable clustering problem. Calculate its scaling under the correct training-only rule and under the incorrect rule that pools training and held-out values.
Correct training-only calculation. The training mean is \(5\). The squared deviations are \(25,9,9,25\), so the training variance under the stated population-form convention is \(68/4=17\), and the standard deviation is \(\sqrt{17}\approx4.1231\). The training value 10 therefore standardizes to \((10-5)/\sqrt{17}\approx1.2127\).
Incorrect pooled calculation. The mean of all five values is \((0+2+8+10+20)/5=8\). The squared deviations from 8 are \(64,36,0,4,144\), summing to \(248\). The pooled variance is \(248/5=49.6\), and its standard deviation is \(\sqrt{49.6}\approx7.0427\). Under this incorrect scaler, the training value 10 becomes \((10-8)/\sqrt{49.6}\approx0.2840\), while the held-out value 20 becomes \((20-8)/\sqrt{49.6}\approx1.7044\).
Interpretation and check. The training observation 10 has changed from approximately \(1.2127\) to \(0.2840\) solely because the held-out value was included in preprocessing. In a multivariable distance calculation, changing one coordinate's scale can change distances between observations and hence fitted cluster membership. The one-dimensional example itself does not establish a changed assignment: multiplying all distances in one dimension by the same positive constant preserves their order. The leakage is still present because the fitted transformation used information unavailable in the training fold.
Common mistake avoided. It is incorrect to argue that clustering is unsupervised and therefore cannot leak information. A validation outcome is not needed for its predictors to alter the fitted representation.
From Clusters to a Supervised Prediction
Once training observations have cluster-derived features, fit the supervised model using those features and the training outcomes. For example, a regression model can use cluster indicators as predictors, or a classification model can use cluster membership alongside the original predictors. Held-out observations receive cluster-derived features from the training-fitted scaler and clustering model, after which the fitted supervised model produces predictions.
For \(k\)-means, a held-out observation is commonly assigned to the nearest fitted center. This gives a defined rule for new observations. Hierarchical clustering does not, by itself, always supply a general out-of-sample assignment rule: the analyst must specify one, such as a rule based on fitted representatives, before evaluating prediction. Re-clustering the combined training and validation data is not an out-of-sample assignment rule.
The derivation behind the nearest-center assignment follows directly from the \(k\)-means objective. For a fixed cluster \(C\), the sum of squared distances from its observations \(z_i\) to a candidate center \(c\) is minimized by their arithmetic mean. To see why, let \(\bar{z}_C\) be the cluster mean and write \(z_i-c=(z_i-\bar{z}_C)+(\bar{z}_C-c)\). Summing squared Euclidean distances gives:
because the cross term is zero: \(\sum_{i\in C}(z_i-\bar{z}_C)=0\). The second term is nonnegative and is zero when \(c=\bar{z}_C\). Thus the fitted center is the cluster mean. Given those fixed centers, assigning a new observation to the closest center selects the cluster that would add the least squared distance to its assigned center. This is a coherent assignment rule; it does not claim that the assignment is certainly correct in any underlying population.
Worked Example: Use a Cluster Indicator in a Validation Prediction
A hypothetical training fold has four applications with scores \(x=0,2,8,10\) and binary outcomes \(y=1,1,5,5\), where \(y\) is a numeric review-cost index. Use the two clusters and training-only standardization from the first example. Fit a model that predicts the mean training outcome within each cluster, then predict the outcome for a held-out application with \(x=4\) and observed \(y=5\). Calculate its squared prediction error.
Method and fitted cluster outcomes. The low-score cluster contains the two training outcomes 1 and 1, so its fitted mean is \((1+1)/2=1\). The high-score cluster contains outcomes 5 and 5, so its fitted mean is \((5+5)/2=5\). This is a cluster-indicator regression with a separate fitted mean for each cluster.
Held-out feature construction. From the earlier calculation, the training mean is 5, the training standard deviation is \(\sqrt{17}\), and \(x=4\) standardizes to \(-1/\sqrt{17}\). Its distance to the low center \(-4/\sqrt{17}\) is \(3/\sqrt{17}\), and its distance to the high center \(4/\sqrt{17}\) is \(5/\sqrt{17}\). It is assigned to the low cluster, so its feature is \((1,0)\).
Prediction and score. The fitted prediction is the low-cluster mean, \(\hat{y}=1\). After the prediction is made, compare it with the held-out outcome \(y=5\). The residual is \(5-1=4\), and the squared prediction error is \(4^2=16\).
Check and common mistake. The held-out outcome was used only to calculate the assessment error, not to assign the cluster or fit its mean. Assigning this observation to a cluster based on its outcome would directly leak the answer into the predictor.
A Fold-by-Fold Workflow
In cross-validation, each held-out fold needs its own fitted preprocessing and cluster model. The same procedure is repeated for each split, with the model fit on that split's training portion and evaluated on its held-out portion.
Set aside the fold to be assessed. Do not use its outcomes, predictors, or summaries to fit the fold's preprocessing or clusters.
Estimate scaling values and the clustering model using only the training portion. Any choice among candidate cluster constructions must also be made without using the held-out assessment results.
Derive cluster labels, indicators, or distances for training observations, then fit the supervised model using their outcomes.
Apply the stored scaler and assignment rule, construct the same features, predict, and calculate the chosen validation metric only after predictions are fixed.
Cluster labels such as “cluster 1” and “cluster 2” are arbitrary identifiers. Their numbers can swap when a model is refitted in another fold, even when the underlying segmentation is similar. Within a fold, however, the validation assignment must use the label convention of that fold's fitted training model. Do not assume that label 1 has the same meaning across independently fitted folds, or pool cluster-specific coefficients as though the labels were automatically aligned.
When Using All Predictors Can Be Appropriate
There is an important distinction between inductive prediction and a prespecified transductive task. In an inductive task, the model is built from a training sample and must predict future observations whose predictors were not available during fitting. Training-only scaling and clustering are required for a faithful evaluation of that task.
In a transductive setting, the entire batch of predictor values to be segmented may genuinely be available before outcomes are predicted, and the intended procedure may explicitly cluster that batch together. Such a procedure can be evaluated, but the assessment design must reproduce that exact information availability. It should not be presented as evidence for an inductive model that will assign one new applicant at a time. The evaluation procedure must match how the model will actually be used.
Check Your Understanding
For each problem, decide what may be fitted from training data and how held-out observations receive cluster-derived features.
- A training fold has a predictor mean of 12 and standard deviation of 4. A held-out value is 20. What standardized value should be used if those training-fold quantities are fixed? (Answer: 2.)
- In a two-dimensional \(k\)-means model, a held-out point has distances 3.2 and 2.7 to the two training-fitted centers. Which cluster should it receive, and what is its two-indicator feature if the centers are ordered first, second? (Answer: The second cluster; \((0,1)\).)
- A training-fold scaler uses values \(2,4,6\), and a validation value is 12. Using the population-form standard deviation, find the training mean, standard deviation, and standardized validation value. (Answer: Mean \(4\); standard deviation \(\sqrt{8/3}\approx1.6330\); standardized value \(8/\sqrt{8/3}\approx4.8990\).)
- A team fits scaling and clusters once to all observations, then performs cross-validation on the resulting cluster indicators. What is the central evaluation flaw? (Answer: Each held-out fold's predictors influenced the scaler and cluster representation, so the score does not assess a procedure fitted without those observations.)
- A hierarchical clustering algorithm has been fitted to a training fold, but no rule for assigning new observations has been specified. May the analyst recluster training and validation observations together to obtain validation labels for an inductive evaluation? (Answer: No; specify an out-of-sample assignment rule using the training-fitted representation, or use a method that provides one.)