Why the Split Must Match Feature Availability
Cluster memberships and PCA scores are created from predictor data, not from the response. That does not make their construction automatically safe: the records used to estimate scaling constants, cluster centers, or component loadings must have been available when the prediction would have been made. The same issue arises when records repeat for a person, policy, clinic, or other group. If the intended use is to predict for a new group, that group must not help construct the features used to evaluate its predictions.
The design question is therefore more specific than “Which records are in training?” Ask both which entities and dates are represented and which records could have informed the feature mapping. This tutorial develops those checks for grouped and time-ordered validation. The principles are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Make the Validation Target Explicit
A “group” is an entity that contributes one or more rows, such as a customer with multiple policies or a clinic with repeated observations. Whether a group may appear on both sides of a split depends on the intended prediction task. If the task is to predict a later record for a group already observed, allowing earlier records from that group into training can be appropriate. If the task is to predict for previously unseen groups, all records from each validation group must be excluded from training, including when fitting unsupervised features.
Time creates a related constraint. For a prediction made at a cutoff date, future records cannot be used to estimate the means, scales, cluster centers, or PCA directions that produce its features. A time split should therefore respect the order of information availability, not just the order of row numbers. A record dated earlier may still contain information that would only be known later, so the analyst must define what “available” means in the application.
For a training set \(T\) and validation set \(V\), let \(\phi_T\) denote a feature mapping fitted on \(T\). The validation features are \(\phi_T(X_j)\) for \(j\in V\), not features from a mapping fitted on \(T\cup V\). When testing performance on new groups, also require that the group identifiers in \(T\) and \(V\) do not overlap. For future prediction, require the latest training information to precede the validation prediction times.
Here \(G(T)\) and \(G(V)\) are the sets of groups represented in the two partitions, and \(t_T,t_V\) denote their record or prediction times. These are design conditions, not universal rules for every application. In particular, the group-disjoint condition is inappropriate if the intended task is explicitly prediction for already-known groups.
Construct Each Split and Feature Map in Order
A reliable workflow separates partitioning from feature fitting. First choose the partition that reflects the prediction task. Then estimate every data-dependent transformation from the training partition alone. Finally, apply those fitted transformations to validation records. This includes steps that may seem routine, such as centering and scaling before PCA or selecting a cluster solution based on the distribution of predictor values.
Specify whether validation represents new entities, later observations for known entities, future dates, or a combination.
Keep groups intact when evaluating new groups. For future prediction, place records after the cutoff in validation and preserve the time order.
Estimate imputation values, scaling constants, cluster centers, PCA loadings, and any choices based on predictor distributions using only eligible training information.
Use the training means, scales, centers, and loadings. Do not recalculate them using validation records.
Interpret the result as evidence about the specified group and time setting, not about every possible deployment setting.
Worked Examples
Worked Example: Holding Out Entire Groups Before Clustering
A hypothetical service network wants to predict outcomes for clinics that have never contributed records to model development. Each clinic has multiple observations described by two numeric predictors, \(x_1\) and \(x_2\). The training clinics have these values: clinic A has \((0,0)\) and \((0,2)\); clinic B has \((8,0)\) and \((8,2)\). Clinic C is held out and has \((1,1)\) and \((9,1)\). For illustration, fit a two-cluster mapping whose training centers are the two cluster means. Find the centers and assign clinic C's records using squared Euclidean distance.
Method. Keep clinic C completely out of feature fitting because the intended task is prediction for a new clinic. Compute the training centers from A and B only.
The mean of clinic A's two training records is \(((0+0)/2,(0+2)/2)=(0,1)\). The mean of clinic B's records is \(((8+8)/2,(0+2)/2)=(8,1)\). Thus, the fitted centers are \(c_1=(0,1)\) and \(c_2=(8,1)\).
For validation record \((1,1)\), the squared distances are \((1-0)^2+(1-1)^2=1\) to \(c_1\), and \((1-8)^2+(1-1)^2=49\) to \(c_2\). It is assigned to cluster 1.
For validation record \((9,1)\), the squared distances are \((9-0)^2+(1-1)^2=81\) to \(c_1\), and \((9-8)^2+(1-1)^2=1\) to \(c_2\). It is assigned to cluster 2.
Check. Each record is assigned to its nearer training center: \(1<49\) and \(1<81\). Clinic C contributes neither to the centers nor to any training scaling rule. The common mistake is to include clinic C while fitting the clusters, which would evaluate a mapping partly shaped by the very group meant to represent unseen clinics.
Worked Example: Applying a Training PCA Mapping to a Later Record
A hypothetical monitoring model will be used to predict later observations. At the training cutoff, three records have predictor pairs \((0,2)\), \((2,4)\), and \((4,6)\). A later validation record is \((4,6)\). Standardize each predictor using its training mean and sample standard deviation, then project the validation record onto the first principal component. For this illustration, the fitted first-component loading vector is \((1/\sqrt{2},1/\sqrt{2})\).
Method. Estimate standardization constants from the three earlier records, then apply them and the supplied fitted loading vector to the later record. The response values are not needed to construct this unsupervised mapping.
For \(x_1\), the training mean is \((0+2+4)/3=2\). The sample variance is \(((0-2)^2+(2-2)^2+(4-2)^2)/(3-1)=(4+0+4)/2=4\), so the sample standard deviation is \(2\). For \(x_2\), the training mean is \((2+4+6)/3=4\), and the sample variance is \(((2-4)^2+(4-4)^2+(6-4)^2)/2=4\), also giving standard deviation \(2\).
The validation record \((4,6)\) therefore standardizes to \(z_1=(4-2)/2=1\) and \(z_2=(6-4)/2=1\). Its first-component score is
The score is approximately \(1.4142\), rounded to four decimal places. As a leakage check, suppose someone instead uses the validation record when estimating the means, while temporarily keeping the training scales and loading fixed. The resulting means would be \(2.5\) and \(4.5\), making the validation standardized values \(0.75\) and \(0.75\), and the score \(1.5/\sqrt{2}\approx1.0607\). That different score shows that even a target-free validation record can change the mapping. The proper score uses only the training estimates.
Worked Example: Combining a Time Cutoff With New-Group Validation
A hypothetical risk team will score new partner organizations in April using a cluster feature learned from information available by March 31. The eligible historical records are from organization P: \((0,0)\), \((0,2)\), and organization Q: \((10,0)\), \((10,2)\). Organization R is new and has two April records, \((2,1)\) and \((9,1)\). Fit the two-center mapping from eligible history and assign the April records.
Method. Enforce both constraints: use no April records when fitting the mapping, and exclude the new organization R from training. Compute the two historical centers and then transform R's records with those centers.
Organization P's center is \(((0+0+0)/3,(0+2+0)/3)\) only if its third value were present; here it has exactly two records, so its center is \(((0+0)/2,(0+2)/2)=(0,1)\). Organization Q's center is \(((10+10)/2,(0+2)/2)=(10,1)\). Use \(c_1=(0,1)\) and \(c_2=(10,1)\) as the fitted cluster centers.
For R's first record \((2,1)\), squared distances are \((2-0)^2+(1-1)^2=4\) and \((2-10)^2+(1-1)^2=64\), so it maps to cluster 1. For R's second record \((9,1)\), the squared distances are \(81\) and \(1\), respectively, so it maps to cluster 2.
Check. The assignments follow because \(4<64\) and \(1<81\). All center-estimation records are historical and from organizations other than R. The common mistake is to fit the centers using all available rows in a retrospective table, including April or R's records, even though neither would be available for the stated new-organization prediction.
Why the Rule Holds
Validation is meant to approximate the procedure that will be used when a prediction is made. Let \(D_T\) be the eligible training predictor records, and let the fitting operation produce a mapping \(\phi_T=\operatorname{fit}(D_T)\). A future or held-out record \(x\) receives the feature \(\phi_T(x)\). This is precisely the sequence available in deployment: first learn a mapping from permitted history, then transform the new record.
If validation predictors are included in the fitting step, the evaluated procedure becomes \(\operatorname{fit}(D_T\cup D_V)\), where \(D_V\) contains records that are supposed to be unavailable at prediction time. The resulting validation features are no longer generated by the deployment procedure. This conclusion does not depend on whether the response was used: a changed mean, center, or loading changes the features themselves. That is why target-free feature construction can still leak information.
For group validation, the same reasoning applies to the group boundary. When the question concerns a new entity, no record from that entity should affect \(\phi_T\); otherwise the mapping has access to characteristics of the supposedly unseen entity. For a known entity's later observation, earlier records for that entity may be eligible, but later records remain unavailable. The right boundary follows the prediction question, not a blanket rule that groups must always be separated.
Practical Checks and Common Traps
- Check the unit being predicted. “A new claim” and “a claim from a new policyholder” imply different validation designs if policyholders contribute repeated records.
- Check all fitted steps. Scaling, imputation, clustering, and PCA are all fitted transformations. A pipeline can leak through a preprocessing step even if the final model is trained only on the intended training rows.
- Check the time of availability. A date field does not guarantee that every predictor was known on that date. For example, a later-updated status can be unavailable at the nominal prediction cutoff.
- Do not update the mapping with validation observations. Use training means and scales, cluster centers, and PCA loadings when transforming validation records.
- Do not claim a broader result than the split supports. A group-disjoint split assesses new-group performance; a time split assesses later-period performance under its chosen cutoff. Neither alone establishes performance in every deployment setting.
Check Your Understanding
For each problem, identify the valid feature-fitting data or compute the requested derived feature. Treat any stated validation observations as unavailable when fitting the mapping.
- A model is intended for new laboratories. Laboratories A and B are training groups, and C is the validation group. A and C both have records in the same month. Which laboratory records may be used to fit scaling constants and cluster centers? (Answer: Use records from A and B only; exclude every record from C.)
- Training values for a predictor are \(2,4,6\), and a later validation value is \(8\). Using the training mean and sample standard deviation, calculate its standardized value. (Answer: Training mean \(=4\), sample standard deviation \(=2\), standardized value \(=(8-4)/2=2\).)
- A new group has predictor record \((3,1)\). Training cluster centers are \((1,1)\) and \((7,1)\). Using squared Euclidean distance, which cluster is assigned? (Answer: Distances are \(4\) and \(16\), so assign cluster 1.)
- A team predicts future records for groups already seen. Its training records end on June 30 and validation records begin July 1. May the feature mapping use earlier records from a validation group? May it use that group's July records? (Answer: Earlier records may be used if they would be available at prediction time; July validation records may not be used to fit the mapping.)
- Three training records standardize to \((-1,-1)\), \((0,0)\), and \((1,1)\). A fitted first-component loading is \((1/\sqrt{2},1/\sqrt{2})\). A later record standardizes to \((1,1)\) using training constants. What is its score, and should validation data be used to recompute the loading? (Answer: The score is \((1+1)/\sqrt{2}=\sqrt{2}\approx1.4142\); do not recompute the loading using validation data.)