Why a Validation Plan Can Give the Wrong Answer
A validation score is useful only if the validation procedure represents the prediction task. Randomly allocating rows is not automatically appropriate, and a clean separation of row numbers does not guarantee separation of time periods, people, or information used during preprocessing. A carefully calculated score can therefore answer a different question from the one a business decision requires.
The central audit question is: what information would be available when the model is used, and what observations will it be asked to predict? A validation design should reproduce those conditions as closely as practical. For example, predicting next year's claims is different from predicting another claim record for a policyholder already represented in the training data.
This tutorial focuses on four recurring failures: using a random split for a future-prediction task, allowing related entities to appear on both sides of a split, fitting preprocessing before rather than within resampling, and repeatedly using test results to choose a model. The earlier material on time-series, grouped, and stratified folds gives specific resampling designs; the goal here is to recognize when a seemingly ordinary split answers the wrong question.
Trap 1: Random Splits for Future Prediction
A random split treats observations as exchangeable for allocation: each row may go to either the training or assessment set. That can be reasonable when deployment concerns new observations from the same stable population and observations are suitably independent. It is generally unsuitable when the goal is to predict later periods and the data contain time trends, changing processes, or information that would not yet have been available.
For a future-prediction task, training observations should precede assessment observations. A random split of historical records can put later observations into training and earlier observations into assessment. The model is then evaluated with information from the future relative to some assessment records, unlike the intended use. A forward-looking split instead fits on an earlier period and assesses on a later period.
Worked Example: Auditing a Random Split for Next-Year Prediction
A hypothetical insurer wants to predict claim severity for the next calendar year. A random split of past records produced four assessment residuals, defined as observed severity minus predicted severity, of \(1, 2, 1,\) and \(2\) thousand dollars. A forward holdout using the latest two years produced residuals of \(4\) and \(5\) thousand dollars. Calculate each RMSE and assess which design answers the stated question. Assume the reported residuals are the complete assessment results for each design.
Method. For \(n\) assessment observations, RMSE is the square root of the average squared residual:
For the random split, square each residual: \(1^2=1,\;2^2=4,\;1^2=1,\;2^2=4\). Their sum is \(1+4+1+4=10\), and the mean squared error is \(10/4=2.5\). Thus the random-split RMSE is \(\sqrt{2.5}=1.5811\) thousand dollars, or about \(\$1{,}581\) per claim.
For the forward holdout, the squared residuals are \(4^2=16\) and \(5^2=25\). Their sum is \(41\); the mean squared error is \(41/2=20.5\). The forward RMSE is \(\sqrt{20.5}=4.5277\) thousand dollars, or about \(\$4{,}528\) per claim. Both values are rounded to the nearest dollar only at the final step.
The forward holdout is the more relevant design because the intended use is prediction in a later year. The scores are based on different assessment records, so their difference alone does not prove that time caused worse performance. It does show that a low random-split score cannot substitute for an evaluation of future-period prediction. Common mistake avoided: choosing the random split just because it has more assessment records or a smaller RMSE.
A forward holdout also requires care about the intended horizon. A model retrained every month may need rolling-origin assessments at several dates, while a model used once for a fixed future period may be assessed on that period. The essential requirement is that each assessment prediction use only information available at its prediction date.
Trap 2: Separating Rows but Not Entities
Rows are not necessarily independent units. A data set may contain several records for each policyholder, patient, vehicle, property, or employer. A random row split can place one record for an entity in training and another record for the same entity in assessment. A model can then benefit from familiarity with that entity, and repeated or closely related records can make the apparent assessment sample larger than its effective independent information.
If the intended task is prediction for new entities, assign all records from an entity to the same partition. Grouped folds do this within cross-validation. If deployment instead predicts future records for known entities, a grouped split by entity may be too severe or may represent a different task; the split should reflect the actual prediction setting, often with time ordering as well.
Worked Example: Detecting Policyholder Overlap
A hypothetical insurer has multiple annual records per policyholder. A model evaluated on four randomly held-out rows has residuals of \(0, 0, 1,\) and \(1\) thousand dollars. However, every held-out policyholder also has another year in the training set. A separate grouped assessment that withholds complete policyholders has residuals of \(3, 4, 2,\) and \(3\) thousand dollars. Calculate the RMSEs and identify which result is relevant when pricing new policyholders.
For the row-random assessment, the squared residuals are \(0,0,1,1\). The sum is \(2\), and the mean squared error is \(2/4=0.5\). Therefore, RMSE is \(\sqrt{0.5}=0.7071\) thousand dollars, or about \(\$707\).
For the grouped assessment, the squared residuals are \(3^2=9,\;4^2=16,\;2^2=4,\;3^2=9\). The sum is \(9+16+4+9=38\); the mean squared error is \(38/4=9.5\). Therefore, RMSE is \(\sqrt{9.5}=3.0822\) thousand dollars, or about \(\$3{,}082\). Each final dollar amount is rounded to the nearest dollar.
For pricing new policyholders, the grouped result is the relevant estimate because no records for those policyholders should be available during training. The row-random result measures a task with policyholder overlap and can look better when the model benefits from information about familiar entities. Common mistake avoided: claiming that no row is duplicated means the assessment set is independent of the training set.
Before splitting, inspect how many records each entity contributes and decide what a prediction unit is. Keep related records together when the deployment target is a new group, and preserve time order too when the target is future performance. A split can need both restrictions.
Trap 3: Fitting Preprocessing Outside the Fold
Imputation, scaling, and other learned transformations are part of the modeling procedure. In cross-validation, fit each such transformation using only that fold's training portion, then apply the fitted transformation to its assessment portion. Repeat the fitting independently for each fold. Fitting preprocessing once on the full data allows assessment data to influence the analysis, even if the outcome column was not used.
The same rule applies to supervised preprocessing, such as selecting predictors using their relationship with the outcome: selection must be repeated using only the training portion of each fold. Otherwise, assessment outcomes have already helped determine the predictors being assessed. This is distinct from merely applying a previously specified, fixed transformation.
Worked Example: Finding an Imputation Leak Across a Fold
In one cross-validation fold, the training predictor values are \(2, 4,\) and missing. The assessment predictor value is \(100\). A proposed workflow computes a mean for imputation before creating folds. Show the correct training-fold imputation value, the value produced by the flawed workflow, and the correction.
Correct method. Compute the imputation mean from the observed training values only: \((2+4)/2=3\). Impute the training missing value with \(3\). If an assessment value were missing, apply this same training-derived value of \(3\) to it; do not recompute the mean in the assessment fold.
Flawed method. If the mean is computed before the split, the observed values include \(2,4,\) and the assessment value \(100\). The mean becomes \((2+4+100)/3=106/3=35.333\ldots\). The training missing value is then imputed using information from an assessment observation. The value \(100\) has influenced the fitted preprocessing rule.
The corrected procedure fits the imputer on the training fold and carries that fitted rule over to the assessment fold. For a final test set, the imputer is likewise fitted on development data only and then applied unchanged to the test set. Common mistake avoided: assuming that using predictor values but not assessment outcomes makes full-data preprocessing harmless. It still lets the assessment distribution influence the procedure and may not match deployment.
There is a narrow exception in the definition of the task: a genuinely transductive setting may make an entire future batch of predictor values available before predictions are produced. In that case, using batch-wide predictor information may be part of the intended procedure. It must be specified in advance and reproduced in validation; it is not a justification for silently using assessment data in an ordinary predictive workflow.
Trap 4: Selecting a Model on the Test Set
A final test set is reserved for evaluating a modeling procedure after choices have been made using training data and, as appropriate, validation or cross-validation. If a modeller tries many models, features, thresholds, or preprocessing choices and then reports the best test score, the test set has become a selection set. The reported result is no longer an untouched evaluation.
The mechanism can be shown directly. Suppose \(m\) candidate procedures have the same true error \(R\), and each observed test error is \(\hat{R}_j=R+\epsilon_j\), where each \(\epsilon_j\) has mean zero. Selecting the lowest observed error yields \(R+\min_j\epsilon_j\). Since the minimum is no larger than the average,
Taking expectations gives \(E[\min_j\epsilon_j]\leq (1/m)\sum_j E[\epsilon_j]=0\). Thus the selected minimum error is, on average, no greater than the common true error; it is strictly smaller when the errors vary and the minimum is not always equal to their average. This justification assumes equal true errors and mean-zero estimation errors. With unequal true errors, selection still favors candidates with fortunate assessment noise, although the amount of optimism depends on the setting.
Worked Example: Catching Test-Set Model Selection
A hypothetical actuary tries four candidate severity models and records test RMSEs of \(7.4,\;7.8,\;8.1,\) and \(9.0\) thousand dollars. The actuary reports \(7.4\) as the final test RMSE after choosing the first model because it scored best. Diagnose the problem and give a defensible evaluation plan.
The minimum is \(7.4\) thousand dollars, but four choices were compared using the same test set. That test set influenced which model was selected, so \(7.4\) is a selection result, not an independent final assessment of the selected procedure. It should not be described as an untouched estimate of future RMSE.
A defensible plan is to use training data and cross-validation to compare the four candidates, choose the model and all related settings from those results, then evaluate the chosen procedure once on a protected test set. If the reported values above were already obtained by repeatedly inspecting that protected set, reserve new untouched evaluation data if feasible; otherwise state the limitation rather than relabeling \(7.4\) as unbiased. Common mistake avoided: believing that a test set remains independent simply because it was not used in coefficient fitting.
This issue also arises when a modeller repeatedly checks a test score while changing the feature set, tuning parameters, or deciding which analysis to report. A test set is protected by its role in the workflow, not by its name in a file. The earlier material on protecting the final test set and on reuse of validation results explains how to allocate model development and final assessment.
A Practical Audit of the Split and Resampling Plan
An audit should trace both observations and learned information through the complete process. The following sequence helps distinguish a valid evaluation from one that only appears separated.
Specify whether predictions concern later dates, new entities, future records for known entities, or another defined population.
Confirm that the split reflects the prediction target. Preserve chronology for future prediction and keep related entities together when evaluating new-entity performance.
For each fold, fit imputation, scaling, feature selection, and other data-dependent steps on its training portion only; then apply the fitted steps to that fold's assessment portion.
Use validation results to make development choices. Do not return to the final test score to choose among those choices.
Report what population, time period, and prediction task the evaluation represents, and note limitations such as a small number of independent entities.
A common exam trap is to treat any one of these checks as sufficient. For example, grouped folds do not by themselves fix time leakage, and a time-ordered split does not prevent preprocessing from using the assessment period. Each source of information must be audited separately. A validation plan aligned with the SOA Exam PA (Predictive Analytics) syllabus links the design to the intended prediction task and explains what the resulting score can, and cannot, support.
Check Your Understanding
For each situation, identify the main validation concern and the appropriate correction.
- A hospital predicts next quarter's readmissions. Its random split puts records from the last quarter into training and earlier records into assessment. What is the main issue? (Answer: Future information can enter training; use a time-ordered assessment with training records preceding the assessment quarter.)
- A property model is intended for buildings not previously observed. Each building has monthly rows, and a random row split places months for the same building in both sets. What split change is needed? (Answer: Keep all rows for each building in one partition, using grouped validation for new-building performance.)
- In one fold, training values are \(5, 9,\) and missing; the assessment value is \(26\). What mean should impute the training missing value, and why? (Answer: \(7\), because \((5+9)/2=7\) using observed training values only; the assessment value must not enter the fitted imputer.)
- A modeller tries six feature sets and chooses the one with the lowest score on a held-out test set. Can that minimum be reported as an untouched final test score? (Answer: No; the test set was used for selection. Choose using training data or cross-validation, then evaluate once on a protected test set.)
- A model predicts future claims for existing policyholders, and the evaluation uses a random row split with each policyholder represented in both partitions. Is a grouped split automatically the correct replacement? (Answer: Not automatically; define the deployment task. For future records of known policyholders, preserve time order and ensure only information available at each prediction date is used.)