Why Partition Boundaries Matter
A holdout set is useful only if it represents observations that did not influence the model-development decisions being evaluated. The key issue is not merely whether a row was placed in the training partition or the holdout partition. It is whether information from the holdout could flow back into fitting, transformation, feature selection, or model choice. If it does, the reported performance may no longer estimate how the chosen procedure will perform on genuinely new observations.
This matters in actuarial prediction because a seemingly small workflow shortcut can change the question an evaluation answers. A test score is intended to describe performance after development is complete. If the test set has already influenced development, that score instead describes a process that had access to information it would not have for future predictions.
A partition is therefore a boundary on information use, not just a label attached to rows. Training observations may be used to estimate the model. A validation set may be used for specified development decisions, such as comparing candidate procedures, but then it is no longer an untouched final evaluation set. A final test set is most informative when it remains unused until the modeling procedure and its decisions are fixed.
Leakage does not mean that training and evaluation observations must have different kinds of predictors. They should usually represent the same intended prediction population. The concern is that evaluation observations, or information derived from them, have affected the procedure being assessed. A model can use only predictor values at prediction time and still be evaluated improperly if those predictor values from the holdout influenced development.
Where Information Can Cross the Boundary
Information can cross a partition boundary at several stages. For example, a data-driven transformation may be estimated using the combined data, or a feature may be retained because it appears useful after examining holdout results. In both cases, the holdout has helped determine the procedure whose performance is then reported on that same holdout.
A useful diagnostic question is: Would this decision have been made in the same way if the evaluation observations had not been available? If the answer is no, the evaluation may have influenced development. The next tutorial addresses fitting preprocessing using training data; here, the focus is recognizing why that boundary matters.
| Stage | Potential boundary problem | Why it matters |
|---|---|---|
| Transforming predictors | Estimate a transformation using training and holdout rows together | The transformed training data can depend on the holdout distribution |
| Choosing features | Use holdout performance or associations to decide which predictors to retain | The holdout has influenced the model being evaluated |
| Selecting a model | Compare candidates on a holdout, choose the strongest, then report that same score as an independent estimate | The selected score benefits from the selection process |
| Fitting a model | Allow evaluation observations to contribute to model fitting | The evaluation is no longer based on observations kept separate from fitting |
Worked Examples of Boundary Leakage
Worked Example: A Pooled Transformation
A hypothetical claim-severity model uses a predictor \(x\), transformed by min-max scaling. The training values are 10 and 20, and the holdout value is 100. The analyst scales all three values together and then reports holdout performance. Identify the boundary issue and calculate the transformed values under both approaches.
Method. For a set with minimum \(a\) and maximum \(b\), min-max scaling is \(x^*=(x-a)/(b-a)\), provided \(b>a\). For a holdout evaluation intended to simulate future prediction, the transformation should be based on the training data, then applied to the holdout.
Training-based calculation. The training minimum is 10 and maximum is 20, so \(b-a=10\). The scaled training values are \((10-10)/10=0\) and \((20-10)/10=1\). Applying those same training values to the holdout gives \((100-10)/10=9\).
Pooled calculation. If all three observations determine the scaling range, the minimum is still 10 but the maximum is 100, so \(b-a=90\). The pooled transformed values are \(0\), \((20-10)/90=1/9\), and \(1\), respectively.
The holdout value changes the transformation applied to the training value 20: its scaled value is 1 using training data alone, but \(1/9\) using the pooled range. The training transformation has therefore depended on an evaluation observation. Common mistake: treating a transformation as harmless because it does not use holdout labels. The correction is to ask whether the transformation uses any holdout information that would be unavailable in the intended prediction process. Here, the holdout predictor value changes the training representation, so the evaluation procedure has crossed the boundary.
Check. The training-based range is \(20-10=10\), and the pooled range is \(100-10=90\). The two calculations agree with the original values and show exactly where the holdout enters.
Worked Example: Selecting a Candidate on the Holdout
A hypothetical insurer compares two binary classifiers on the same eight holdout policies. The observed outcomes are \(1,1,0,1,0,1,0,0\). Candidate A predicts \(1,1,0,0,0,1,1,0\), while candidate B predicts \(1,0,0,1,1,1,0,1\). The analyst chooses the candidate with higher holdout accuracy and reports its accuracy as the final performance estimate. Calculate the scores and diagnose the issue.
Method. Accuracy is the number of correct classifications divided by the number of observations. Compare each prediction with the corresponding outcome.
For candidate A, the predictions are correct on policies 1, 2, 3, 5, 6, and 8. Thus, it has 6 correct classifications out of 8, giving \(6/8=0.7500\).
For candidate B, the predictions are correct on policies 1, 3, 4, 6, and 7. Thus, it has 5 correct classifications out of 8, giving \(5/8=0.6250\).
The analyst selects A because \(0.7500>0.6250\). The calculation of A's observed accuracy is correct, but reporting it as an independent estimate after choosing A on these same outcomes is not. The outcomes helped determine which candidate would be reported. Common mistake: assuming that a holdout remains an untouched final test set even after its results determine which model to keep. The correction is to describe this set as part of model selection, or to evaluate the fixed choice on a separate, previously unused test set.
Check. Candidate A has 6 correct predictions and B has 5; the respective accuracies are \(0.7500\) and \(0.6250\). The selected value is therefore \(0.7500\), but its use as an independent final estimate is compromised by the selection.
Worked Example: Holdout Values Affecting an Imputation
A hypothetical pricing model has training predictor values 2, 6, and one missing value. The holdout predictor value is 100. An analyst replaces missing values by the mean of observed values, but calculates that mean using both partitions. Compare the replacement value with the value that would be obtained using training data only.
Method. The mean of observed values is their sum divided by their count. For this example, the imputation value is estimated from the observed values available in the specified set.
Using only training observations, the observed sum is \(2+6=8\), and there are 2 observed values. The training mean is \(8/2=4\), so the missing training value would be replaced by 4.
Using training and holdout observations together, the sum is \(2+6+100=108\), and there are 3 observed values. The pooled mean is \(108/3=36\), so the missing training value would instead be replaced by 36.
The holdout value has changed a value inserted into the training data from 4 to 36. Any model fitted after that replacement is based partly on the holdout predictor distribution. Common mistake: saying that this cannot be leakage because the missing value itself belongs to training. The correction is to trace the source of the replacement value: it was estimated using a holdout observation. The arithmetic checks directly: \(2+6=8\), \(8/2=4\); and \(2+6+100=108\), \(108/3=36\).
Why Reusing Evaluation Results Can Be Optimistic
The model-selection example illustrates a general reason that an evaluation score can become optimistic. Assume there are \(J\) fixed candidate models, each has a true performance \(M_j\) on the target population, and its holdout estimate is \(\widehat{M}_j=M_j+\varepsilon_j\). Assume each estimate is unbiased, so \(E[\varepsilon_j]=0\), and larger performance values are better. If the analyst chooses the candidate with the largest holdout estimate, the reported score is \(\max_j\widehat{M}_j\).
Let \(j^*\) be a candidate with the largest true performance. For every set of observed estimates, the maximum estimated performance is at least the estimate for \(j^*\). Therefore:
Taking expectations preserves the inequality. Since \(\widehat{M}_{j^*}\) is unbiased, this gives:
This is a short justification for why selecting the largest noisy holdout score tends to overstate the performance of the best fixed candidate. It does not say that every selected score must exceed the selected model's true performance in every sample. It says that, under the stated assumptions, the expected maximum estimated score is at least the best candidate's true performance. The more candidates or development decisions are made using the same evaluation results, the more chances there are to select a score that is high partly because of sampling variation.
A Practical Boundary Audit
Before interpreting an evaluation, trace the information path from each partition into the final result. A clear audit distinguishes model fitting, permitted development decisions, and final assessment. It also states what information would be available when the model is actually used.
Specify what is predicted, when predictions are made, and which information would be available at that point.
Identify observations used to fit, observations used for development choices, and any observations reserved for final evaluation.
Check whether a transformation, feature decision, model setting, or other choice depends on evaluation observations or their results.
If evaluation information influenced development, do not describe the resulting score as an untouched estimate of future performance.
A partition can have a legitimate role without being a final test set. For instance, using a validation set to compare candidates may be appropriate when the design calls for it. The important point is to be accurate about what the resulting score measures. Once results from that set have informed development, a separate evaluation is needed for an assessment independent of those choices.
Common Mistakes and Their Corrections
- “No outcome was used, so there is no leakage.” Predictor values can affect transformations or representations. Check all information flows, not only outcome values.
- “The holdout was never used to fit the model.” It may still have influenced feature selection, model choice, or other development decisions. Model fitting is only one possible pathway.
- “The best holdout score is the final test result.” If that score determined which candidate was chosen, the same observations have served both selection and evaluation roles.
- “A correct calculation guarantees a valid estimate.” A score can be arithmetically correct but answer a different question because of the workflow that produced it.
Check Your Understanding
For each case, identify the boundary issue and determine what conclusion the reported score can support.
- A training partition has predictor values 4 and 12; a holdout value is 20. A model uses min-max scaling estimated on all three observations. Calculate the training-based scaled value for 12 and the pooled scaled value for 12. (Answer: Training-based \(1\); pooled \(8/16=0.5\); the holdout value influenced the transformation.)
- Three fixed candidates have holdout accuracies \(0.70\), \(0.75\), and \(0.65\). The analyst reports \(0.75\) after choosing the second candidate using these results. What is the boundary concern? (Answer: The holdout selected the candidate, so \(0.75\) is not an independent final estimate.)
- Training values for a predictor are 3 and 9, with one missing training value; the holdout value is 30. Find the training-only mean imputation and the pooled mean imputation. (Answer: Training-only mean \(=(3+9)/2=6\); pooled mean \(=(3+9+30)/3=14\); using 14 lets the holdout affect training preprocessing.)
- An analyst compares 12 model variants on one validation set, selects the highest-scoring variant, and reports that score as performance on future cases. Name the problem and state a sound next step. (Answer: Reusing the evaluation results for selection can make the reported score optimistic; evaluate the fixed procedure on a separate, unused test set.)
- A transformation is fitted using training predictors only, then applied without refitting to holdout predictors. The holdout score is inspected once after all model choices are fixed. Does this alone indicate boundary leakage? (Answer: No; this describes a separated transformation and a final evaluation, assuming no other holdout information influenced development.)