Why an Impressive Score Can Be Misleading
A model contest can produce a clear numerical winner and still answer the wrong question. The apparent advantage may come from information that would not exist when predictions are made, preprocessing that used evaluation data, or candidate models scored on different groups of cases. These are not minor reporting details: they change what the comparison estimates.
The objective is to estimate how well each complete modeling procedure will predict outcomes for the intended future population, using only information available at the prediction time. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
A random train/test split does not, by itself, prevent these problems. Leakage can cross the split through a feature constructed after the outcome, a transformation fitted before splitting, or repeated records from the same underlying unit. The key audit question is not simply whether the rows were separated. It is whether the whole prediction procedure respects the information, timing, and population of the intended use.
Start With the Prediction-Time Information Set
Before evaluating a model, specify the prediction target, the point at which the prediction will be made, and which fields are available at that point. A field can be strongly associated with the outcome and still be invalid if it is recorded only afterward. Conversely, a field is not leakage merely because it is predictive: it is appropriate when it is genuinely available at the intended prediction time and was not constructed using future outcome information.
A useful justification follows directly from the quantity a test score is meant to estimate. Let \(Y\) be the future target, let \(X\) denote information available at prediction time, and let \(f(X)\) be the prediction rule. For squared-error loss, the target quantity is the expected loss under future use, \(E[(Y-f(X))^2]\). If the fitted rule instead uses an additional field \(Z\) that is only observed after \(Y\), then its score estimates the performance of a different rule, \(f(X,Z)\), under an information set unavailable in deployment. A test set can be completely separate from training and still give an invalid estimate if it contains \(Z\).
This same logic explains why learned preprocessing belongs inside the training process. A transformation estimated using all rows depends on the evaluation data, so the fitted procedure is no longer independent of the cases used to assess it. For cross-validation, each fold must therefore have its own preprocessing fit using only that fold's training portion.
Worked Example: A Post-Outcome Field Creates a False Winner
A hypothetical insurer wants to predict each reported claim's ultimate payment, in thousands of dollars, using information available on the report date. Four held-out claims have ultimate payments \(Y=(10,20,30,40)\). A contest entry includes the ultimate payment amount as a predictor, while a second model uses report-date information and predicts \((12,18,35,35)\). Compare their test RMSEs and decide whether the first model is a valid winner.
Method. Compute RMSE as the square root of the mean squared prediction error, then audit whether the predictors are available at the prediction time.
For the first model, the leaked ultimate-payment field lets it reproduce every target exactly. Its errors are \(0,0,0,0\), so the squared errors sum to \(0\). Thus
For the report-date model, the errors \(Y-\hat{Y}\) are \(-2,2,-5,5\). The squared errors are \(4,4,25,25\), with total \(58\). Therefore
The result is in thousands of dollars, rounded to two decimal places. The zero-RMSE entry is not a valid winner: it uses the outcome itself, which is unavailable on the report date. The correct comparison excludes that field and evaluates only procedures based on information available then. The common mistake is to accept the lowest test score before checking feature timing; separation of test rows does not repair target leakage.
Preprocessing Must Be Learned Within the Split
Imputation, scaling, encoding decisions, and feature selection can all depend on data. If a procedure estimates a mean, median, scale, category set, or selected predictor set from the entire data set before splitting, evaluation cases have influenced the fitted procedure. The exact consequence depends on the transformation and model: global preprocessing does not necessarily make every score look better, but it does make the evaluation no longer a clean assessment of a training-only process.
For supervised feature selection, the risk is especially direct. If predictors are chosen by measuring their association with the target, that selection must use training outcomes only. Choosing features on the full data set allows test outcomes to influence which predictors the model uses, even if the final coefficients are estimated using training rows alone.
Worked Example: A Global Scaler Uses the Test Predictor
A candidate standardizes one numeric predictor using the mean and population standard deviation. The training values are \(0\) and \(2\); the test value is \(100\). Calculate the test value under (a) a scaler fitted on training data only and (b) a scaler incorrectly fitted on all three values. Explain which transformation is appropriate for this evaluation.
Method. For each stated fitting set, calculate its mean and population standard deviation, then apply \(z=(x-\bar{x})/\sigma\). The population variance divides the sum of squared deviations by the number of values.
(a) Training-only fit. The training mean is \((0+2)/2=1\). The population variance is \(((0-1)^2+(2-1)^2)/2=1\), so the standard deviation is \(1\). Applying this scaler to the test value gives
(b) Incorrect global fit. The mean of \(0,2,100\) is \(102/3=34\). The population variance is
The standard deviation is \(\sqrt{6536/3}\approx46.676\). The globally scaled test value is therefore \((100-34)/46.676\approx1.414\). The training-only transformation is appropriate because the test value must not contribute to estimating the scaler. The global result is not evidence that the score must be optimistic; it is evidence that the evaluation procedure has used information from the test predictor when fitting preprocessing. The mistake to avoid is calling a transformation “unsupervised” and therefore automatically safe to fit globally.
Make the Evaluation Cases Comparable
Candidate scores are comparable only when they refer to the same target definition, eligible population, time period, and evaluation cases. A model that cannot produce predictions for some cases may appear better if those difficult or incomplete cases disappear from its score. A model scored on a broad population cannot be directly ranked against one scored only on a selected subset.
When a candidate genuinely cannot score every case, define the comparison population explicitly. Ideally, use a consistent evaluation set on which each complete modeling procedure can make predictions. If the intended system must handle missing inputs, its missing-data handling is part of the procedure and should be evaluated too. Do not silently drop cases separately for each model and then compare the resulting metrics as if they were calculated on the same sample.
Worked Example: Different Test Samples Reverse the Ranking
Two models are evaluated for a hypothetical portfolio. Model A has predictions for 100 cases and reports RMSE \(4.00\), so its total squared error is \(100(4.00^2)=1{,}600\). Model B has predictions for only 80 cases and reports RMSE \(3.00\), so its total squared error is \(80(3.00^2)=720\). On the 80 cases scored by both models, A's squared errors sum to \(640\). Compare the models fairly on those common cases.
Method. Recalculate both RMSEs on the intersection of their evaluation cases. Model A's \(1{,}600\) total squared error includes the 80 common cases and 20 additional cases. Its error sum on the common cases is given as \(640\); this leaves \(1{,}600-640=960\) squared-error units on the 20 other cases. Model B's reported 720 is its error sum on the same 80 common cases.
On the common 80 cases, Model A's RMSE is
Model B's RMSE on those cases is
The scores are rounded to two decimal places. On the common cases, A has lower RMSE, although the original reported values made B appear better. The reversal is possible because A's extra 20 cases contributed substantial error, while B was not evaluated on them. The original comparison does not establish which model is better for the full 100-case population; it compares different samples. The common mistake is to compare the displayed RMSEs without checking their denominators and case membership.
Rebuild the Contest Before Naming a Winner
A defensible comparison begins again from the prediction task, not from the leaderboard. Specify the target and prediction time, establish a common evaluation population, split before fitting data-dependent steps, and then assess each complete candidate procedure on the same held-out cases. The following sequence makes the audit concrete.
Write down the target, prediction time, eligible population, and data fields available then. Flag fields recorded after the target or derived from it.
Set consistent eligibility rules and identify the exact training, validation, and test cases. Check that all candidates use the same target values and evaluation rows.
Estimate imputations, scales, encodings, and supervised feature selection using only the relevant training portion. Repeat this within each resampling split when resampling is used.
Apply each fitted pipeline to the same evaluation rows and calculate the chosen metric with the same target and units.
Discard contaminated scores, rebuild the comparison, and only then interpret any performance difference. A previous leaderboard is not evidence that its winner remains best.
For illustration, suppose a corrected held-out set has four outcomes \(10,20,30,40\), in thousands of dollars. After removing a post-outcome field and fitting preprocessing on training data only, a GLM predicts \(12,18,35,35\), while a boosted model predicts \(11,23,28,44\). Both predict all four same cases. The GLM squared errors sum to \(58\), as calculated earlier. The boosted model's errors are \(-1,-3,2,-4\), with squared errors \(1,9,4,16\), summing to \(30\). Thus its RMSE is \(\sqrt{30/4}=\sqrt{7.5}\approx2.74\), compared with the GLM's \(\sqrt{58/4}\approx3.81\), in thousands of dollars rounded to two decimals. This rebuilt comparison supports the boosted model on this illustrative test set; it does not validate the contaminated leaderboard or establish that the boosted model will win in every future population.
Common Audit Traps
- “The test set was separate, so there can be no leakage.” A field copied from the outcome or recorded after the prediction date can invalidate a test score even when every row was held out.
- “Scaling uses no target, so fitting it globally is harmless.” Global scaling still uses test predictors to fit the evaluated procedure. Fit it on training data; in cross-validation, refit it within every training fold.
- “The larger test set makes the score more representative.” A larger but different sample does not make a direct comparison fair. First make the cases and eligibility definitions consistent.
- “A lower score proves the model is better.” It proves only that the reported calculation is lower. Check timing, preprocessing, target definition, case membership, and units before interpreting the ranking.
Check Your Understanding
For each problem, identify the issue and calculate or state the corrected result requested.
- A model predicts two held-out values \(5\) and \(9\) as \(4\) and \(11\). Calculate its RMSE, rounding to four decimal places. (Answer: \(\sqrt{2.5}=1.5811\))
- A numeric predictor has training values \(2\) and \(6\), and a held-out value \(20\). Find the training-only mean and the mean obtained by incorrectly pooling all three values. (Answer: training-only mean \(4\); pooled mean \(28/3\approx9.3333\))
- On the same five evaluation cases, Model A has squared errors summing to \(50\) and Model B has squared errors summing to \(80\). Calculate both RMSEs and identify the lower one, rounding to four decimals. (Answer: A \(=\sqrt{10}=3.1623\); B \(=4.0000\); Model A is lower)
- A claim model is intended to predict the payment expected using information available at first notice. One input is the final settlement amount, recorded after the claim closes. Is this a valid predictor for the intended task, even if it improves held-out performance? (Answer: No; it is post-outcome target leakage and must be excluded from that prediction task.)
- In a contest, Candidate A is scored on 120 eligible cases and Candidate B on 95 cases because B's preprocessing drops rows with missing predictors. What must be done before their reported scores support a direct ranking? (Answer: Define one eligible evaluation cohort and score both complete procedures on the same cases, with appropriate missing-input handling if required.)