Why Leakage Matters
A validation result is useful only if it represents the prediction task the model will face in practice. For a ratemaking model, that means predictions must be based on information that would be available at the time the rate is selected or the policy is priced. If a model sees information created by the outcome, or validation information that would not be available in actual use, its measured performance can look much better than its real predictive performance.
This problem is called information leakage. It can arise through a predictor recorded after the prediction date, records that share the same loss experience across samples, duplicate representations of one risk, or preprocessing whose fitted values depend on the validation sample. A clean split by row is not enough if the rows themselves carry information across the split.
Start by stating the prediction unit, target, and prediction time. For example, the unit might be a policy term, the target might be its subsequent claim frequency, and the prediction time might be the start of the term. A field can be valid for one task and leakage for another: a claim status may be useful when estimating an already-reported claim’s ultimate cost, but it is not available for pricing a new policy before any claim occurs.
Post-Outcome Variables and the Information Cutoff
A variable is not safe merely because it appears in the data extract or has a date that looks close to the policy period. Ask when the value was created, what events were used to calculate it, and whether those events occur before or after the prediction time. Common concerns include claim counts, paid amounts, case reserves, closure status, and summaries calculated using the full experience period being predicted.
The same check applies to target construction. If the target is losses during a future policy term, a predictor that includes any of those losses is circular. Even a field with a legitimate name, such as “prior claims,” can leak if its calculation accidentally includes claims from the target term. Preserve a clear information cutoff and compute predictors from data available before it.
Worked Example: A Claim Count Recorded After Binding
A hypothetical insurer wants to predict claim counts for four one-year policy terms at the start of each term. Two records are assigned to training and two to validation. The target counts are 0 and 1 in training, and 0 and 1 in validation. A field called “claims recorded for term” equals the target count on every record because it is populated after the term ends. Compare a toy rule that predicts from this field with a training-mean benchmark.
Assumptions. These are deterministic illustrative records, with no interest rate, mortality assumption, or probability distribution. The prediction time is the start of each term; the claim count for that term is not yet known.
Step 1: Check availability. The “claims recorded for term” field is created after the insured period and is exactly the target. It cannot be used as a predictor at the start of the term.
Step 2: Calculate the leaked rule’s apparent validation error. The toy rule predicts the target from the field, so the two validation predictions are 0 and 1. The absolute errors are \(|0-0|=0\) and \(|1-1|=0\); the mean absolute error is \((0+0)/2=0\).
Step 3: Calculate a valid benchmark. The training targets are 0 and 1, so their mean is \((0+1)/2=0.5\). Predicting 0.5 for each validation record gives absolute errors \(|0-0.5|=0.5\) and \(|1-0.5|=0.5\), for mean absolute error \((0.5+0.5)/2=0.5\).
The zero error is not evidence of predictive skill: the rule was given the answer after it occurred. The correction is to exclude the post-term field and build predictors only from information available at the start of the term. The common mistake is to treat a field as eligible because it is present in the modeling table, rather than checking its availability at prediction time.
Overlapping Policy Periods and Shared Experience
Rows can be different records but still share target information. Suppose one record covers January through December and another covers July through the following June for the same risk. If a claim in November is included in both target calculations, putting one row in training and the other in validation allows the same event to influence both sides. A random row-level split cannot make this overlap disappear.
The appropriate response depends on the intended use. If the goal is to assess predictions for genuinely new, non-overlapping experience, define the target windows so the same exposure and loss cannot contribute to both samples. If multiple records represent one risk or policy over time, assign records with shared experience consistently, rather than treating them as independent rows. Keep the prediction target and exposure definition aligned with that assignment.
Worked Example: A Claim Appearing in Two Policy Windows
A hypothetical fleet vehicle has two monthly-record aggregates: one covers January through December 2024 and is assigned to training; the other covers July 2024 through June 2025 and is assigned to validation. Each target is total claim payments for its 12-month window. A single $1,200 claim occurs in November 2024 and is included in both totals. Exposure is measured in months divided by 12.
Assumptions. The record values and claim are deterministic. No interest rate, mortality assumption, or probability distribution is used. The shared six calendar months have equal exposure weight, and the November claim is counted in both target totals as stated.
Step 1: Find shared exposure. The common period is July through December, or 6 months. Its exposure is \(6/12=0.5\) policy-years. Each full record has \(12/12=1\) policy-year.
Step 2: Find shared target information. The $1,200 November claim is present in both records. Therefore the training target and validation target are not based on wholly separate loss experience, despite being different rows.
Step 3: Correct the design. Do not use these two overlapping aggregates as independent training and validation observations. For a clean assessment of later experience, define non-overlapping target windows or place all records sharing this risk and event on the same side of the split. Then verify that no claim transaction or exposure month contributes to both samples.
The common mistake is to check only whether policy-period rows have different identifiers. The dates and the underlying loss transactions determine whether the outcome information is actually separate.
Duplicate Risks and Group Assignment
Duplicate or linked records create a related problem. A risk may appear under multiple policy identifiers because of a data conversion, endorsement, rewritten policy, or duplicate extract. If one representation is in training and another is in validation, the model may effectively encounter the validation risk during fitting. This is especially serious when identifiers, detailed characteristics, or a repeated claim make the records easy to match.
Before splitting, identify the unit whose future performance matters and create a stable grouping key for records that share that unit or experience. Check duplicates using the available risk identifiers and relevant period information. Correct true data duplication where appropriate; otherwise, ensure linked records remain together. A useful rule is that no information from the same underlying risk or loss event should cross the split unless that information would legitimately be available for the intended prediction.
Worked Example: One Vehicle Under Two Policy Identifiers
A hypothetical data extract contains two records for the same vehicle and coverage period. An administrative conversion created policy IDs P17 and Q42, but both records refer to vehicle key V8 and the same $900 claim. P17 is assigned to training and Q42 to validation. A fitted lookup rule predicts $900 for any validation vehicle key seen in training.
Assumptions. The two rows are confirmed duplicate representations of one vehicle-period and one claim, not separate insured exposures. The amounts are deterministic; there is no interest rate, mortality assumption, or probability distribution.
Step 1: Compare the records. The policy IDs differ, but vehicle key V8, the coverage period, and the $900 claim are the same. Thus the records represent one underlying risk outcome, not two independent observations.
Step 2: Show the apparent result. Because V8 appears in training with a $900 target, the lookup rule predicts $900 for validation Q42. Its absolute error is \(|900-900|=0\).
Step 3: Prevent the leakage. Use the underlying vehicle-period as the grouping key, and assign both records to the same sample. If they are exact duplicate rows, remove the redundant copy under a documented data rule. Re-evaluate only after the same outcome cannot be memorized across the split.
The common mistake is to assume distinct policy numbers imply distinct risks. Identifiers are administrative fields; grouping must reflect the actual prediction unit and shared experience.
Preprocessing Must Be Learned From Training Data
Leakage can occur before a GLM is fitted. Imputation values, scaling constants, category groupings, variable-selection decisions, and other data-dependent transformations are part of the modeling process. If they are estimated using training and validation records together, validation information has already influenced the model input, even if validation targets were not used.
Fit each learned transformation on training data, save the resulting parameters, and apply those same parameters to validation data. For example, estimate a training median and use it to fill missing values in both samples; do not calculate a new median from validation records. A pre-specified business rule that does not depend on the sample, such as a fixed unit conversion, does not need to be estimated from training data. If a transformation uses target values, it requires particular care: validation targets must not contribute to its fitted values.
Worked Example: Median Imputation Using the Wrong Sample
A hypothetical rating variable has training values 10 and 20. Validation contains one missing value and one observed value of 100. Compare filling the missing validation value using a median calculated from training alone with a median calculated after pooling all observed values.
Assumptions. Values are deterministic and no probability distribution is assumed. There is no interest rate or mortality assumption. The missing-value rule is median imputation, and the validation observations are not available when fitting the training pipeline.
Step 1: Fit on training. The training median is the average of 10 and 20, so it is \((10+20)/2=15\). The proper imputed validation value is 15.
Step 2: Calculate the pooled median. Pooling observed values gives 10, 20, and 100. In sorted order the middle value is 20, so pooled imputation would fill the validation missing value with 20.
Step 3: Identify the leakage and correction. The value 100 is a validation observation. Using the pooled median lets the validation distribution alter the transformation. Use 15, the training median, for the validation missing value; retain 100 as observed. The common mistake is to assume leakage requires validation targets: here the validation predictors alone have changed a fitted preprocessing parameter.
A Practical Leakage Review
A disciplined review is easier when it is part of sample preparation rather than a response to unexpectedly strong validation results. Document the prediction time, inspect field creation dates and calculation logic, and trace each target back to its source events. Then check overlap and grouping at the underlying risk and exposure level, not only at the row level.
State the prediction unit, target period, and exact time at which the rate or prediction is intended to be available.
For each predictor, ask when it is measured and whether it contains any part of the target outcome or later experience.
Look for overlapping periods, duplicate risks, repeated transactions, and identifiers that link records across samples.
Estimate imputations, scaling, encodings, and other learned transformations on training data, then apply them unchanged to validation.
Confirm that validation targets did not affect fitting, transformations, sample assignment, or choices made after reviewing validation performance.
The central question is not simply “Were the rows separated?” It is “Could any information unavailable in the intended prediction task have influenced the prediction or its evaluation?” That question connects timing, risk grouping, and preprocessing in one leakage check. Careful validation design is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each situation, identify the leakage concern and the appropriate correction.
- A model predicts next-term claim frequency at renewal. Its “current term claim count” field is finalized after the next term ends and equals the target count. (Answer: Post-outcome leakage; exclude the field and use predictors available at renewal.)
- A 12-month record and a second record for the same risk cover 8 common months. Exposure is measured as months divided by 12. How much exposure overlaps, and what should be checked before validation? (Answer: \(8/12=0.6667\) policy-years; check shared losses and keep overlapping experience from crossing the split.)
- Two records with different contract IDs represent the same building-period and the same $2,400 loss. One is in training and one in validation. What is the concern and correction? (Answer: Duplicate-risk leakage; group by the underlying building-period and assign together, or remove a confirmed duplicate.)
- Training values for a predictor are 4 and 12; a validation record is missing. What median should be used to impute it, and why? (Answer: 8; it is the training median \((4+12)/2\), and validation data must not fit the imputation.)
- A model uses an average claim severity by territory. The averages were calculated once using all records, then rows were split into training and validation. What must change if the average uses claim outcomes? (Answer: Recalculate territory averages using training outcomes only and apply the resulting values to validation without using validation outcomes.)