Why Leakage Is More Than a Split Problem
A model can perform well on a holdout set for the wrong reason. Perhaps one predictor was recorded only after the outcome occurred, or nearly identical records appeared on both sides of the split. In either case, the evaluation may reward access to information that would not be available in the intended use of the model.
Leakage is therefore defined relative to a prediction task: the prediction unit, prediction time, outcome period, and intended deployment population all matter. A record split that is suitable for predicting another observation from a known customer may be unsuitable for predicting outcomes for entirely new customers. A chronological split may prevent future information from travelling backward, but it does not by itself prevent the same claim from appearing in both partitions.
The first question is not simply, “Was this variable in the data?” It is: Would this value have been available, in this form, at the moment the prediction is meant to be made? For a real prediction, the recorded date and the date the information became knowable can differ. A claim might be assigned a loss date in April but not reported until May. A field may later be corrected, backdated, or populated from a final settlement record. A defensible feature set must use the information available at the prediction time, not merely information whose value refers to an earlier date.
Post-Outcome Predictors and Label Maturity
A post-outcome predictor is a variable whose value is determined by, or recorded after, the outcome being predicted. Examples include a final claim payment when predicting whether the claim will be settled, a cancellation reason when predicting future cancellation, or a case-closure code when predicting whether a case will close. Such variables can be strongly associated with the target precisely because the outcome has already happened.
A related timing issue is label maturity. If a target describes an event over a future window, every training example needs enough observation time to determine whether the event occurred during that window. A label that is still incomplete is not a reliable negative label. For example, “no lapse observed so far” is not the same as “no lapse during the full outcome window” when part of that window has not yet elapsed.
Worked Example: Checking Predictor Timing and a Three-Month Label
A hypothetical insurer plans to predict, at each monthly snapshot, whether a policy will lapse during the next three calendar months. A snapshot is dated June 1. The database is complete through August 31. Candidate predictors include the policy status recorded on May 31 and a closure code entered on August 10. Determine which predictor is eligible and whether the label for the June 1 snapshot is mature.
Method: establish the prediction time, outcome window, and information-availability date for each field. The prediction is made on June 1. The policy status recorded May 31 is available before that prediction, so it is eligible, provided it is not subsequently overwritten with a later value.
The closure code entered on August 10 was not available on June 1. It is therefore ineligible as a predictor for this prediction, even if the code describes a status related to the policy. Including it would give the model information from after prediction time.
The next three calendar months after the June 1 snapshot are July, August, and September. The full outcome window ends on September 30. Because the database is complete only through August 31, the label is not mature. The data do not yet establish whether a lapse occurred during September.
Check: The predictor decision uses what was available on June 1, while label maturity uses whether the full July-through-September outcome period has been observed. The common mistake is to treat June, July, and August as the next three months, or to call an unfinished September outcome a negative. Neither is correct. The full label can be assessed only after September 30, assuming the required September information is then complete.
A useful justification is to treat each record as a prediction made at a particular time. If a feature value is created after that time, the model is conditional on information that the real decision-maker did not possess then. The evaluation answers a different question—how well the model predicts with later information—not how well it would have predicted at the specified decision point. Similarly, an unfinished target window confuses “not yet observed” with “did not occur,” changing the meaning of the response variable.
Duplicates and Repeated Entities
Duplicates can create a different kind of leakage. If the same record, or a near-identical copy of it, occurs in both training and holdout data, a model may effectively encounter the holdout case while fitting. Exact duplicate rows can result from repeated file imports or joins that multiply records. Near-duplicates may represent the same claim or transaction with a slightly changed field.
Repeated entities are not automatically duplicates. Several rows for one policyholder may be legitimate monthly observations. Whether that repetition is a problem depends on the deployment target. If the goal is to predict for new policyholders, all observations from a policyholder should be kept together when partitioning. Otherwise, the model can learn customer-specific patterns from training rows and receive an easier test on other rows from that same person. If the goal is to predict future outcomes for existing policyholders, using earlier observations from those policyholders may match deployment—but the split must still respect time and predictor availability.
Worked Example: Preventing Exact Claims From Crossing Partitions
A hypothetical claims data set has eight rows. Rows A1 and A2 are exact copies of one claim, rows B1 and B2 are exact copies of another claim, and rows C, D, E, and F each represent a different claim. A random row split puts A1, B1, C, and D in training, and A2, B2, E, and F in holdout. Identify the problem and give a defensible repair.
Method: identify the independent prediction unit, audit duplicate keys, and assign whole units to partitions. Here the prediction unit is a claim, not a row. Claim A appears in training as A1 and in holdout as A2. Claim B also appears in both partitions. Thus two of the four holdout rows are exact copies of records used in training.
For an evaluation on unseen claims, first resolve exact duplicates. If A1/A2 and B1/B2 are accidental repeated imports, retain one copy of each claim rather than treating the copies as independent examples. If the rows reflect legitimate multiple records for one claim, retain the rows but assign all rows for each claim to the same partition.
One valid grouped assignment is to put claims A, B, C, and D in training and claims E and F in holdout. Before resolving duplicates, that assignment contains six training rows (two each for A and B, plus C and D) and two holdout rows (E and F). No claim crosses the boundary. After deduplicating A and B, it contains four distinct training claims and two distinct holdout claims.
Check: The repair prevents a claim from being represented on both sides. The common mistake is to regard eight rows as eight independent claims and preserve the row-level split. That leaves exact claim information in training while testing on its copy, so the holdout score does not measure performance on unseen claims.
Worked Example: Choosing Between Group and Time Partitions
A hypothetical renewal model has two monthly records for each of five customers, A through E. The intended deployment is to score customers who have never appeared in the model’s development data. A row-level random split places the first month for A, B, and C and the second month for D in training; it places the remaining six rows in holdout. Choose a partition that matches the deployment goal.
Method: partition by the unit expected to be new at deployment. Because the goal is to predict for new customers, the customer—not the monthly row—is the unit that must be held out. Under the stated row split, A, B, C, and D occur in training and holdout. The holdout therefore contains customers whose earlier or later records the model has already seen.
Assign all records for A, B, and C to training and all records for D and E to holdout. Training then contains \(3 \times 2 = 6\) rows from three customers. Holdout contains \(2 \times 2 = 4\) rows from two entirely different customers. No customer appears in both partitions, which matches the stated new-customer deployment goal.
If the intended task instead were to predict the next renewal for existing customers, a future-time holdout could appropriately contain later records for customers with earlier records in training. In that setting, the evaluation should reproduce the future prediction process: training records precede the test prediction times, features are available as of those times, and each test outcome is observed only after its outcome window matures.
Check: The group assignment is correct for new customers, but it is not a universal rule for every prediction task. The common mistake is to group-split automatically without asking what “new” means in deployment. The partition must represent the population and information flow the model will face.
Why Leakage Inflates Evaluation
A simple calculation explains why records that cross a partition boundary can produce an optimistic score. Suppose a fraction \(q\) of holdout cases are effectively familiar to the model because duplicates or highly identifying repeated records appeared in training. Suppose the model is correct on those familiar cases with probability 1 and has genuine accuracy \(p\) on the remaining, unfamiliar cases. By conditioning on whether a holdout case is familiar, the expected measured accuracy is
Since \(q \geq 0\) and \(p \leq 1\), the apparent uplift is \(q(1-p) \geq 0\). For example, if \(q=0.25\) and genuine accuracy is \(p=0.60\), then measured accuracy is \(0.25+0.75(0.60)=0.70\), an increase of 0.10. This calculation is illustrative rather than a claim that every leakage mechanism gives perfect accuracy on familiar records. Its point is that evaluation can improve solely because some test cases are easier through information overlap, not because the model generalizes better.
A Practical Diagnosis Sequence
Before interpreting a validation score, write down the prediction protocol. State what one row represents, when the prediction is made, what future period defines the target, and whether deployment involves new entities or entities with known histories. Then use those facts to audit the data and partition.
For each feature, ask when its value became available—not only what date the value describes. Exclude fields first created or finalized after prediction time.
State the exact period being predicted and confirm that every included label has complete observation for that period and any required reporting delay.
Check whether copies of the same event, claim, transaction, or person occur in multiple partitions. Distinguish accidental duplicate rows from legitimate repeated measurements.
Use time separation when the task is prediction into the future; keep entities together when the task concerns new entities. Where both conditions apply, enforce both.
Describe exactly what population and information set the holdout score represents. If leakage is found, correct the design and evaluate again before interpreting performance.
A common trap is to believe that one partitioning change fixes every leakage risk. A chronological split helps prevent training on future records to predict earlier ones, but the same person can still appear on both sides. A group split keeps people together, but it can still use a post-outcome field or an immature label. The safeguards address different failure modes and must be combined when the prediction task requires them.
Check Your Understanding
For each problem, identify the relevant leakage risk and the appropriate correction.
- A model predicts whether a policy will lapse during the next three calendar months from a snapshot dated June 1. The data are complete through August 31. Is the full outcome window mature? (Answer: No; the next three calendar months are July, August, and September, so the full label is mature only after September 30, assuming September outcomes are complete.)
- A claim model predicts settlement at the time a claim is first reported. A candidate predictor is the final amount paid, recorded six months later. Should it be used? (Answer: No; it is post-outcome information unavailable at first report.)
- Two exact copies of one transaction appear in different partitions. The deployment goal is to predict outcomes for previously unseen transactions. What is the correction? (Answer: Deduplicate accidental copies or assign all records for that transaction to one partition; do not leave copies across the boundary.)
- A model will be used for policyholders with no prior records. The data contain multiple monthly rows per policyholder. What partition principle is needed? (Answer: Group by policyholder, assigning every record for a policyholder to the same partition.)
- A model will predict future monthly outcomes for existing customers. Earlier customer records are in training and later records are in holdout. Name two additional checks needed before interpreting the score. (Answer: Confirm that predictors use only information available at each prediction time, and that each holdout outcome window is fully mature; also verify that the time ordering matches deployment.)