Why the Split Unit Matters
A policy portfolio may contain several records for one person: renewals, endorsements, or multiple policy terms. Those records are not automatically independent observations. They can share stable characteristics, history, and information that makes one record unusually informative about another. If a model will be used for people who have never appeared in the modeling data, validation must mimic that situation.
The key decision is therefore not simply how many rows to put in training and testing. It is which units must be kept separate. For a new-policyholder deployment target, every record belonging to a given policyholder must be assigned to the same partition. A row-level random split can put one renewal in training and another renewal for the same person in testing, making the test set less representative of genuinely new people.
This design assumes that the policyholder identifier correctly links repeated records and that the intended prediction target is a person not represented in training. If the operational task is instead to predict a later renewal for an already-known person, a time-based design may be more appropriate. Group separation and time separation answer different questions; neither automatically substitutes for the other.
How Row-Level Splitting Creates Leakage
Suppose each policyholder has repeated records and rows are assigned independently to training with probability \(0.8\) and to testing with probability \(0.2\). For a person with three records, the chance that all three go to training is \(0.8^3=0.512\). The chance that all three go to testing is \(0.2^3=0.008\). Thus the chance that the person is represented in both partitions is:
Nearly half of such three-record groups are expected to cross the partition boundary under this row-level procedure. That does not mean every model will exploit the overlap equally, but it means the test result is not a clean assessment for new policyholders. A model can benefit from stable person-specific patterns even if the identifier itself is excluded as a predictor.
Worked Example: Quantifying Row-Level Leakage
Worked Example: Quantifying Row-Level Leakage
A hypothetical portfolio has 20 policyholders, each with exactly three annual records. An analyst independently assigns every row to training with probability \(0.8\) and to testing with probability \(0.2\). Find the expected number of policyholders represented in both partitions, then state a suitable correction.
Method. Calculate the probability of a mixed assignment for one policyholder, then multiply by the 20 groups. The assignments are assumed independent across rows and policyholders.
For one policyholder, all three records are in training with probability \(0.8^3=0.512\). All three are in testing with probability \(0.2^3=0.008\). These are the only two assignments that do not cross the boundary, so:
Let \(I_j\) equal 1 if policyholder \(j\) is split across the partitions and 0 otherwise. Then the number of mixed groups is \(\sum_{j=1}^{20} I_j\), whose expectation is the sum of the individual expectations:
The expected number is 9.6 mixed policyholders. This is an expectation, not a claim that a particular realized split must contain 9.6 people.
Correction. Randomly assign the 20 policyholder IDs to partitions, then include all three records for each person in that person’s assigned partition. The method avoids the common mistake of splitting rows first and hoping repeated records happen to stay together.
Check. The two probabilities excluded from the mixed case sum to \(0.512+0.008=0.520\); therefore the mixed probability is \(1-0.520=0.480\), as used above.
Designing a Group-Based Holdout
A practical holdout plan begins by defining the population and deployment target, then assigning unique policyholders—not records—to training and test sets. Only after that assignment are the records brought into each partition. The following procedure makes the unit of separation explicit.
State whether predictions concern unseen policyholders, future periods for known policyholders, or another population. This tutorial’s group split targets unseen policyholders.
Use the identifier that links all records for one policyholder. Check for missing IDs, inconsistent IDs, and records that should belong to the same person but do not.
Choose the holdout proportion at the policyholder level. If useful, balance groups using available group-level information, while retaining each entire group in one partition.
Confirm that the intersection of training and test IDs is empty, and that every source record is assigned exactly once.
Decide whether the result should represent a typical policy record or a typical policyholder. Use that choice consistently when summarizing performance.
Balancing needs care. A group with many records contributes more rows than a group with one record, so a simple group-count split can produce very different record counts. Conversely, balancing only row counts can put a small number of large groups in one partition. A defensible plan may consider both group counts and row counts, and, where appropriate, the distribution of outcomes across groups. Balancing must never be achieved by scattering one person’s records across partitions.
Worked Example: Building a New-Policyholder Holdout
Worked Example: Building a New-Policyholder Holdout
A hypothetical insurer has the following 15 annual records. It reserves policyholders G7 and G8 as the test groups before model fitting. The table gives the number of records per group; for the four test records it also gives observed claim counts and model predictions. Calculate test mean absolute error (MAE) by record and verify that the split suits a new-policyholder target.
| Policyholder | Records | Partition | Test actual counts | Test predictions |
|---|---|---|---|---|
| G1 | 2 | Training | — | — |
| G2 | 1 | Training | — | — |
| G3 | 3 | Training | — | — |
| G4 | 2 | Training | — | — |
| G5 | 1 | Training | — | — |
| G6 | 2 | Training | — | — |
| G7 | 2 | Test | 0, 1 | 0.2, 0.6 |
| G8 | 2 | Test | 2, 0 | 1.5, 0.5 |
Partition check. Training contains \(2+1+3+2+1+2=11\) records from six policyholders. Testing contains \(2+2=4\) records from two different policyholders. The total is \(11+4=15\) records, and no test ID appears in training.
Calculate absolute errors. For G7, the errors are \(|0-0.2|=0.2\) and \(|1-0.6|=0.4\). For G8, they are \(|2-1.5|=0.5\) and \(|0-0.5|=0.5\). The total absolute error is \(0.2+0.4+0.5+0.5=1.6\), over four test records.
The test MAE is 0.4 claims per record. Since both held-out policyholders were absent from training, the partition matches the stated new-policyholder target. The common mistake would be to reserve only two rows at random: that could leave other records for G7 or G8 in training and produce an optimistic assessment for new people.
Check. The four errors sum to 1.6 and there are four test observations, so the calculated MAE is \(1.6/4=0.4\).
Group-Based Cross-Validation
Cross-validation repeats the same separation principle across several folds. In \(K\)-fold group cross-validation, each policyholder is assigned to exactly one of \(K\) folds. For each run, one fold is held out and the other \(K-1\) folds are used for fitting. Every record for a held-out policyholder is scored in that run, and that policyholder’s records are not used for fitting that run.
The folds should be checked at the ID level, not just by comparing row counts. With rare outcomes, it can be useful to distribute groups with and without claims across folds, but any stratification must operate on whole groups. A large group can dominate a fold’s number of rows; this is a balancing issue, not a reason to split the group.
A final choice is what the score should weight. A record-weighted error answers a question about a typical record. A policyholder-weighted error first summarizes a person’s records and then gives each person equal weight. With repeated records, those are different targets. State the scoring unit before interpreting a validation result.
Worked Example: Grouped Folds and the Scoring Unit
Worked Example: Grouped Folds and the Scoring Unit
A hypothetical four-policyholder portfolio is evaluated using two-fold cross-validation. Assign A and B to fold 1, and C and D to fold 2. The absolute prediction errors are: A has one record with error 1; B has one record with error 3; C has three records, each with error 2; D has one record with error 6. Calculate each fold MAE, the simple average of fold MAEs, the pooled record MAE, and the equal-policyholder average.
Method. Keep each person’s records in one fold. Calculate the fold summaries, then compare the two possible weighting conventions rather than treating them as interchangeable.
Fold 1 contains A and B, with errors 1 and 3. Its MAE is \((1+3)/2=2\). Fold 2 contains C’s three errors and D’s one error, so its MAE is \((2+2+2+6)/4=12/4=3\). The simple average of the two fold MAEs is \((2+3)/2=2.5\).
For the pooled record MAE, add all six record errors: \(1+3+2+2+2+6=16\). There are six records, so the pooled MAE is \(16/6=2.6667\), rounded to four decimal places. (The denominator is six; the numerator is 16.)
For an equal-policyholder summary, the four individual MAEs are A: 1, B: 3, C: \((2+2+2)/3=2\), and D: 6. Their mean is \((1+3+2+6)/4=12/4=3\).
These answers differ because the fold average weights folds equally, the pooled MAE weights records equally, and the policyholder average weights people equally. For a record-level deployment target, the pooled record MAE is the natural summary here. For a typical new policyholder, the equal-policyholder summary may better express the target. The common mistake is to call these summaries equivalent when fold sizes and group sizes differ.
Check. Fold 1 has two records and total error 4; fold 2 has four records and total error 12. The pooled total is 16 across six records, while the person-level total is 12 across four people.
Common Traps and Limits
- Splitting records instead of people. A random row split does not guarantee person-level separation. Group IDs must determine partition membership.
- Using an unreliable identifier. If one person has multiple IDs, their records may still cross partitions. If an ID is reused for different people, unrelated records may be forced together. Resolve the linkage problem before splitting.
- Balancing by breaking groups. Fold sizes need not be exactly equal. A small imbalance is preferable to putting one policyholder in both training and testing.
- Ignoring the deployment target. Group-based validation estimates performance for unseen people. It does not by itself test performance in a later calendar period, nor does it estimate performance on future records for people already represented in training.
- Leaving the score’s unit unstated. Record-weighted and policyholder-weighted results answer different questions when group sizes vary.
A grouped split removes one important source of leakage, but it does not guarantee that the test population represents every future portfolio or that other sources of information leakage are absent. The split should be documented with its target population, grouping ID, partition rule, and scoring unit. These design choices are aligned with the SOA Exam PA (Predictive Analytics) syllabus because they connect validation mechanics to the actual prediction task.
Check Your Understanding
For each problem, keep the deployment target and the unit of separation explicit.
- A hypothetical portfolio has 12 policyholders with two annual records each. Rows are independently assigned to training with probability 0.75 and testing with probability 0.25. What is the probability that one policyholder appears in both partitions? (Answer: \(1-(0.75^2+0.25^2)=0.3750\))
- Six policyholders have record counts 1, 2, 2, 3, 1, and 3. The last two policyholders are held out. How many test records are there, and how many policyholders are in the training set? (Answer: 4 test records and 4 training policyholders)
- In a grouped test set, one person has absolute errors 0.5 and 1.5, while a second person has one error of 2.0. What are the record-weighted MAE and equal-policyholder average of individual MAEs? (Answer: record-weighted MAE \(=4/3=1.3333\); equal-policyholder average \(=(1.0+2.0)/2=1.5\))
- In three-fold group cross-validation, policyholders P1 and P2 are assigned to fold 1, P3 to fold 2, and P4 and P5 to fold 3. When fold 2 is held out, which policyholders may be used for fitting, and which must be excluded? (Answer: fit using P1, P2, P4, and P5; exclude all records for P3)
- A model will score later renewals for people whose earlier policies are already in the training data. Is a group-disjoint split by itself a direct match to this deployment target? State the design issue. (Answer: No; group-disjoint validation targets unseen policyholders, while this target concerns later records for known people, so the time ordering and known-person deployment scenario must be represented.)