Why the Fold Assignment Unit Matters
Cross-validation divides data into folds, fits a model on all but one fold, and evaluates it on the held-out fold. For a ratemaking GLM, the prediction unit might be a vehicle, location, coverage, or policy term. But the records can be related: several vehicles may belong to one policyholder, multiple insured locations may belong to one employer, and multiple records may describe the same underlying risk. If related records are split across training and validation data, the validation records are not fully new to the model-fitting process.
The fold-assignment unit should match the unit about which generalization is being assessed. When the intended question is how well the model predicts for a policyholder, employer, or risk not represented in fitting, all records for that entity must be kept together. Grouped cross-validation enforces that boundary. It changes the assignment rule, not the GLM itself or the definition of a validation metric.
A group may be a policyholder, employer, insured risk, or another entity defined by the validation target. It is not necessarily the same as a row, policy term, or exposure unit. If one policyholder has records for several terms and the target is performance on new policyholders, all those terms belong to one group. Conversely, grouping every record by a broad category such as state would be inappropriate unless the stated target is prediction for states omitted from fitting.
The Grouped Fold Design
First specify the generalization target and the grouping key. Then allocate distinct groups, rather than individual rows, to folds. The fold sizes may be checked by row count, exposure, or other relevant quantities, but balancing must never be achieved by splitting a group. With \(K\) folds, each group appears in exactly one fold and, in a standard \(K\)-fold run, each fold is held out once.
State whether validation concerns new policyholders, employers, risks, or another population. Choose the group key to match that target.
Confirm that all records representing the same entity carry the same identifier, including records from different periods when those periods belong to one group for the stated task.
Assign each distinct identifier to one fold. A deterministic assignment or a reproducible random assignment can be used, provided the grouping constraint is maintained.
Compare group counts and, where appropriate, exposure totals or important outcome features. If folds are highly uneven, reassign whole groups rather than dividing a large group.
For each held-out fold, fit the model only on the other folds and produce predictions for every record in the held-out groups. Keep the split rule fixed for the full validation procedure.
Why Grouping Prevents a Specific Form of Leakage
Suppose a group contains multiple observations with a shared, partly unmeasured feature, such as a persistent policyholder behavior or a common employer characteristic. If one observation is used in fitting and another is held out, the model’s fitted structure may benefit from information associated with that group. The held-out record then tests prediction for a group already represented in training, not prediction for a new group. The observations need not be duplicates for this mismatch to matter.
A simple calculation shows how readily a row-wise assignment can split a group. Assume a group has \(m\) records, each independently assigned with equal probability to one of \(K\) folds. The first record can be in any fold. For all \(m\) records to remain together, each of the other \(m-1\) records must match the first, with probability \(1/K\) each. Thus:
This is a justification for the grouping constraint, not a claim that every split group causes a large prediction error. It shows that independent row assignment does not reliably preserve the group boundary. Grouped assignment makes the probability of a group appearing in more than one fold exactly zero by construction.
Worked Example: Finding and Fixing Policyholder Leakage
A hypothetical insurer has 12 vehicle records belonging to six policyholders. The record counts are A: 3, B: 2, C: 1, D: 2, E: 3, and F: 1. The validation target is prediction for policyholders not used to fit the model. A proposed row-level split places two of A’s records in training and A’s third record in validation. Diagnose the issue and construct three grouped folds with equal record counts.
Method. Treat the policyholder, not the vehicle record, as the assignment unit. The row-level split fails the stated target because policyholder A appears on both sides of the split.
Step 1: Count the records. The total is \(3+2+1+2+3+1=12\). Three folds with equal record counts would contain \(12/3=4\) records each, if a whole-group allocation can achieve that balance.
Step 2: Assign complete groups. One valid allocation is Fold 1: A and C, Fold 2: B and D, and Fold 3: E and F. Their record totals are \(3+1=4\), \(2+2=4\), and \(3+1=4\), respectively.
Step 3: Check the boundary. Each policyholder occurs in exactly one fold. When Fold 1 is held out, all three A records and C’s record are excluded from fitting; the same whole-group rule applies to the other folds.
Correction and trap avoided. Keeping two A records in training while validating on the third is not grouped validation, even if the total row counts appear balanced. The corrected folds preserve equal record counts without splitting any policyholder. No loss distribution or fitted-model parameter is assumed in this allocation example; the issue is the sampling unit, not the outcome model.
Balancing Exposure Without Splitting Groups
Equal fold sizes are not always attainable or desirable. A large group can make exact balance impossible, and the number of records may not represent the amount of risk being validated. For ratemaking, exposure totals can be a useful balance check when exposure is relevant to the intended comparison. Other checks may be appropriate for sparse outcomes or important rating segments. These checks guide allocation; they do not override the requirement that a group remain intact.
Worked Example: Balancing Employer Groups by Exposure
A hypothetical commercial insurer wants to assess predictions for employers not used in fitting. It has nine employer groups. Each entry below gives the employer’s exposure in policy-years and observed claim count: E1: 4 and 2; E2: 3 and 1; E3: 2 and 0; E4: 4 and 3; E5: 3 and 1; E6: 2 and 2; E7: 1 and 0; E8: 1 and 1; E9: 1 and 0. Construct three folds with equal exposure, keeping every employer intact, and check their claim totals.
Step 1: Find the total exposure. The exposure is \(4+3+2+4+3+2+1+1+1=21\) policy-years, so the target exposure per fold is \(21/3=7\) policy-years.
Step 2: Allocate whole employers. Assign E1 and E5 to Fold 1, E2 and E4 to Fold 2, and E3, E6, E7, E8, and E9 to Fold 3. The exposures are \(4+3=7\), \(3+4=7\), and \(2+2+1+1+1=7\) policy-years.
Step 3: Check the outcomes and group counts. The claim totals are \(2+1=3\) in Fold 1, \(1+3=4\) in Fold 2, and \(0+2+0+1+0=3\) in Fold 3. The folds contain 2, 2, and 5 employers, respectively. Exposure is exactly balanced, while employer count and claim count are not.
Interpretation. This is a valid allocation for the stated group boundary, and exposure balance is achieved without dividing an employer’s experience. The unequal employer counts are a consequence of whole-group allocation. A claim count difference of one does not justify moving an individual claim record to another fold. If the model includes a rate structure and an exposure offset, the same grouping rule still applies: the offset does not make records from one employer independent.
Grouped Folds in Repeated Cross-Validation
Repeated cross-validation uses multiple fold assignments to examine how results vary with the partition. When groups are the assignment units, the grouping constraint must be enforced independently in every repetition. A group can be assigned to a different fold in a later repetition, but it must remain whole within that repetition. A row-level repeat cannot be made valid merely by averaging its results with those from a grouped split.
For each repetition, each observation is held out once and receives a prediction from a model fitted without its fold. In a grouped design, this means that the prediction is generated without any records from that observation’s group. If repeated held-out predictions are retained, preserve their repetition labels; predictions from the same record across repetitions are not independent observations simply because the fold assignments changed.
Worked Example: Checking a Repeated Four-Fold Design
A hypothetical portfolio contains eight employer groups, each with three location records, for 24 records total. The target is prediction for an employer absent from model fitting. A team plans three repetitions of four-fold cross-validation, with two whole employers assigned to each fold in every repetition. Check the number of fits, the held-out records per fold, and the chance that a three-record employer would be split by an independent row-level four-fold assignment.
Step 1: Check the group allocation. Eight employers divided among four folds gives \(8/4=2\) employers per fold. Since each employer has three records, every held-out fold contains \(2\times3=6\) records. Across the four folds, all \(8\times3=24\) records are held out once in that repetition.
Step 2: Count model fits and predictions. Four folds require four fits per repetition. Three repetitions therefore require \(3\times4=12\) fits. Each record receives one held-out prediction in each repetition, or three repetition-specific held-out predictions over the complete procedure.
Step 3: Quantify the row-splitting risk. Under independent assignment of each record to one of four folds, the probability that all three records for a given employer land together is \((1/4)^2=1/16\). Therefore the probability that they are split across folds is \(1-1/16=15/16=0.9375\).
Conclusion and trap avoided. Assigning the eight employers, rather than 24 location records, to folds guarantees that no employer crosses a fold boundary. The \(15/16\) calculation applies only to independent row assignment; it is not the split probability under the grouped design, where that probability is zero. The number of model fits is 12, and the six held-out records in each fold remain associated with two complete employers.
Choosing the Group Key and Interpreting Results
The group definition is a modeling and validation decision that should be documented alongside the fold assignment. A useful check is to ask: if this group is held out, would every record connected to the target entity also be absent from model fitting? If not, the key may be too narrow. For example, treating each policy term as a separate group does not assess new-policyholder performance if the same policyholder has other terms in training.
The opposite error is grouping too broadly. If a group combines unrelated risks, fewer distinct groups remain available for splitting, and folds may become unnecessarily uneven. The right choice is not the broadest identifier available; it is the entity whose future records are intended to be new under the validation question. The earlier material on defining the validation target and prediction unit is useful for making that choice precise.
Grouped validation is not automatically the right design for every prediction task. If deployment specifically concerns additional records for entities already represented in the model’s data, a group-held-out design may answer a different question from the intended one. State the target first, then choose the split. In all cases, grouped assignment prevents a particular form of cross-boundary information sharing; it does not by itself address every other source of information leakage or ensure that validation data represent future conditions.
Check Your Understanding
For each problem, use the stated validation target to decide the assignment unit and verify the arithmetic.
- A hypothetical auto portfolio has policyholders A, B, C, and D with 2, 1, 3, and 2 vehicle records, respectively. The target is new-policyholder performance. A two-fold allocation places A and D in Fold 1 and B and C in Fold 2. How many records are in each fold, and does the allocation preserve groups? (Answer: Fold 1 has 4 records and Fold 2 has 4 records; yes, every policyholder is kept whole.)
- Five hypothetical employers have exposures of 5, 4, 3, 2, and 2 policy-years. For two-fold validation, place the first and fifth employers together and the second, third, and fourth together. What are the fold exposures, and are they exactly balanced? (Answer: \(5+2=7\) and \(4+3+2=9\) policy-years; no, the folds are not exactly balanced, but the allocation is group-valid.)
- A risk group has four records independently assigned to one of five folds. What is the probability that the records are split across folds under row-level assignment? (Answer: \(1-(1/5)^3=124/125=0.9920\).)
- A hypothetical liability portfolio has 12 employer groups and uses six-fold grouped cross-validation. Each group has two records. If groups are divided equally among folds, how many employers and records are held out per fold? (Answer: 2 employers and 4 records per fold.)
- A commercial account has three locations in each of four annual records. The target is prediction for accounts not represented in fitting. Should the annual records be assigned as separate groups or should all 12 records remain together? Explain. (Answer: Keep all 12 records together as one account-level group, because the target is a new account and no record from that account should enter model fitting.)