Why Group Records Before Making Folds?
A data set may contain several rows for the same policyholder, clinic, vehicle, or other entity. If the goal is to predict outcomes for new entities, validation must mimic that situation: a held-out entity should be absent from the training data. Assigning folds independently to rows does not guarantee this. It can place some records for an entity in training and other records for that same entity in validation, allowing the model to benefit from information about an entity it is supposed to predict as new.
Grouped cross-validation prevents that boundary crossing. First identify the entity that must remain intact, then assign each entity to one fold. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus: the split should represent the intended prediction task, not merely divide the available rows.
Let \(g\) index groups and let \(D_g\) denote all records belonging to group \(g\). If \(G_j\) is the set of groups assigned to fold \(j\), its validation records are the union of their records, \(V_j=\bigcup_{g\in G_j}D_g\). Its training records are the remaining records. The sets \(G_1,\ldots,G_K\) must partition the groups: each group appears in one set, and in no other.
This design is appropriate when the target is performance on entities not represented in training. It is not automatically the right design for every repeated-data question. For example, if the real target concerns further observations from entities already known to the model, holding out entire entities may not represent that target. State the target first, as in the earlier material on matching partitions to the prediction target.
Constructing and Checking Grouped Folds
The essential operation is to assign fold labels to unique groups, then transfer each group's label to every row with that group ID. Do not generate a separate random fold label for each row. A practical construction is to shuffle the list of groups, then place groups into folds while tracking the number of rows already assigned. When group sizes differ, a greedy assignment to the currently smallest fold by row count can improve balance. It does not guarantee equal fold sizes, because a large group cannot be split.
Balance is a practical consideration, not a reason to break a group. If one large group dominates the data, the folds may differ materially in size. Record that limitation and inspect the distribution of group sizes and, where relevant, outcome values. Stratifying rows independently would undo the group constraint; any effort to balance outcomes must operate on whole groups.
Choose the entity whose records must stay together, such as a policyholder. Check that the identifier represents the entity consistently across the data.
Give each unique group one fold label. Use a reproducible randomized order or a deliberate balancing rule, and do not split a group to equalize row counts.
For fold \(j\), put every row from groups assigned to \(j\) in validation and all other rows in training.
For each fold, verify that the training and validation group-ID sets have an empty intersection, and that every row is validated once over the full cross-validation run.
Estimate preprocessing and the model using that fold's training records only, then predict its validation records. This prevents information from validation records entering the fitted procedure.
A useful integrity check is to compare the distinct group IDs on each side of each fold. A zero overlap is required; checking only that the row counts add up is not enough. Also check that every row receives exactly one out-of-fold prediction. If a group has several records, those records all receive predictions from the same fold-specific model.
Worked Example: Assigning Unequal-Sized Groups to Four Folds
A hypothetical insurer has records from eight policyholders. Their record counts are 4, 3, 3, 2, 2, 1, 1, and 1. The objective is to estimate prediction performance for policyholders not used to fit a model. Construct four grouped folds with reasonably balanced row counts.
Method. Assign policyholders, not individual records. There are 17 records in total, so an ideal row count would be \(17/4=4.25\) per fold. Exact equality is impossible because records from a policyholder must remain together.
Label the policyholders A through H in the order of the stated sizes. One valid assignment is: fold 1 contains A and H, fold 2 contains B and G, fold 3 contains C and F, and fold 4 contains D and E.
Check each fold's row count. Fold 1 has \(4+1=5\) rows. Fold 2 has \(3+1=4\) rows. Fold 3 has \(3+1=4\) rows. Fold 4 has \(2+2=4\) rows. The total is \(5+4+4+4=17\), as required. Each fold contains two policyholders.
In fold 1, all five records for A and H are validation records; none of A or H's records may appear in training for that fold. The same rule applies to the other folds. Each policyholder is held out once, so its records receive out-of-fold predictions once across the four-fold run.
Common mistake avoided. Splitting 17 rows into four nearly equal sets first could put, for example, two records from A in validation and two in training. That would violate the new-policyholder target even if the row counts looked perfectly balanced.
Why Row-Wise Folds Can Mislead
Repeated records from the same entity are often related. A policyholder may have a distinctive claim pattern; a clinic may have a characteristic average outcome. If records from one entity appear on both sides of a row-wise split, a model can use what it learned from that entity's training records when predicting its validation records. Its score then answers a different question from performance on an entirely new entity.
The following calculation isolates the issue. Assume a squared-error loss, \(L(y,\hat{y})=(y-\hat{y})^2\), and a hypothetical prediction rule that memorizes the mean outcome for each entity seen in training. For an entity absent from training, the rule predicts the overall training mean. These assumptions are intentionally simple so the effect of the split can be calculated exactly.
Worked Example: Comparing Row-Wise and Grouped Validation
Four hypothetical clinics each have two records. Both records for clinic A have outcome 0, both for B have outcome 10, both for C have outcome 20, and both for D have outcome 30. Compare a row-wise two-fold split that places one record from each clinic in each fold with leave-one-clinic-out validation. Use the prediction rule just defined and calculate mean squared error (MSE).
Row-wise split. In either fold, the training data contain one record from each clinic. The model therefore observes each clinic's outcome and predicts its held-out record using that clinic's training mean. The predictions are respectively 0, 10, 20, and 30. Every squared error is zero, so the row-wise MSE is \(0/8=0\).
Grouped split. Hold out one complete clinic at a time. For clinic A, the training mean is \((10+20+30)/3=20\). Each of its two validation records has squared error \((0-20)^2=400\), contributing \(800\).
For clinic B, the training mean is \((0+20+30)/3=50/3\). Each squared error is \((10-50/3)^2=(-20/3)^2=400/9\), contributing \(800/9\) for its two records. For clinic C, the training mean is \((0+10+30)/3=40/3\). Each squared error is \((20-40/3)^2=(20/3)^2=400/9\), again contributing \(800/9\). For clinic D, the training mean is \((0+10+20)/3=10\). Each squared error is \((30-10)^2=400\), contributing \(800\).
The total squared error is \(800+800/9+800/9+800=16000/9\). There are eight validation records, so grouped MSE is \((16000/9)/8=2000/9\), or 222.22 when rounded to two decimal places. The row-wise estimate of zero looks excellent but reflects that the model already saw every clinic. The grouped estimate evaluates the intended new-clinic task.
Common mistake avoided. Do not interpret a low row-wise score as evidence of strong performance on new entities when the same entities appear in training and validation. The score can be valid for a different target, but not for the one specified here.
Scoring Folds With Different Sizes
After obtaining out-of-fold predictions, choose how to aggregate their losses. Suppose record \(r\) has outcome \(y_r\), out-of-fold prediction \(\hat{y}_r\), and loss \(L(y_r,\hat{y}_r)\). The row-weighted cross-validation loss is the total loss over all records divided by the total number of records. Equivalently, it is a weighted average of fold-level mean losses, with each fold weighted by its number of validation records.
Here \(n_j\) is the number of validation records in fold \(j\), and \(n=\sum_j n_j\). The identity follows by substituting the definition of \(\widehat{R}_j\): multiplying each fold mean by \(n_j/n\) gives its total loss divided by \(n\). Summing those terms counts every record's loss once because every record belongs to exactly one validation fold.
This row-weighted summary answers a record-level question: what is the average loss over records? It gives larger groups more influence because they contribute more records. If the target instead gives every entity equal importance, first calculate an entity-level mean loss, then average those entity means. An unweighted average of fold means is yet another quantity; it weights folds equally, not records or entities. These summaries coincide only under conditions such as equal fold sizes or particular loss patterns.
Worked Example: Choosing the Correct Loss Aggregation
A grouped two-fold validation run holds out clinic A alone in fold 1 and clinics B and C together in fold 2. Clinic A has four records, each with absolute prediction error 0.5. Clinic B has two records, each with absolute error 3. Clinic C has one record with error 0. Compare the row-weighted mean absolute error (MAE), the unweighted average of fold MAEs, and the mean of clinic-level MAEs.
Fold 1. Its total absolute error is \(4(0.5)=2\), so its MAE is \(2/4=0.5\).
Fold 2. Its total absolute error is \(2(3)+1(0)=6\) across three records, so its MAE is \(6/3=2\). The unweighted average of the two fold MAEs is \((0.5+2)/2=1.25\).
Across all seven records, total absolute error is \(2+6=8\). Therefore the row-weighted MAE is \(8/7=1.142857\ldots\), or 1.14 to two decimal places. For equal clinic weighting, the clinic MAEs are 0.5, 3, and 0, giving \((0.5+3+0)/3=3.5/3=1.166667\ldots\), or 1.17 to two decimal places.
The three answers differ because they weight the validation results differently. Use the row-weighted MAE for an average record; use the equal-clinic summary if each clinic should count equally. The unweighted fold average gives equal weight to the two folds, even though their sizes differ.
Common mistake avoided. Averaging fold scores without checking fold sizes can silently change the target quantity. State whether the intended unit of equal weight is a record, an entity, or a fold.
Practical Interpretation and Common Traps
Grouped folds can be smaller in number of independent units than the row count suggests. A data set with many records but only a few entities cannot support many well-populated folds. With too few groups per fold, a validation score may be unstable or a fold may omit important outcome patterns. Reduce \(K\) when needed, inspect the group allocation, and describe the limitation rather than splitting groups to create an attractive row count.
A group ID is not necessarily a useful predictor. Including it may allow a model to learn entity-specific patterns, which is precisely why a row-wise split can be misleading for a new-entity target. Under grouped validation, a held-out entity ID may be unseen. The modeling procedure must have a defined way to handle an unseen ID, or the ID should not be used as a predictor. A validation pipeline that cannot make predictions for a new group does not answer the intended question.
Keep the purpose of the grouped split explicit when reporting results. “Grouped cross-validation by policyholder, with each policyholder held out as a unit” is more informative than simply “four-fold cross-validation.” Also state the metric and aggregation unit. These details let a reader understand which entities the evaluation represents and how the reported score was calculated.
Check Your Understanding
For each problem, identify the group-level decision before calculating or interpreting the validation result.
- A portfolio has six policyholders with 3, 2, 2, 1, 1, and 1 records. Construct three folds with three records per fold, keeping policyholders intact. How many policyholders are in each fold in one valid assignment? (Answer: Assign the 3-record policyholder to fold 1; pair the 2-record policyholder with a 1-record policyholder in each of folds 2 and 3. The fold counts are 1, 2, and 2 policyholders, with 3 records in each.)
- A row-wise five-fold split places some records from the same customer in training and others in validation. The target is prediction for customers absent from training. Is this split suitable, and what must be changed? (Answer: No. Assign each unique customer to exactly one fold, then put all records for that customer in that fold's validation set.)
- Three clinics each have two records with outcomes 2, 8, and 14 respectively, constant within clinic. In leave-one-clinic-out validation, predict a held-out clinic using the mean outcome of the other two clinics. Find the pooled record-level MSE. (Answer: The clinic predictions are 11, 8, and 5; squared errors per record are 81, 0, and 81. Pooled MSE \(=2(81+0+81)/6=54\).)
- Two grouped folds have 3 and 5 validation records, with fold MAEs 2 and 4. Find the row-weighted MAE and the unweighted average of fold MAEs. (Answer: Row-weighted MAE \(=(3(2)+5(4))/8=3.25\); unweighted fold average \(=(2+4)/2=3\).)
- A validation report gives one MAE per entity: 1, 2, and 6. The three entities have 10, 2, and 1 records. Find the equal-entity mean MAE, and explain why it differs from a record-weighted entity summary. (Answer: Equal-entity mean \(=(1+2+6)/3=3\); the record-weighted summary is \((10(1)+2(2)+1(6))/13=20/13\approx1.5385\), because the larger entity receives more weight.)