Why the Unit of Splitting Matters
A data set may contain many rows but far fewer independent sources of information. For example, an insurer might record several claims for each policyholder, a health study might record repeated observations for each patient, or a model might use measurements from several sites. Rows from the same source can be similar because they share characteristics that are not present in the recorded predictors.
When the intended prediction is for an entirely new household, patient, or site, validation must test that task. If some rows from a cluster are used for training and other rows from that same cluster are used for validation, the model may benefit from information about a cluster it would not know at deployment. This can make its measured performance too optimistic for new-cluster predictions.
The splitting unit should match the prediction target. If deployment concerns another record from a cluster already represented in the training data, row-wise validation may be relevant. If deployment concerns a cluster not represented in training, the cluster—not the individual row—must be held out. These are different predictive questions and can have different error rates.
Constructing Grouped Folds
Let there be \(G\) clusters, with cluster \(g\) containing \(n_g\) observations. In \(K\)-fold grouped cross-validation, assign each cluster to exactly one fold, while aiming for a reasonable balance in the number of observations or clusters per fold. For fold \(k\), fit the model on all observations whose clusters are not in fold \(k\); evaluate it on all observations belonging to clusters in fold \(k\). Repeat until each cluster has been held out once.
Balancing fold sizes is useful, but it is secondary to keeping clusters intact. When cluster sizes vary, equal numbers of clusters per fold need not give equal numbers of observations. A practical allocation can balance total observations while preserving the grouping. The assignment should also be made at the cluster level, not by assigning individual rows independently.
Decide whether validation should represent new observations from known clusters or observations from entirely new clusters.
Record which observations belong to each subject, household, site, or other relevant unit.
Balance fold sizes where practical, but never divide a cluster between folds when estimating performance on new clusters.
For each fold, fit using the other folds and calculate the chosen prediction loss on the held-out clusters.
Worked Example: Allocating Clinics to Three Folds
A hypothetical health model is intended for use at clinics not represented in its training data. Six clinics have the following numbers of records: A has 4, B has 2, C has 3, D has 3, E has 2, and F has 2. Construct three grouped folds with reasonably balanced record counts.
Method. Assign whole clinics, aiming for three similar fold sizes. The total number of records is \(4+2+3+3+2+2=16\), so the ideal average is \(16/3=5.333\) records per fold.
One valid allocation is fold 1: clinics A and E, for \(4+2=6\) records; fold 2: clinics B and C, for \(2+3=5\) records; fold 3: clinics D and F, for \(3+2=5\) records. The fold totals are 6, 5, and 5, which sum to 16. Every clinic appears in exactly one fold.
For the fold 1 assessment, train on the records from B, C, D, and F, then evaluate on A and E. For fold 2, train on A, D, E, and F, then evaluate on B and C. For fold 3, train on A, B, C, and E, then evaluate on D and F.
Check. No clinic is split, and each record is held out exactly once across the three assessments. A row-wise split would not satisfy the new-clinic target if it placed records from the same clinic in both training and validation.
Common mistake avoided: treating 16 records as 16 independent splitting units. Here the six clinics are the units that must be allocated.
Why Row-Wise Splitting Can Be Optimistic
Suppose observations from one cluster share a latent feature, such as a household-specific pattern, a site-specific measurement process, or a subject-specific baseline. When rows from that cluster appear in both training and validation, the training data can reveal part of that pattern. A validation prediction may then be easier than a prediction for a cluster never seen during fitting.
The justification follows directly from the validation task. A new-cluster assessment requires the training set to contain no observations from a validation cluster. Grouped folds enforce that condition by construction. A row-wise split does not: for a cluster with multiple observations, some rows can enter training while the remaining rows enter validation. Thus row-wise validation can estimate performance on partially familiar clusters rather than on new clusters.
This does not mean grouped cross-validation is always preferable. It is appropriate when the target is generalization to new clusters and the observed clusters are representative of the future clusters. If the intended use is prediction for additional records from clusters already represented in training, a split that holds out entire clusters may answer a harder, different question.
Worked Example: Comparing Row-Wise and Grouped Validation
A hypothetical model is evaluated on 12 records from six households, with two records per household. The intended use is prediction for new households. In a three-fold row-wise split, the model makes one error among the 12 held-out predictions. In a three-fold grouped split, it makes seven errors among the 12 held-out predictions. Calculate both error rates and interpret the difference.
Row-wise error rate. The total number of errors is 1 and the total number of held-out predictions is 12, so \(1/12=0.0833\) to four decimal places.
Grouped error rate. The total number of errors is 7 and the total number of held-out predictions is 12, so \(7/12=0.5833\) to four decimal places.
The grouped estimate is \(0.5833-0.0833=0.5000\), or 50 percentage points, higher. Under the row-wise design, a validation record's household may also contribute a record to training. Under the grouped design, all records from a held-out household are absent from training, matching the stated new-household target.
These results illustrate how the split can change the question being answered; they do not establish that grouped validation must always produce a higher error. In an actual analysis, the fitted models and held-out predictions determine the rates.
Common mistake avoided: interpreting the row-wise rate as performance on new households merely because validation rows were excluded from their own training folds. Other rows from those same households can still be present in training.
Choosing How to Average Validation Loss
Keeping clusters intact determines which observations are held out. A separate decision is how to combine their losses. For a binary outcome, define an error indicator \(I_{gj}\) equal to 1 if observation \(j\) in cluster \(g\) is misclassified and 0 otherwise. A row-weighted error rate gives each observation equal weight. An equal-cluster error rate first calculates each cluster's error rate and then gives each cluster equal weight.
The first rate estimates error for a randomly selected record from the evaluated records, so larger clusters contribute more. The second summarizes the error rate for a typical cluster when clusters are weighted equally. Neither is universally correct: select the one that matches the intended performance summary. For example, if each site matters equally for a decision, equal-site weighting may be more relevant than giving the largest site the greatest influence.
The distinction follows from the weights. In the row-weighted rate, a cluster with \(n_g\) records contributes \(n_g\) terms to the numerator and denominator. In the equal-cluster rate, its within-cluster rate is one of \(G\) terms, regardless of its size. The rates can therefore differ even when they are calculated from the same held-out predictions.
Worked Example: Row-Weighted Versus Equal-Household Error
A hypothetical classification exercise has six held-out households. Their record counts and numbers of classification errors are: H1, 3 records and 1 error; H2, 2 and 0; H3, 2 and 1; H4, 2 and 0; H5, 1 and 1; H6, 3 and 0. Calculate the row-weighted and equal-household error rates.
Row-weighted rate. The total number of errors is \(1+0+1+0+1+0=3\). The total number of records is \(3+2+2+2+1+3=13\). Thus the row-weighted rate is \(3/13=0.2308\) to four decimal places.
Equal-household rate. The household-specific rates are H1: \(1/3\), H2: \(0/2=0\), H3: \(1/2\), H4: \(0/2=0\), H5: \(1/1=1\), and H6: \(0/3=0\). Their sum is \(1/3+1/2+1=11/6\), so their average is \((11/6)/6=11/36=0.3056\) to four decimal places.
The equal-household rate is higher because the one-record household H5 has an error and receives the same weight as each other household, while the three-record household H6 has no errors. H1 also has an error, but it has three records and its household error rate is \(1/3\), not 1.
Common mistake avoided: confusing the number of households with the number of records, or describing H1 as a one-record household. The data give H1 three records; only H5 has one record.
Common Traps and Practical Checks
- Splitting rows instead of clusters. A random row split can put the same subject or site in training and validation. Confirm that each cluster ID occurs in only one fold.
- Using the wrong validation target. State whether the goal is prediction in a known cluster or in a new one before choosing the split.
- Assuming equal fold sizes are more important than intact groups. Unequal fold sizes may be unavoidable when cluster sizes vary. Do not split a cluster just to make the row counts identical.
- Leaving the metric ambiguous. A row-weighted loss and an equal-cluster loss answer different averaging questions. State which one is reported.
- Overgeneralizing the result. Grouped validation estimates performance for future clusters only to the extent that the held-out clusters represent the future prediction setting.
Check Your Understanding
For each problem, identify the splitting or scoring calculation that matches the stated target.
- A study has four sites with 5, 4, 3, and 2 records. Construct two grouped folds with balanced record counts. (Answer: put the 5-record and 2-record sites in one fold and the 4-record and 3-record sites in the other; each fold has 7 records.)
- A model will be used for new policyholders, and each policyholder contributes several claims. Should claims be assigned independently to folds, or should all claims from a policyholder remain together? (Answer: keep all claims from each policyholder together, assigning the entire policyholder to one fold.)
- Three held-out households have sizes 2, 2, and 4, with 1, 1, and 0 errors, respectively. Calculate the row-weighted and equal-household error rates. (Answer: row-weighted \(2/8=0.2500\); equal-household \((1/2+1/2+0)/3=1/3=0.3333\), to four decimal places.)
- In a three-fold grouped assessment, the folds contain 6, 7, and 5 records. Their error counts are 2, 1, and 2. What is the row-weighted error rate across all held-out records? (Answer: \(5/(6+7+5)=5/18=0.2778\), to four decimal places.)
- A model is intended to predict outcomes at hospitals with no training records. A proposed split holds out some patients but leaves other patients from every hospital in training. Does this validate the stated target? (Answer: no; hold out entire hospitals so no validation hospital contributes records to training.)