Why Related Records Require Careful Evaluation
An evaluation score is useful only to the extent that its test observations represent the cases to which predictions will be applied. A random split of rows can fail at this when several rows come from the same person, policy, household, clinic, device, or other source. Records from one source often share characteristics that make them more alike than records from different sources.
If records from the same source appear in both the development data and the evaluation data, a model may benefit from information specific to that source. Its evaluation score can then look strong even when the intended task is to predict for a source it has never encountered. This is leakage across related or repeated observations. The split can be leakage-prone even when no outcome value was accidentally copied into a predictor.
The right split depends on the prediction goal. Predicting another record for a policy already represented in the data is not the same task as predicting for a new policyholder. Row-wise evaluation may be appropriate for the first goal if it reflects the actual future information available. For a new-policyholder goal, the evaluation data should contain policyholders absent from the development data. In either case, the split must respect what information would truly be available at prediction time.
How a Row Split Can Create Overlap
Suppose a group has \(m\) records, and each record is independently assigned to training with probability \(p\) or evaluation with probability \(1-p\). Conditional on one particular record being assigned to evaluation, each of its other \(m-1\) records still has probability \(p\) of being in training. The probability that none of those related records is in training is \((1-p)^{m-1}\). Therefore, the probability of at least one related training record is:
This calculation uses independent row assignment; a fixed-size random partition has slightly different probabilities. The formula nevertheless shows why even a routine split can create substantial overlap when groups contain several records.
Worked Example: Quantifying the Chance of Group Overlap
A hypothetical study has four repeated measurements per participant. Each row is assigned independently, with probability \(0.8\) of entering training and probability \(0.2\) of entering evaluation. Given that a measurement is in evaluation, calculate the probability that its participant has at least one other measurement in training. State the implication for a task intended to predict for new participants.
Method. Use the complement rule: first calculate the chance that all three other measurements avoid training.
There are \(m-1=4-1=3\) other measurements. For any one of them, the probability of not being in training is \(1-0.8=0.2\). Under the stated independent assignment assumption, the probability that all three avoid training is:
Thus, the probability that at least one related measurement is in training is:
The result is \(0.9920\), or \(99.20\%\). A test row will almost certainly have a participant represented in training. That evaluation does not, by itself, establish performance for new participants. The common mistake is to report \(80\%\) as the overlap probability because it is the training fraction; the relevant calculation conditions on a row being in evaluation and asks whether any of its three related rows is in training.
Group-Wise Evaluation and What It Measures
In a group-wise split, the unit assigned to a subset is the group rather than the individual row. If a policyholder has six records, all six go to training or all six go to evaluation. This prevents direct overlap of that policyholder across the split. A group-wise test score is informative about new groups only when the held-out groups resemble the groups to which the model will be applied.
When groups are used in cross-validation, each fold holds out complete groups and trains on the remaining groups. To summarize error, decide whether the target is an average over records or an average over groups. These may differ when group sizes differ. The record-weighted mean absolute error gives more weight to groups with more records; the equally weighted group mean gives each group the same weight. Neither is universally correct: use the weighting that matches the target prediction population.
Worked Example: Comparing Row-Wise and Group-Wise Error
A hypothetical pricing model is evaluated on four groups of policyholders, each with two repeated records. Both records in each group have the same target value: group A has \(10\), B has \(20\), C has \(30\), and D has \(40\). The model memorizes the mean target for a group when it has training records from that group; for a group absent from training, it predicts the mean of the training-group means. Compare a row-wise split that places one record from each group in training and one in evaluation with leave-one-group-out evaluation. Use mean absolute error (MAE). Assume the two records within each group are identical for this illustration.
Row-wise split. Every group appears in training, so the model knows each group's target mean. The four evaluation predictions are \(10, 20, 30,\) and \(40\), matching the targets. The absolute errors are \(0, 0, 0,\) and \(0\), giving:
Leave-one-group-out evaluation. Hold out one entire group at a time. If A is held out, the training-group mean is \((20+30+40)/3=30\), so the absolute error is \(|10-30|=20\). If B is held out, the training mean is \((10+30+40)/3=80/3\), so the error is \(|20-80/3|=20/3\). If C is held out, the training mean is \((10+20+40)/3=70/3\), so the error is \(|30-70/3|=20/3\). If D is held out, the training mean is \((10+20+30)/3=20\), so the error is \(|40-20|=20\).
Each group has two identical records, so the record-level errors within a held-out group are both the same. The leave-one-group-out MAE is therefore:
The row-wise score of zero reflects the model's access to the same groups in training and evaluation; it is not evidence of zero error for new groups. The group-wise score is much more relevant to that new-group goal under this example's model and data. The common mistake is to treat two repeated records as two independent examples of generalization when both carry the same group-specific information.
Repeated Records and the Target Population
A related-record problem can arise even if rows are not exact duplicates. Two records for one person may share stable characteristics; multiple transactions from one policy may share underwriting information; measurements from one device may share calibration effects. If the model can recognize or infer these common features, an evaluation row from a familiar group may be easier than a row from a new group.
A useful diagnosis is to ask two questions before splitting:
- What is the unit of future prediction? Is it a new transaction for an existing customer, a new customer, a new clinic, or a new time period?
- What information will be available then? If the prediction uses prior history for an existing customer, that history may legitimately be available. If the target is a new customer, customer-specific history cannot be assumed.
These questions prevent a mechanical rule from replacing the actual prediction task. A group-wise split is not automatically superior for every problem. For a model explicitly intended to update predictions for known entities using earlier records, an evaluation split can preserve entities while respecting time order. But the evaluation records must not use information from the future, and a separate new-entity evaluation may be needed if performance for unfamiliar entities is also important.
Worked Example: Interpreting Classification Accuracy for New People
A hypothetical wearable-device study has six participants, each with two similar sensor records and a binary outcome that is constant within participant. Four participants have outcome \(0\), and two have outcome \(1\). A classifier can memorize a participant's outcome when that participant appears in training; for an unseen participant it predicts the training majority class. In a row-wise split, exactly one record per participant is in training and one in evaluation. In a group-wise evaluation, hold out one outcome-\(0\) participant and one outcome-\(1\) participant; the other four participants are in training. Calculate both accuracies, treating every evaluation record equally.
Row-wise evaluation. Each of the six participants is represented in training. The classifier memorizes the participant's outcome, so it correctly classifies all six evaluation records. Accuracy is:
Group-wise evaluation. The training data contain three participants with outcome \(0\) and one with outcome \(1\). The majority class is \(0\), so the classifier predicts \(0\) for both records from each held-out participant. The two records for the held-out outcome-\(0\) participant are correct; the two records for the held-out outcome-\(1\) participant are incorrect. Therefore:
The row-wise result measures performance when the same participants are already represented and their outcomes can be memorized. The group-wise result measures performance on the two unseen participants in this specified split. It does not claim that every new-person accuracy will equal \(50\%\); it illustrates why the row-wise score does not answer the new-person question. The common mistake is to treat two records from one person as independent evidence about performance on two different people.
Choosing the Split and the Score
A group-aware split addresses overlap between groups; it does not solve every evaluation problem. In particular, it does not automatically preserve time order. If observations occur over time and future predictions are the goal, training should use only information available before the evaluation period. When both new groups and future periods matter, the design may need to hold out groups and a later time period together.
Specify whether the future case is a new row from a known group or a row from a group not used in model development.
Find repeated entities, clustered sampling units, duplicate or near-duplicate records, and other sources that could place closely related observations on both sides of a split.
For new-group prediction, assign whole groups together. For future-time prediction, also ensure that training information precedes evaluation information.
Use record-weighted or group-weighted error according to the population quantity of interest, and state which weighting was used.
Group-wise evaluation may have fewer independent units than the row count suggests. For example, 2,000 records from 40 clinics are not 2,000 independent opportunities to test performance on a new clinic. A group split can also produce variable scores when there are few groups, or when groups differ greatly. Interpret the score in light of the number and composition of held-out groups rather than assuming that a large row count guarantees a stable assessment.
Check Your Understanding
For each problem, identify the relevant split or calculation and check whether the stated evaluation target is respected.
- A dataset has five visits per patient. Each row is independently assigned to training with probability \(0.7\). Given that one visit is in evaluation, what is the probability that at least one of the patient's other visits is in training? (Answer: \(1-(0.3)^4=0.9919\), or \(99.19\%\).)
- A model will predict for entirely new households. The dataset contains several records per household. Should all records be randomly split row by row, or should entire households be assigned together? (Answer: Assign entire households together, so a household cannot appear in both development and evaluation data.)
- A validation set contains 8 records from group A, each with absolute error \(2\), and 2 records from group B, each with absolute error \(8\). What are the record-weighted MAE and the equally weighted mean of the two group MAEs? (Answer: Record-weighted MAE \(=(8(2)+2(8))/10=3.2\); equally weighted group MAE \(=(2+8)/2=5\).)
- A study evaluates predictions for future years for existing policyholders. Each row is a policy-year, and the model may use earlier years but not future information. Is holding out whole policyholders the only valid design? (Answer: No. A time-ordered split can evaluate future years for known policyholders if it uses only information available before each prediction; a separate group-wise evaluation is needed for a new-policyholder target.)
- There are 12 clinics, with 100 patient records per clinic. A random row split places records from every clinic in both training and evaluation. The stated goal is performance at a clinic never represented in training. Does the row count of 1,200 make this evaluation appropriate? (Answer: No. Hold out entire clinics; the relevant independent units for that goal are clinics, not individual rows.)