Why Split by Entity?
A data set may contain multiple rows for the same policyholder, provider, customer, or other entity. A row-level random split treats those rows as separate observations. It can therefore place one entity’s records in both the training set and the holdout set. If the intended task is to predict outcomes for new entities, that arrangement can make evaluation too optimistic: the model has already encountered information associated with the entity it is being asked to predict.
A group-based split assigns each entity as a whole to one partition. The entity identifier is used to decide the split; it is not automatically a predictor. This approach is useful when the records are repeated measurements or transactions and the target task is generalization to entities absent from model fitting. It is not automatically the right design for every task: if the intended use is prediction for an entity already represented in training, evaluation should reflect that use case.
The first decision is therefore substantive: what counts as one entity for the prediction task? If a hypothetical insurer wants to evaluate predictions for policyholders not previously observed, policyholder is a natural grouping variable. If policies are the prediction units and multiple policies can belong to one policyholder, splitting by policy may still allow the same person to appear in both sets. The group definition should match the unit whose future or new outcomes the evaluation is meant to represent.
Group Proportions and Row Proportions
Selecting a fraction of entities does not generally select the same fraction of rows. When some entities contribute many records and others contribute few, the row count in the holdout depends on which groups are selected. A target such as “25% holdout” must therefore be interpreted carefully: it may mean about 25% of entities, not 25% of rows.
This difference can also affect the performance summary. A row-level metric gives more influence to entities with more records. An entity-level metric first summarizes each entity’s performance and then gives each entity equal influence. Neither is universally preferable; the choice should reflect the quantity the analysis is intended to estimate.
Worked Example: A Holdout With Unequal Entity Sizes
A hypothetical insurer has 8 policyholders with respectively 4, 1, 3, 2, 5, 1, 2, and 2 recorded claims. It selects policyholders 2 and 5 for holdout evaluation. The absolute prediction errors on policyholder 2’s one claim are 1 dollar; on policyholder 5’s five claims they are 3, 0, 2, 4, and 2 dollars. Calculate the group and row proportions in holdout, then compare row-weighted and entity-weighted mean absolute error.
Method. First count entities and records separately. Then calculate both metrics from the stated errors. This avoids treating a selected fraction of groups as if it must be the same fraction of rows.
There are \(8\) policyholders. The two selected policyholders make the holdout group proportion \(2/8=0.25\), or 25%. The total number of claim records is \(4+1+3+2+5+1+2+2=20\). The two holdout policyholders contribute \(1+5=6\) records, so the row proportion is \(6/20=0.30\), or 30%. The training set has 6 policyholders and \(20-6=14\) records.
The holdout absolute errors sum to \(1+3+0+2+4+2=12\) dollars across 6 rows. The row-weighted mean absolute error is \(12/6=\$2.00\). For entity weighting, policyholder 2 has mean absolute error \(1/1=\$1.00\); policyholder 5 has mean absolute error \((3+0+2+4+2)/5=11/5=\$2.20\). The average of those two entity means is \((1.00+2.20)/2=\$1.60\).
Check. The entity-weighted result gives the one-claim policyholder and the five-claim policyholder equal weight. The row-weighted result gives policyholder 5 five times as much influence as policyholder 2. The two metrics differ because the groups have unequal sizes and different average errors.
Common mistake. Calling the holdout “25% of the data” is ambiguous here. It is 25% of policyholders but 30% of claim records. Report the unit used for the proportion and state how the evaluation metric weights entities.
Assigning Groups, Not Rows
A practical group split proceeds by forming the set of unique group identifiers, assigning those identifiers to partitions, and then bringing every row for each selected identifier into the corresponding partition. The crucial check is that the training and holdout identifier sets have no overlap. In R, the conceptual operation is to sample from the unique entity IDs and use membership in the sampled IDs to select all rows; sampling row numbers directly does not enforce a group split.
State whether the goal is performance on new policyholders, new providers, or another defined entity type.
Confirm that records from the same target entity share a reliable ID, and that distinct entities are not accidentally combined.
Choose the training and holdout groups using the desired group-level allocation. Do not split the rows first and attempt to repair overlaps later.
Check both the number of groups and the number of rows in each set, and confirm that no group ID occurs in both.
A group-level allocation may yield a noticeably different row allocation when group sizes vary. If the holdout is too small in groups, the estimate may also be unstable even when it contains many rows: many repeated records from a few entities do not provide the same breadth of entity-level evaluation as records from many distinct entities.
Outcome Stratification With Groups
When outcome frequencies differ across entities, it may be useful to consider outcome composition while assigning groups. The constraint remains that a whole group goes into one partition. Stratifying individual rows and then splitting them separately would reintroduce the very within-entity overlap the group split is intended to prevent.
It also matters what outcome is being balanced. A label such as “ever had a claim” is an entity-level outcome; the fraction of positive claim records is a row-level outcome. Balancing the number of entities in each class does not guarantee that the row-level class frequency will match between partitions, especially when entities have different numbers of records. If groups contain mixed row outcomes, the distinction is particularly important.
Worked Example: Entity-Level Stratification Does Not Fix Row Prevalence
A hypothetical portfolio has eight policyholders. Four have at least one claim: A has 4 claim records, of which 2 are positive; B has 2 records, of which 1 is positive; C has 3 records, of which 1 is positive; and D has 1 record, which is positive. Four have no positive claim records: E has 3 records, F has 2, G has 1, and H has 1. To preserve the entity-level frequency of “ever had a claim,” the analyst assigns A and E to holdout. Calculate the entity-level and row-level positive frequencies in holdout and in the full data.
Method. Treat “ever had a claim” as the entity-level label, then separately count positive records for the row-level frequency. This avoids assuming the two definitions of prevalence are interchangeable.
The holdout contains 2 of the 8 policyholders, one of whom has ever had a claim. Its entity-level positive frequency is \(1/2=0.50\). The training set contains the remaining 6 policyholders: B, C, and D have ever had a claim, so its entity-level positive frequency is \(3/6=0.50\). The entity-level frequency is therefore matched.
For the row-level calculation, A contributes 4 records with 2 positives and E contributes 3 records with 0 positives. Holdout row-level prevalence is \(2/(4+3)=2/7\), approximately 0.2857. Across all entities there are \(4+2+3+1+3+2+1+1=17\) records and \(2+1+1+1=5\) positive records, giving full-data row prevalence \(5/17\), approximately 0.2941. The training set has 10 records and 3 positives, so its row prevalence is \(3/10=0.30\).
Check. The entity-level split preserves 50% positive entities in each partition, while the holdout row prevalence is about 28.57%, not the full-data row prevalence of about 29.41%. Small differences can arise because entire groups, rather than individual records, are assigned together.
Common mistake. Calling the split “stratified” without naming the unit can conceal what was balanced. State whether the strata describe entities or rows, and verify the corresponding outcome frequencies after assignment.
Why Row Splits Can Put One Entity in Both Sets
The risk of an entity appearing in both sets grows with the number of records it contributes. A simple probability calculation makes the mechanism clear.
Worked Example: Probability of a Split Entity
A hypothetical provider contributes 4 records to a data set. A row-level split independently assigns each record to holdout with probability \(q=0.25\). Calculate the probability that this provider appears in both training and holdout, and explain how a group split changes that probability.
Method. Use the complement of the two cases in which all records land on one side. This avoids counting the many possible mixed assignments one by one.
The probability all 4 records go to training is \(0.75^4=0.31640625\). The probability all 4 go to holdout is \(0.25^4=0.00390625\). These cases cannot happen at the same time, so the probability of a mixed assignment is
Thus the probability is 0.6797, or 67.97%, to four decimal places for the probability and two decimal places for the percentage. Under a group-based split, the provider’s 4 records are assigned together, so the probability that this entity appears in both partitions is zero by construction.
Check. The mixed-assignment probability is substantial because there are several ways for four rows to be divided across the two sets. The calculation concerns the split mechanism; it does not prove that a particular model’s measured performance will be optimistic. Optimism is a concern when shared-entity information makes holdout records easier to predict than records from genuinely new entities.
Common mistake. Assuming that a modest holdout percentage makes within-entity overlap negligible. The relevant probability depends on both the holdout rate and the number of rows per entity.
Interpretation and Practical Checks
A group-based holdout estimates performance on the kinds of entities represented in its holdout groups, subject to how those groups were selected. It does not automatically guarantee representative groups, balanced outcomes, or precise performance estimates. Inspect the number of groups, the distribution of group sizes, and the relevant outcome frequencies in both partitions. If a few large entities dominate a row-based metric, report that weighting clearly or provide an entity-weighted summary when that better matches the intended decision.
Group assignment can also create practical constraints. With few entities, a requested holdout fraction may be impossible to match closely. A rare outcome concentrated in one or two entities cannot be distributed across both partitions without splitting those entities. In such cases, preserving the group boundary takes priority when the objective is evaluation on new entities; the resulting limitation should be stated rather than hidden by a row-level split.
A useful final audit is to compare the unique group IDs in the two partitions and verify their intersection is empty. Then describe the split in terms of both entities and records, and state whether the metric weights rows equally or entities equally. This makes the evaluation design interpretable to a reviewer and aligned with the actual use case. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, identify the relevant unit and check the arithmetic before deciding what the split says about performance.
- A data set has 12 providers. A holdout contains 3 providers contributing 2, 4, and 7 records; the full data set contains 36 records. What are the holdout group and row proportions? (Answer: 25% of providers; \(13/36\), or approximately 36.11%, of records.)
- Under independent row assignment with holdout probability 0.20, one entity has 3 rows. What is the probability it appears in both partitions? (Answer: \(1-0.8^3-0.2^3=0.4800\), or 48.00%.)
- Two entities have absolute errors of 1 and 3 across two records, and 4 across one record, respectively. Calculate row-weighted and entity-weighted mean absolute error. (Answer: Row-weighted \(8/3\), approximately 2.6667; entity-weighted \((2+4)/2=3\).)
- There are 10 entities, 3 with a positive entity-level label. A group holdout selects 3 entities: 1 positive and 2 negative. What are the positive entity proportions in holdout and training? (Answer: Holdout \(1/3\); training \(2/7\), approximately 0.2857.)
- A prediction task concerns new policyholders, but each policyholder may have several policies and each policy may generate repeated records. Which identifier should define the groups, and why? (Answer: Policyholder ID, because all records for a policyholder must remain together when evaluating performance on previously unseen policyholders.)