Why the Unit of Splitting Matters
A genomic data set may contain several rows for one participant: technical replicates, repeated visits, or samples from different tissues. Those rows are distinct measurements, but they are not necessarily independent examples of the prediction task. They can share a genotype, stable molecular features, processing history, and the same outcome. If one row is used for training and another row from that participant is used for testing, the model may exploit information that would not be available for a genuinely new participant.
The key design choice is therefore not simply how many rows to place in each fold. It is which unit the validation is meant to represent. If the intended use is prediction for a new participant, all samples from one participant must be held out together. If the intended use is prediction for another measurement from a participant whose other measurements are already available, a sample-level split may answer a different, narrower question. The split should reproduce the intended prediction setting.
Let row \(i\) have predictors \(x_i\), outcome \(y_i\), and group identifier \(g(i)\). A group-disjoint split into training indices \(T\) and test indices \(V\) satisfies
This condition is about the grouping unit, not the number of rows. A participant with six tissue samples still contributes one group, while a participant with one sample also contributes one group. Technical replicates should likewise remain together when the target is performance on an independent biological sample or participant.
How Row-Wise Splitting Creates Identity Leakage
A random row split treats rows as though they can be separated independently. For repeated samples, this can put a participant's measurements on both sides of the split. A model may then recognize stable participant-specific patterns, or simply memorize the relationship between an identity-linked feature and that participant's outcome. This is called identity leakage in this setting: information tied to a test participant is available in training through another row from the same participant.
A simple probability calculation shows why even a modest number of repeated rows matters. Suppose a participant has \(m\) rows, and each row is independently assigned to training with probability \(r\) or to test with probability \(1-r\). Conditional on one specified row being in test, each of the other \(m-1\) rows has probability \(r\) of being in training. The probability that at least one other row from that participant is in training is therefore
For \(m=3\) and \(r=0.8\), this probability is \(1-(0.2)^2=0.96\). Thus, under this independent assignment model, 96% of test rows from three-row participants have at least one same-participant row in training. Fixed-size random splits have a different exact probability, but the underlying risk remains: splitting rows does not guarantee separation of identities.
Worked Example: A Fingerprint Feature Makes Sample-Level Accuracy Misleading
Consider a hypothetical data set with four participants, each measured twice. The outcome is a participant-level binary label, and a feature records a stable participant fingerprint. The same fingerprint appears in both samples from that participant.
| Sample | Participant fingerprint | Outcome |
|---|---|---|
| A1 | A | 0 |
| A2 | A | 0 |
| B1 | B | 0 |
| B2 | B | 0 |
| C1 | C | 1 |
| C2 | C | 1 |
| D1 | D | 1 |
| D2 | D | 1 |
For illustration, define a deliberately simple exact-recognition classifier: if a test fingerprint was observed in training, predict the training outcome associated with that fingerprint; if it was not observed, predict 0. This classifier has access to no special scientific knowledge—it simply memorizes fingerprint-outcome pairs.
Sample-level split. Put A1, B1, C1, and D1 in training, and A2, B2, C2, and D2 in testing. Every test fingerprint has an exact match in training: A2 matches A1, B2 matches B1, C2 matches C1, and D2 matches D1. Because each participant has the same outcome in both rows, the classifier makes four correct predictions. Its test accuracy is
That perfect score measures recognition of participants already represented in training, not prediction for new participants. The result follows specifically because the fingerprint feature is the same for both samples from each participant and because the participant-level outcomes agree across those samples.
Participant-level split. Instead, train on both A samples and both B samples, and test on both C samples and both D samples. Neither test fingerprint, C nor D, occurs in training. The stated classifier therefore uses its default prediction of 0 for all four test rows. All four true outcomes are 1, so the accuracy is
This toy example is intentionally extreme, but it isolates the issue: row-wise splitting let the classifier see each test participant's identity during training. Group-disjoint splitting removes that route to apparent success. It does not establish that every low score is caused by leakage, nor that every group-disjoint score is unbiased for every deployment setting.
Constructing Group-Disjoint Folds
The basic procedure is to assign group identifiers to folds, then place all rows belonging to each identifier in its assigned fold. Each model is trained on the rows in the other folds and evaluated on the held-out fold. The folds need not contain the same number of rows: unequal group sizes make exact row balance incompatible with keeping groups intact.
Decide whether the intended test cases are new participants, new visits from known participants, new tissues, or another unit. This determines which observations must be held out together.
Use the participant identifier, or another identifier corresponding to the independent prediction unit. Check that all replicate and tissue rows assigned to that unit share one group value.
Assign each group to exactly one fold. Where feasible, balance the number of groups and the distribution of outcomes across folds without breaking groups apart.
Fit the full modeling procedure on training-fold rows and predict held-out-fold rows. Any data-dependent preprocessing must be learned from training data, as in a leakage-safe pipeline.
Worked Example: Assigning Unequal-Sized Groups to Three Folds
Suppose six hypothetical participants have 2, 2, 1, 3, 1, and 2 samples, respectively. Their identifiers are A through F. One possible three-fold assignment is Fold 1 = participants A and D, Fold 2 = B and E, and Fold 3 = C and F.
The row count in Fold 1 is \(2+3=5\). Fold 2 contains \(2+1=3\) rows, and Fold 3 contains \(1+2=3\) rows. The total is \(5+3+3=11\), matching the total \(2+2+1+3+1+2=11\) samples. Every participant appears in exactly one fold, so no participant has rows in both a training set and its corresponding test fold.
For the first cross-validation fit, the five rows from A and D form the test fold, and the six rows from B, C, E, and F form training data. For the second fit, B and E are held out; for the third, C and F are held out. Fold sizes are unequal, but the required identity separation holds in every fit. If outcomes differ substantially across participants, fold allocation can also seek outcome balance at the participant level; a row-level balancing procedure must not split a participant to achieve it.
Choose the Performance Summary for the Target
Grouping controls which identities cross the train-test boundary. A separate choice is how to summarize performance when participants contribute different numbers of samples. A sample-weighted metric gives greater influence to participants with more rows. A participant-weighted metric first summarizes performance within each participant and then gives participants equal weight. Neither is universally correct: the choice should match the intended use and be stated clearly.
Worked Example: Sample-Weighted and Participant-Weighted Accuracy
Suppose a hypothetical group-disjoint evaluation holds out three participants. Participant P has four samples, of which three are classified correctly. Participant Q has one sample and it is classified incorrectly. Participant R has one sample and it is classified correctly.
The sample-weighted accuracy pools all six predictions. There are \(3+0+1=4\) correct predictions, so
For participant-weighted accuracy, calculate each participant's accuracy: P has \(3/4=0.75\), Q has \(0/1=0\), and R has \(1/1=1\). Their unweighted mean is
The difference arises because P contributes four samples to the pooled calculation but only one participant-level value to the second calculation. If deployment decisions are made for participants and each participant should count equally, the second summary may be more relevant. If each sample is the unit of use and samples are the intended target, the pooled summary may be appropriate. The split and the metric answer distinct design questions.
Assumptions, Diagnostics, and Alternatives
Group-disjoint validation is most directly aligned with prediction for a new group drawn from a population comparable to the groups represented in training. It assumes that the chosen group identifier captures the dependence that must not cross the split. For example, grouping by participant does not prevent leakage across duplicate records if the same participant appears under two identifiers. Conversely, grouping too broadly can change the prediction target and reduce the effective number of independent groups.
Before interpreting results, check the split itself: for each fold, verify that the sets of training and test group identifiers have an empty intersection. Also inspect group counts, samples per group, outcome distribution by fold, and whether any group has been omitted or assigned more than once. A large discrepancy between sample-level and group-level performance can be a warning that repeated identities or other shared structure matter, though it is not proof of a particular mechanism. Very few groups can make the estimate unstable even when there are many rows; the effective supply of independent test units is limited by the number of groups.
If the prediction target is a new participant, group \(K\)-fold cross-validation is often a practical choice. Leave-one-group-out validation holds out one group at a time; it uses each group for testing but can yield highly variable fold results, especially with few groups or unequal group sizes. A sample-level split is appropriate only when the intended task genuinely concerns new samples from identities that may already be represented during model fitting. These designs do not estimate the same quantity.
Repeated measurements may also be structured by time, family, site, or cohort. The correct grouping or split may need to respect more than participant identity, depending on the intended use; group splitting by participant alone does not automatically address temporal or population differences. Validation under cohort and population shift is a related design question, but it is distinct from keeping repeated samples from one identity together. Grouping also does not cure every form of leakage: test information can still enter through preprocessing, feature selection, or model tuning if those steps are not confined to the appropriate training data.
The essential principle is simple but consequential: split at the level of independence required by the prediction claim. Grouping repeated samples prevents a model from benefiting directly from another row belonging to the same held-out participant, while careful fold construction and metric choice make explicit what population of future predictions the reported performance represents.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A study has three tissue samples from each participant and aims to predict the outcome for a new participant. What should define the cross-validation groups, and why?
- A participant has four rows. Under independent row assignment with training probability \(r=0.75\), what is the conditional probability that a test row has at least one same-participant row in training?
- Two participants have sample counts 3 and 2. If each participant is assigned to one of two folds, what fold row counts result, and is a 3-versus-2 count compatible with keeping the participants intact?
- In a group-disjoint test set, one participant has 8 correct predictions out of 10 samples and another has 0 correct out of 1. Calculate sample-weighted and participant-weighted accuracy.
- Does participant-grouped validation alone guarantee that preprocessing or feature selection has not used test-fold information? Explain.
Solutions. (1) Use participant as the group, so all three tissue samples from one person are assigned to the same fold; this evaluates prediction for people not represented in training. (2) Here \(m=4\), so the probability is \(1-(1-r)^{m-1}=1-(0.25)^3=1-0.015625=0.984375\), or about 0.9844. (3) Assigning one participant to each fold gives row counts 3 and 2. These counts are compatible with intact groups; forcing equal row counts would require splitting a participant or changing the assignment. (4) Sample-weighted accuracy is \((8+0)/(10+1)=8/11=0.7273\). Participant-weighted accuracy is \((8/10+0/1)/2=(0.8+0)/2=0.4000\). The summaries differ because the first participant contributes ten samples but each participant contributes equally to the second. (5) No. Grouping prevents the same participant from appearing in both sets, but preprocessing and feature selection must also be fitted using training data only; otherwise information from held-out samples can still influence the fitted procedure.