Why the Split Unit Matters
A validation result is useful only if the held-out data resemble the cases on which a predictor will be used. In genomic studies, a row often represents an individual, but individuals may share a family, recruitment site, assay batch, or calendar period. Splitting rows at random can place members of the same group on both sides. That may be reasonable for some uses and misleading for others.
The key decision is therefore not simply how many folds to make. It is which units should be kept together when separating training data from validation data. The answer follows from the deployment question: will the model predict for new individuals from populations already represented in training, new families, new sites, or a future time period?
The prediction target, defined in the earlier material on genomic prediction targets, must be fixed first. For example, “predict disease status for a new person at an existing hospital” and “predict disease status for a person at a new hospital” are different targets. They imply different validation designs even if the outcome and genomic measurements are identical.
A Risk-Based Justification
Let \(Q\) denote the distribution of cases encountered at deployment, and let \(L(Y,\hat{Y})\) be the chosen prediction loss for outcome \(Y\) and prediction \(\hat{Y}\). For a fitted predictor \(f\), its deployment risk is the expected loss for a new deployment case:
Here \(X\) contains the predictors, including genomic features. The distribution \(Q\) is not just a distribution of rows: it includes how new people, families, sites, and times are represented. If deployment cases are sampled from new sites, for instance, validation cases from sites already represented in training may not have the same distribution of site-specific conditions.
The basic justification is direct. Suppose a predictor \(f\) is fixed using training data, and a held-out unit \(U\) is sampled from the same deployment distribution \(Q\), independently of the training data. Then the expected loss on that unit is
The equality holds because, conditional on the fitted predictor, the held-out unit has the deployment distribution. If validation instead holds out individuals while allowing their relatives or site-mates into training, that equality need not describe risk for a new family or a new site. The design has answered a different question.
This argument is about matching the population and dependence structure, not a claim that a particular split gives an exact estimate of a final model's risk. Training-set size, sampling variability, and model fitting can also affect the reported performance. The unit decision comes first because even a precisely computed result for the wrong deployment distribution is not the desired answer.
Choosing Among Individuals, Groups, and Time
| Intended use | Natural validation unit | What should be absent from training? |
|---|---|---|
| New individuals from the same represented cohort and settings | Individual | The held-out individuals themselves |
| Individuals from families not represented in training | Family | All members of each held-out family |
| Individuals at sites not represented in training | Site | All observations from each held-out site |
| Predictions for a later period | Time block or period | Observations from the future period; training must not use future information |
These are not rigid rules independent of the target. A model intended for new individuals in the same clinics can reasonably be evaluated with individual-level holdouts if the training data may include other patients from those clinics. A model intended to transfer to clinics never seen during training requires site-level holdouts. The same distinction applies to relatives: prediction for another member of a represented family differs from prediction for a member of a new family.
When multiple grouping structures matter, the intended use may require a design that holds out more than one kind of unit. For example, if deployment is to new families at new sites, a design should not let either those families or those sites be represented in training. Such a design can leave fewer independent units for evaluation, so the target and available data must be considered together.
Worked Example: New Individuals Within Existing Clinics
A hypothetical genomic cohort contains 600 individuals from 6 clinics, with 100 individuals at each clinic. The intended use is to predict outcomes for additional patients at these same clinics. Assume patients are sampled independently within each clinic for the prediction task, and the deployed workflow will retain access to data from all 6 clinics.
A random individual-level holdout of 20% gives \(600(0.20)=120\) validation individuals, about \(120/6=20\) from each clinic. The remaining \(600-120=480\) individuals are available for training. All 6 clinics can occur in both sets, which reflects the intended use: a new patient comes from a clinic already represented in training.
If instead the evaluation holds out one clinic, the validation set contains 100 patients, but that clinic is absent from training. This evaluates transfer to an unrepresented clinic, not the stated task. It may be a valuable additional analysis, but its result answers a different question.
The individual split is appropriate here because the deployment unit is a new individual, and existing clinic representation is part of the intended information available to the model. This conclusion would change if clinic-specific patterns were not available at deployment or if the model were intended for new clinics.
Worked Example: Prediction for Members of New Families
Consider a hypothetical cohort of 240 individuals from 80 families, with 3 siblings per family. The intended use is prediction for individuals whose families contributed no samples to model training. If 16 families are held out, the validation set has \(16(3)=48\) individuals, and the other \(80-16=64\) families contribute \(64(3)=192\) individuals to training.
A row-wise 20% holdout would also produce \(240(0.20)=48\) validation individuals. But if each sibling were assigned independently with probability 0.20, the probability that a particular family has at least one validation member and at least one training member is
Thus, under that random assignment, 48% of families are expected to be split across training and validation. With 80 families, that is \(80(0.480)=38.4\) families on average. Validation individuals from those families have relatives in training, so this design does not evaluate the stated new-family task.
Holding out whole families aligns the design with deployment. The calculation does not say that any family overlap always invalidates prediction: it shows why overlap conflicts with this particular target. If the intended task instead were prediction for a new sibling in a family already represented in the training data, family-level holdout would be a more difficult and mismatched evaluation.
Worked Example: Prediction in a Future Time Period
A hypothetical consortium has genomic and clinical records from 2018 through 2023, with 500 eligible participants per year. It plans to deploy a predictor beginning in 2024, using only data available by the end of 2023. A simple evaluation target is prediction in a later period after training on earlier data.
Suppose the evaluation holds out 2023 and trains on 2018–2022. The training set has \(5(500)=2{,}500\) records and the validation set has 500 records. The ordering matches the use: every training year precedes the held-out year.
A random 20% individual split would put \(6(500)(0.20)=600\) validation records across all six years, including some from 2018–2023, while training would also contain observations from those same periods. It would evaluate prediction for randomly withheld records from the observed era, not performance in a later period. If outcome definitions, treatment, assays, or patient mix change over time, the distinction can be consequential.
A temporal holdout is appropriate because the deployment target is future time. It does not guarantee that future conditions will resemble 2023; substantial changes after the holdout period remain a limitation. The example uses one held-out year to make the arithmetic clear, not to prescribe a universally adequate number of years.
Diagnostics, Failure Modes, and Alternatives
Before interpreting validation performance, check whether the split actually separated the units it was intended to separate. For group-based designs, tabulate the number of families or sites represented in each partition and check for overlap. For temporal designs, check the date ranges and confirm that no training record occurs after a validation record when the target is future prediction. Also verify that the held-out units resemble the intended deployment population in relevant ways, such as ancestry composition, assay platform, eligibility criteria, or outcome prevalence.
Warning signs include very high performance under individual splitting that falls substantially under family- or site-level splitting; validation records sharing a family or site with training despite a “new group” target; and a random split used to claim future-time performance. These patterns do not by themselves establish the cause of a performance difference, but they signal that validation designs may be measuring different tasks.
The main alternatives are not competing ways to estimate one universal quantity. They correspond to different questions:
- Individual-level splits use more granular assignment and can suit new individuals drawn from represented settings. They may be unsuitable for claims about unseen families or sites.
- Group-level splits keep a chosen group intact and directly assess transfer to groups absent from training. They often have fewer independent units and can yield less precise or more variable results.
- Time-ordered splits respect the direction of deployment into the future. They can expose temporal drift, but results from one period may not represent later periods.
No split unit removes every limitation. A few large sites may provide weak evidence about transfer to a broad range of future sites. Family-level evaluation can be difficult if there are few families or family sizes vary widely. A time holdout can confound calendar change with other shifts in the cohort. The chosen design should be stated alongside the deployment claim, including which units appear in both training and validation.
Common Mistake: Treating Random Rows as Independent Deployment Cases
Wrong reasoning: “The data table has one row per person, so randomly splitting rows must provide an unbiased test of prediction for new people.”
Correction: A row identifies the observation, but it does not determine the deployment unit or dependence structure. If deployment is to members of new families, related individuals must be kept together. If deployment is to new sites, all observations from a held-out site must be absent from training. If the target is a future period, training must precede validation. Individual-level random splitting is justified only when its held-out individuals represent the intended prediction setting.
A related mistake is to choose the split that produces the most favorable score. The evaluation design should follow the intended use, not the desired result. When several deployment settings matter, report separate evaluations and name the distinct questions each one addresses.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A study plans to use a genomic predictor for new patients at clinics that were represented in training. Should validation necessarily hold out whole clinics? Explain.
- A cohort has 50 families of four siblings each. Under independent row assignment to training with probability 0.75 and validation with probability 0.25, what is the probability that a family is split across the two sets?
- A model will be applied to people at hospitals not represented during training. What is the appropriate validation unit, and what overlap should be checked?
- Why does a random split across records from 2019–2024 fail to directly evaluate deployment in 2025?
Solutions. (1) No. Individual-level holdout can match prediction for new patients at already represented clinics, if that is the deployment setting. Holding out clinics would instead test transfer to new clinics. (2) A family is not split only if all four siblings go to training or all four go to validation. Therefore the probability is \(1-(0.75)^4-(0.25)^4=1-0.31640625-0.00390625=0.6796875\), or about 0.680. (3) Hold out whole hospitals; check that no patient record or other observation from a held-out hospital appears in training. (4) A random split can place records from 2024 in training while testing on earlier records, and both sets may cover the same periods. It therefore does not reproduce training only on past information followed by prediction in 2025.