Why the Split Unit Matters
A multi-omics data table may have many rows but far fewer independent sources of information. A row might represent one tissue sample, one assay, one visit, or one aliquot; several rows can come from the same person. People may also belong to related families, be measured at the same site, or come from the same cohort. If related rows are assigned independently to training and test sets, the model can encounter information in training that is closely connected to a test observation. Its test performance may then say little about prediction for a genuinely new person, family, site, or cohort.
The choice of split unit is not simply a technical preference. It defines what “new” means in the evaluation. A model intended to predict outcomes for additional samples from already observed patients has a different target from one intended for new patients. A model intended for deployment at an unseen hospital has a different target again. The relevant unit is the smallest unit that must be new for the intended prediction task, while also accounting for dependence that would otherwise allow related information to cross partitions.
The earlier material on leakage pathways in multi-omics prediction motivates the practical concern: overlap can make evaluation optimistic or otherwise mismatched to deployment. Here the central task is to identify which units must remain together. This is not a claim that rows within a group are identical or that all observations between groups are statistically independent. It is a design rule that blocks specified relationships from crossing the split.
Define the Prediction Target Before Choosing a Unit
Start by stating who or what will receive predictions after model development. If deployment means predicting for a previously unseen patient, all observations from each patient must stay together, even if there are multiple visits, tissue types, or omics assays per patient. If deployment means predicting for new members of already represented families, holding out entire families may be unnecessarily restrictive; if the intended target is new families, family membership must not cross partitions. Likewise, a new-site target requires separating sites, while a target of future patients at represented sites may require separating patients but not sites.
The grouping decision must also reflect how the data were generated. When one subject contributes multiple samples, “subject” is the relevant grouping variable even if each sample has a unique sample identifier. If family members are related in a way that could convey shared signal, splitting by subject alone does not protect evaluation for new families. If cohorts and sites overlap, choosing either label alone may leave related observations on both sides.
Worked Example: Repeated Samples From Each Patient
A hypothetical study has 60 patients, each with three tissue samples. The outcome is a patient-level diagnosis, and the stated deployment target is diagnosis for a patient who was not used to develop the model. A planned test partition should contain approximately 30% of the patients.
Choose the unit. The patient is the split unit. Assigning individual samples independently would allow a patient’s samples to appear in both partitions, contrary to the new-patient target.
Construct the split. Assign 42 patients to training and 18 patients to testing. Keep all three samples from each patient together. The training partition contains \(42\times3=126\) sample rows, and the test partition contains \(18\times3=54\) rows. These are 70% and 30% of the 180 rows, respectively, because every patient contributes the same number of samples.
Check the assignment. For each patient identifier, verify that its rows have one and only one partition label. The patient identifiers in training and testing must have an empty intersection. The diagnosis may appear on all three rows, but that repetition does not make the three rows three independent patient-level outcomes.
The grouping rule matches the target: test performance estimates performance on held-out patients represented by these data, subject to the usual limits of the study population. It does not directly estimate performance at an unseen hospital, because patients from the same sites may occur in both partitions.
Formal Assignment and a Leakage Probability
Let observations be indexed by \(i\), and let \(g(i)\) denote the split-unit identifier for observation \(i\). For example, \(g(i)\) could be the patient identifier. Let \(Z_g\) be the partition assigned to unit \(g\), with values such as “training” and “test.” A valid group-based split requires
Equivalently, no split-unit identifier may occur in both partitions. This condition is directly auditable from the identifiers and partition labels; it does not depend on a modeling assumption. The rationale can be made precise for a row-wise random assignment. Suppose a subject contributes \(m\) rows, and each row is independently assigned to test with probability \(q\), otherwise to training. The subject is represented in both partitions unless all \(m\) rows go to training or all \(m\) go to test. Therefore,
For \(m=3\) and \(q=0.30\), this probability is \(1-0.70^3-0.30^3=1-0.343-0.027=0.630\). Thus, under this illustrative row-wise assignment, each three-row subject has a 63.0% chance of appearing on both sides. This calculation explains why a row-level split can fail even when every row was assigned without looking at its values. It is not a general estimate of performance bias; the amount and direction of bias depend on the data and prediction problem.
Worked Example: Prediction for New Families
A hypothetical dataset contains 12 families with two measured siblings per family, for 24 subjects total. Each subject contributes one multi-omics record. The target is prediction for a person from a family not represented in model development. A test set is planned to include 4 of the 12 families.
Choose the unit. The family, not the individual subject, is the split unit because the target requires an entirely new family. Put both siblings from each selected family in the same partition.
Calculate the partition sizes. Assign 8 families to training and 4 families to testing. Each family has two subjects, so training has \(8\times2=16\) subjects and testing has \(4\times2=8\). The split is one-third of families in testing and, because group sizes are equal, one-third of subjects in testing.
Audit the result. Check that no family identifier is present in both partitions. Then check that each subject belongs to exactly one family and one partition. A subject-level split could satisfy a no-subject-overlap check yet still place one sibling in training and the other in testing; it would fail the new-family requirement.
This design evaluates held-out families from the represented sampling setting. With only 12 families, results can vary substantially according to which four families are held out. The number of subjects alone does not make the test information equivalent to 24 independent families.
Nested and Overlapping Group Structures
Real studies often have several levels of organization: samples within subjects, subjects within families, and subjects or families observed at sites within cohorts. A useful way to select the unit is to draw these relationships explicitly and mark which level must be unseen at deployment. If the target is an unseen site, all subjects and samples from a site must stay together. If the target is new patients at sites already represented, the patient is the required split unit, although site structure may still affect how well the result generalizes.
A simple hierarchy can be handled by grouping at the required higher level. More difficult are overlapping relationships: for example, a subject may have samples in two cohorts, or a person may be recorded under multiple site labels. If the rule is that no connected observations may cross partitions, construct connected components of the relationship graph. Each observation is a node, and an edge links observations that must stay together. Assign each connected component wholly to one partition. This can create large groups, but splitting a component to achieve a preferred percentage would violate the stated no-crossing rule.
Worked Example: Evaluating at a New Clinical Site
A hypothetical multi-omics dataset has 200 patients from four hospitals, with 80, 60, 40, and 20 patients at the four respective sites. The intended use is at a hospital that contributed no training data. One hospital with 20 patients is designated as the test site; the other three sites provide 180 training patients.
Choose the unit. The site is the split unit because the deployment target is an unseen hospital. All patient records and all assay rows from the designated hospital remain in the test partition. A patient-level split that mixed patients from each hospital across training and test would not answer this question.
State what the result represents. The test result describes performance on 20 patients from one held-out site, not an average over a broad population of hospitals. The test site is one group, so a large number of patient rows does not replace having multiple independently held-out sites. The estimate may be sensitive to that site's case mix, protocols, and measurement conditions.
Audit the partition. Verify that the site identifiers in training and testing do not overlap, then verify that no patient or sample identifier appears on both sides. If a patient appears in records assigned to more than one site, the study must specify whether those records should be linked and kept together; otherwise the claimed site separation may not be meaningful.
Diagnostics, Alternatives, and Limitations
Before fitting a model, perform a split audit using identifiers and study metadata, not just a row count. At minimum, count unique split units in each partition, check the intersection of unit identifiers, and confirm that every observation maps to exactly one partition. Also inspect repeated sample identifiers, patient-to-family mappings, site and cohort labels, and any cross-classification that could leave linked observations separated. If group sizes vary, report both the number of groups and the number of rows in each partition.
The main alternative is to assign individual rows without preserving groups. That can be appropriate only when the intended prediction target and data-generating structure make rows the relevant independent units. It is not a safe default simply because the software offers it. Group-based splitting is the direct alternative when observations share a unit that must be held out together. When the target is a new cohort or site, a group split at that higher level is needed; the later material on random splits versus cohort-aware splits treats that distinction more fully.
Group assignment can make partition sizes uneven, especially when groups differ greatly in size or there are few groups. Do not break a group merely to obtain an exact row percentage. With very few sites or cohorts, a single held-out group can give a noisy and narrow view of generalization. The choice of which groups are held out also matters. If there is no plausible way to create a test partition with enough independent groups for the stated target, report that limitation rather than presenting many rows as evidence of many independent test units.
- Wrong reasoning: “There are hundreds of test rows, so the evaluation is based on hundreds of independent cases.” Correction: Count the relevant groups as well as rows; repeated observations from one subject or site do not create additional independent units for a new-subject or new-site target.
- Wrong reasoning: “Every sample has a unique sample ID, so no identifier crosses the split.” Correction: Check the parent subject, family, site, and cohort identifiers that define the intended prediction target.
- Wrong reasoning: “A subject-level split is always sufficient.” Correction: It is insufficient when the target is a new family, site, or cohort, or when linked observations span multiple subject identifiers.
Check Your Understanding
Answer each question, then compare with the solutions.
- A patient contributes four tissue samples, and the intended target is a new patient. What should be the split unit, and what identifier check is essential?
- There are 30 families with three siblings each. The target is a person from a family not seen during model development. If 10 families are assigned to testing, how many subjects are in testing?
- For a subject with three rows, each independently assigned to test with probability \(q=0.20\), calculate the probability the subject appears in both training and test.
- A study has 500 patients from two hospitals and aims to evaluate use at an unseen hospital. Why is a patient-level split not sufficient for that target?
- Two cohort labels overlap because some subjects have samples in both cohorts. What should be checked before assigning whole cohorts to partitions?
Solutions. (1) Use the patient as the split unit; verify that each patient identifier occurs in only one partition and that training and test patient-ID sets do not overlap. (2) Testing contains \(10\times3=30\) subjects, grouped as 10 families; the independent held-out units for the stated target are the 10 families. (3) The probability is \(1-(1-0.20)^3-(0.20)^3=1-0.512-0.008=0.480\), or 48.0%. (4) A patient-level split can put patients from both hospitals in training and test, so it evaluates new patients under represented-site conditions rather than an unseen hospital. (5) Check the subject-to-cohort links. If subjects link observations across cohorts and must not cross partitions, treat linked observations as connected and assign the resulting groups together, or clarify the target and revise the split design.