Random Splits Versus Cohort-Aware Splits

Why the Split Defines the Question

A prediction score is not a property of a model alone. It depends on the population to which predictions will be applied, the information available during training, and the way evaluation data are separated from training data. In multi-omics studies, subjects may be recruited at different hospitals, measured in different cohorts, or processed under different laboratory conditions. A random subject-level split can put subjects from every site in both training and test sets. A site- or cohort-based split instead withholds one or more entire groups. These procedures answer different deployment questions.

The central choice is therefore not “which split is stricter?” It is “what kind of future observations should this performance estimate represent?” If a model will be used for additional subjects at sites already represented in its training data, a subject-level split may be relevant. If it will be transferred to a site or cohort not represented during training, an entire-site or entire-cohort holdout is more aligned with that use.

Definition: Let \(g\) index sites or cohorts and \(i\) index subjects within group \(g\). Let \(X_{gi}\) denote the available predictors, \(Y_{gi}\) the prediction target, and \(f_D\) a prediction rule fitted using training data \(D\). For a loss function \(L\), prediction risk is the expected loss \(E[L(Y,f_D(X))]\) under a specified deployment population. A split design determines which population the test observations approximate, as well as which data are available to fit \(f_D\).

Two Split Designs, Two Estimands

In a random subject-level split, subjects are assigned to training and test sets individually, while preserving any required grouping of repeated or otherwise dependent observations. If all sites contribute subjects to both sets, the test set mainly represents new subjects drawn from the same mixture of represented sites and conditions. This is useful when deployment resembles that mixture and the training process will encounter similar site environments.

In a site-aware or cohort-aware split, every subject from a held-out site or cohort is kept out of training. The fitted model is then evaluated on a group-level environment not used for fitting. A site-held-out design targets transfer to a new site drawn from a relevant population of sites; a cohort-held-out design targets transfer to a new cohort. The groups need not be identical concepts: a cohort may span sites, and a site may contribute to several cohorts. The holdout unit should match the source of novelty in the intended deployment.

These interpretations require assumptions. For a random subject split, subjects in the test set should represent future subjects conditional on the sites and data-generation conditions that will be encountered. For a site- or cohort-held-out estimate to represent future groups, held-out groups must be relevant to the target deployment population. If all groups are highly atypical or if the target includes conditions absent from every observed group, neither split guarantees useful transportability. In either design, the outcome and predictors must be measured consistently enough for the chosen loss to be meaningful.

R_{\mathrm{subject}} \approx \sum_{g=1}^{G} w_g E\{L(Y,f_D(X))\mid g\}, \qquad R_{\mathrm{new\ group}} \approx E_{G^\ast}\left[E\{L(Y,f_D(X))\mid G^\ast\}\right].

Here \(w_g\) is the share of subjects from group \(g\) in the target mixture, and \(G^\ast\) denotes a new group drawn from the target population of groups. The first risk averages subject-level losses across represented groups, generally weighting larger groups more heavily. The second averages over groups as deployment units. It is not automatically the same as the risk for a randomly selected subject across all future sites: that target may weight large and small sites differently. The weighting scheme must be stated.

A Short Justification for the Random-Split Target

Suppose there are \(n_g\) subjects in site \(g\), and a fraction \(q\) of each site is assigned to test at random. If the split is independent of subject outcomes and predictors within each site, the test subjects from site \(g\) are, in expectation, a sample of that site's subjects. The expected pooled test loss therefore approaches a subject-weighted average of site-specific losses. For a fixed fitted model and a test set of \(m\) subjects, the pooled mean loss is:

\widehat R_{\mathrm{pooled}}=\frac{1}{m}\sum_{(g,i)\in \mathrm{test}}L(Y_{gi},f_D(X_{gi})).

The denominator is the number of test subjects, so a site contributing more test subjects has more influence. This calculation does not show that a random split estimates performance at a new site: the test subjects still come from sites represented in training. Nor does it remove dependence created by common site conditions. The split defines the test population, while the dependence structure affects how uncertain an estimate is.

Worked Example: Pooled Error After a Random Subject Split

Hypothetical data. A two-site study evaluates a model with absolute prediction error measured in outcome units. A random subject-level split places subjects from both sites in training and test sets. The test set contains 20 subjects from Site A with a total absolute error of 40 units, and 10 subjects from Site B with a total absolute error of 30 units. Assume the loss is defined for every test subject, and that this split reflects the intended mix of already represented sites.

The pooled mean absolute error is the sum of individual absolute errors divided by the number of test subjects:

\widehat R_{\mathrm{pooled}} =\frac{40+30}{20+10} =\frac{70}{30} =2.333\text{ outcome units}.

This estimates average error for a subject drawn from the test-set mixture: two-thirds of test subjects came from Site A and one-third from Site B. The site-specific mean errors are \(40/20=2.000\) units at Site A and \(30/10=3.000\) units at Site B. Their unweighted average is \(2.500\) units, not \(2.333\), because it gives both sites equal weight instead of weighting by test-subject counts. The pooled result is informative for the represented subject mixture, but it does not measure transfer to a site absent from training.

What Cohort-Aware Evaluation Adds

A group-held-out evaluation makes the group, not the individual subject, the unit of generalization. One common design repeatedly trains on all but one site and tests on the held-out site. Each site then receives a test prediction only from a model that did not train on that site. To summarize results, report site-specific performance and state how sites are combined. A pooled subject-level score weights sites by their test sample sizes; an unweighted mean of site scores treats each site equally. Neither is universally correct.

If the deployment target is a randomly selected future site, an equal-site average can be a natural summary when observed sites are representative of that future-site population. If the target is a randomly selected future subject from a known site-size mixture, a subject-weighted score may be more relevant. With few sites, however, neither average is a precise characterization of a broad population of possible sites. Many subjects at one site do not substitute for observing many independent sites.

Worked Example: Averaging Performance Across Held-Out Sites

Hypothetical data. A study has three sites. In a leave-one-site-out evaluation, the model is trained without the held-out site each time. Absolute prediction errors are summarized as follows: Site A has 20 test subjects and mean error 2.0 units; Site B has 10 subjects and mean error 3.0 units; Site C has 30 subjects and mean error 4.0 units. Assume each site is a valid held-out group, the outcome scale is common across sites, and the target deployment choice is a site selected with equal probability.

For an equal-site target, first average within each site, then give each of the three sites equal weight:

\widehat R_{\mathrm{equal\ site}} =\frac{2.0+3.0+4.0}{3} =\frac{9.0}{3} =3.000\text{ outcome units}.

For a target that selects a subject from the pooled held-out subjects, calculate the weighted average. The total absolute errors are \(20(2.0)=40\), \(10(3.0)=30\), and \(30(4.0)=120\) units, for 190 units over 60 subjects:

\widehat R_{\mathrm{subject\ weighted}} =\frac{40+30+120}{20+10+30} =\frac{190}{60} =3.167\text{ outcome units}.

The two values differ because Site C is the largest and has the highest error. The equal-site estimate answers a site-weighted question; the pooled estimate answers a subject-weighted question for this observed collection. The assumption that the three sites represent future sites is not established by the arithmetic and must be justified by study design and context.

Choosing a Split for the Deployment Scenario

Describe where predictions will be used.
Specify whether future subjects will come from sites already represented in training, from new sites, from later cohorts, or from a defined mixture of these settings.

Choose the holdout unit.
Use a subject-level split for new subjects in represented settings. Hold out whole sites or cohorts when deployment requires transfer to an unseen group. Keep any dependent observations together as required by the independent-unit design.

Declare the averaging target.
State whether performance is averaged over subjects or groups. Report group-specific results when possible so that a single aggregate does not hide substantial variation.

Check whether the observed groups support the claim.
Inspect group sizes, outcome distributions, predictor coverage, and measurement conditions. A held-out-site score cannot establish broad new-site performance if only one unusual site was held out.

Worked Example: Matching a Split to a Planned Rollout

Hypothetical scenario. A laboratory has data from four hospitals and plans to deploy a multi-omics predictor at a fifth hospital that has not contributed training subjects. A random 80/20 subject split leaves subjects from all four original hospitals in both sets. A site-held-out design instead trains on three hospitals and evaluates on the fourth, repeating this for each hospital.

The random split estimates performance for new subjects from the observed hospital mixture, because every hospital environment appears in training. That does not match the planned fifth-hospital rollout. The leave-one-hospital-out design is more relevant because each test hospital is absent from the corresponding training set. Its four site-specific errors should be shown, alongside a clearly defined summary. If the target is an average hospital, calculate the mean of the four hospital-specific errors; if the target is an average patient from a specified hospital-size mixture, use the corresponding patient weights.

This choice still relies on the assumption that the four hospitals provide useful information about the fifth. If the fifth hospital uses a different assay or serves a population outside the range of these sites, the evaluation cannot validate performance under that new condition. The design is better aligned with the deployment question, but it cannot manufacture evidence about environments not represented in the study.

Diagnostics, Alternatives, and Limitations

Before interpreting either estimate, compare the training and test sets on site membership, cohort membership, sample sizes, outcome prevalence or distribution, and measured predictor ranges. For a random split, verify that all test subjects belong to the intended represented-site mixture and that no dependent observations have been separated improperly. For a group-held-out split, inspect how much the held-out groups differ from the training groups. Large differences may be evidence of the very transport challenge of interest, not a reason to discard the result.

Several warning signs deserve attention:

Site dominance: a pooled score may mostly describe the largest site. Show site-specific scores and clarify the weighting.
Few independent groups: many subjects but only a few sites provide limited evidence about performance across a population of sites.
Extreme group heterogeneity: a single average may conceal sites where performance is materially worse or better.
Unrepresentative holdout: a held-out cohort selected for convenience may not represent future cohorts.
Different target populations: comparing a random-split estimate with a site-held-out estimate as if they were competing estimates of the same quantity can be misleading.

A random subject split is often more statistically stable when there are few groups, but it answers a narrower question about represented conditions. A site-aware split is better aligned with new-group deployment, but its estimate can be variable and sensitive to which sites are available. A temporal cohort split may be preferable when the intended use is prediction in a later period, even if the same sites recur. If deployment mixes familiar and new sites, report separate results for those scenarios or define the mixture and its weights explicitly. These designs are alternatives for distinct target questions, not a universal ranking from “easy” to “correct.”

The estimated error also depends on the training sample size. A model trained on three sites may not perform like the final model trained on all four; the held-out-group result evaluates the training procedure at the size and composition used in that split. Further choices about model selection and data processing must be handled without using test information, but the split comparison itself does not specify those procedures.

Common Mistake

Common pitfall: “The random split has a low test error, so the model will work well at a new hospital.” This conclusion does not follow if the test subjects came from hospitals represented in training. The random split supports a claim about new subjects under the represented hospital mixture, subject to the split assumptions. For a new-hospital claim, evaluate with whole hospitals held out and state how those hospital-specific results are summarized. Even then, generalization to future hospitals requires that the observed hospitals are relevant to the target.

Key takeaway: A split is part of the estimand. Random subject-level splits typically assess performance for new subjects in represented settings; site- or cohort-aware splits assess performance when a group is withheld. Match the split and the weighting rule to the deployment population, and report group-level variation rather than allowing one aggregate score to imply more than it supports.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A random split puts subjects from every hospital in both training and test sets. What deployment situation does its test score most directly represent?
Two held-out sites have mean absolute errors of 2.5 units and 5.5 units. What is their equal-site average, and what target does it describe?
One site contributes 90% of the test subjects and has lower error than the other sites. Why might the pooled score be an incomplete summary?
A model will be applied to later cohorts at the same hospitals. Which split design may better reflect this deployment, and what important limitation remains?

Solutions. (1) It most directly represents new subjects from the mixture of hospitals and conditions represented in training and test, not a wholly new hospital. (2) The equal-site average is \((2.5+5.5)/2=4.0\) outcome units; it gives each of the two sites equal weight and describes an average site-level error for those held-out sites. (3) The pooled score is dominated by the largest site, so it may obscure poorer performance elsewhere; show site-specific scores and consider a target-appropriate weighting rule. (4) A later-cohort holdout is aligned with prediction in a later period. Its result still depends on whether the observed later cohort is representative of future cohorts, and it provides limited evidence when only a small number of cohorts are available.