Diagnosing Split-Strategy Sensitivity

Why the Split Strategy Is Part of the Prediction Question

A classifier can perform well when test subjects resemble training subjects and less well when they come from a different site or cohort. That difference is not merely a technical inconvenience: it changes what the evaluation estimates. A random subject split asks how well a model predicts additional subjects drawn from a mixture like the observed data, where each site or cohort may contribute to both training and testing. A grouped split asks about prediction when a study group is absent from training. Those are distinct questions, even if the outcome and classifier are identical.

For a multi-omics task, study structure may be reflected in assay batches, recruitment practices, clinical care, or population composition. A model can use features that predict the outcome within the observed collection but do not carry over in the same way to a new site or cohort. Comparing split strategies helps reveal this sensitivity, but a score gap alone does not identify its cause.

A site and a cohort are not automatically the same unit. One cohort may be collected across several sites, and a site may contribute to more than one cohort. The grouping unit must therefore be specified from the study design. If the intended deployment is to a new cohort, holding out sites within known cohorts is not a substitute for holding out cohorts.

Define a Comparable Evaluation Before Comparing Scores

Let \(M\) denote the chosen performance metric, with larger values representing better performance, and let \(\hat{M}_{r}\), \(\hat{M}_{s}\), and \(\hat{M}_{c}\) denote the estimated scores under random, site-based, and cohort-held-out evaluation, respectively. These quantities are useful to compare only when the task and metric are aligned. For example, the outcome definition, eligible subjects, prediction horizon, and metric should be the same. Any threshold-dependent metric must use a threshold chosen without the test outcomes.

The model-building process must also be valid within each split. Any data-dependent transformation, feature filtering, imputation, scaling, feature selection, or tuning must be learned using training data only, then applied to the test data. Otherwise the comparison can be distorted by information crossing the split. The earlier material on leakage audits and nested multi-omics evaluation provides the relevant safeguards; here the focus is interpreting the pattern across valid splits.

In repeated random splitting, a reported random-split score may be an average over repetitions. For site-based or cohort-held-out evaluation, one may report each held-out-group score as well as an aggregate. The aggregate needs a defined weighting rule: weighting groups equally describes an average group, while weighting subjects equally gives greater influence to larger groups. Neither is universally correct; the choice should match the intended prediction target.

What Score Patterns Can and Cannot Diagnose

A useful interpretation treats each split strategy as a probe of a different dependence. If random performance is high but site-based performance falls, the model may rely on relationships that do not persist across sites, or the held-out sites may differ in outcome prevalence, measurement, or clinical composition. If site-based performance is acceptable but cohort-held-out performance is lower, the cohort grouping may introduce additional differences that site withholding did not capture. If all three are similar, that is reassuring for these sampled groups, but it is not proof that performance will remain stable in every future population.

A short justification clarifies why grouped scores can differ even without a coding error. Let \(G\) index study groups and let \(M_g\) be the score that would be obtained when testing in group \(g\). An equally weighted group average is

where \(K\) is the number of evaluated groups. A subject-weighted average is instead

where \(n_g\) is the number of test subjects in group \(g\). These averages are generally unequal when group sizes and group scores vary. A random split may also mix groups in each test set, so its score need not equal either grouped average. Thus, a score difference can reflect a change in the evaluation target and weighting as well as a change in the model's ability to transfer.

Diagnostics, Uncertainty, and Alternatives

Before interpreting a gap, check that it is not an artifact of an inconsistent evaluation. Confirm group identifiers and split assignments; verify that no subject, repeated sample, or near-duplicate record crosses a boundary when that would violate the intended split; and confirm that preprocessing and tuning are confined to training data. Inspect the number of subjects and outcome cases in each test group. A small held-out group, or very few cases in one outcome class, can produce an unstable metric. For AUC, a test set needs observations from both outcome classes for the usual empirical AUC to be defined.

Also inspect group-level outcome prevalence, feature distributions, missingness, and assay or processing differences. These checks do not prove a particular mechanism, but they help describe how the test groups differ. If a metric changes markedly across groups, report the individual group scores rather than letting one pooled number hide the variation. For threshold-dependent scores, check that the threshold was selected using training data and that class imbalance is handled consistently.

Uncertainty should be reported in a way that respects the split structure. Resampling individual subjects as if they were independent may understate uncertainty when subjects within a site or cohort share features of recruitment or measurement. With few groups, uncertainty about performance in a new group is especially difficult to estimate precisely. A narrow subject-level interval cannot compensate for having observed only a small number of independent cohorts.

The main alternatives answer different questions. Random splitting is appropriate when future subjects are expected to resemble the same mixture of represented study groups and group overlap is part of the intended setting. Site-based evaluation assesses transfer to a site not used for fitting, while cohort-held-out evaluation targets transfer to a new cohort. Leave-one-cohort-out evaluation displays performance across the cohorts available; a single prespecified held-out cohort can more closely mimic one external evaluation but uses less data for model development. No split strategy is intrinsically best: choose based on the intended deployment, and report the design and group-level results transparently.