Why Hold Out Whole Cohorts?
When a prediction model is intended for use with data from a new study, hospital, or collection site, randomly splitting individuals can give an overly familiar test set. Individuals in the same cohort may share measurement procedures, recruitment patterns, or other characteristics. A random subject-level split can place members of one cohort in both training and test sets, allowing the fitted pipeline to benefit from cohort-specific information that may not be available in a new cohort.
Leave-one-cohort-out (LOCO) evaluation addresses this design question by holding out one entire cohort at a time. The training set for each fit contains all other cohorts. The held-out cohort is used only to assess that fit. The introductory material on Summarizing Performance Across Cohorts covers the need to report cohort-specific results; here the focus is on constructing and interpreting the LOCO evaluation itself.
A prediction pipeline includes every data-dependent operation needed to produce predictions: preprocessing, imputation, feature selection, model fitting, and any tuning or threshold selection. A decision made from data must be made using the training cohorts for that split, not the held-out cohort. If tuning is needed, it must be carried out within the training data, for example by an inner resampling procedure; the held-out cohort remains untouched until final evaluation. This is the same fold-local principle used in other resampling designs, applied here at the cohort level.
Assumptions and the LOCO Procedure
LOCO is an evaluation design, not a guarantee that the resulting estimate applies to every future setting. Its interpretation depends on the data and intended use. At a minimum, cohort membership must be known and correctly recorded, each subject must belong to only one cohort, and outcomes and predictors must be defined comparably enough that the chosen metric has the same meaning across cohorts. Subjects used for fitting should not also appear in a held-out cohort. The pipeline must also be implementable using information available when predicting for a new cohort.
The cohorts do not have to be equal in size. However, a very small cohort may produce a noisy cohort-specific metric, and omitting a large cohort can leave relatively little data for training. If the aim is prediction in a future cohort, it is relevant whether the observed cohorts resemble the settings of intended use. LOCO directly measures performance on the observed held-out cohorts; extending that evidence beyond them requires care.
Specify the cohort boundaries, prediction task, outcome definition, and performance metric before examining held-out results.
For each \(k\), use every cohort except \(\mathcal{D}_k\) for training and reserve all of \(\mathcal{D}_k\) for testing.
Within the training cohorts, estimate preprocessing steps, select features, tune model settings if needed, and fit the final model for that split.
Apply the fitted training pipeline to the held-out predictors, obtain predictions, and calculate the prespecified metric using that cohort’s outcomes.
Report cohort-specific results and choose an overall summary whose weighting matches the prediction target.
Worked Example: Constructing Three LOCO Fits
Suppose a hypothetical multi-omics classifier is evaluated on three independent cohorts with 40, 30, and 50 subjects. The complete pipeline is specified in advance, and its preprocessing and model fitting are repeated separately for each split. The training-set sizes are therefore \(30+50=80\), \(40+50=90\), and \(40+30=70\) subjects.
For the first fit, cohort 1 is held out: train on cohorts 2 and 3, then predict the 40 subjects in cohort 1. For the second, train on cohorts 1 and 3 and predict cohort 2. For the third, train on cohorts 1 and 2 and predict cohort 3. No subject from the held-out cohort contributes to estimating that split’s preprocessing parameters or model.
Assume the resulting held-out predictions classify 34 of 40 subjects correctly in cohort 1, 21 of 30 in cohort 2, and 42 of 50 in cohort 3. The cohort accuracies are
The equal-weight mean of the three cohort accuracies is \((0.850+0.700+0.840)/3=0.7967\), or about 79.7%. This summarizes the average of the three observed cohort-level accuracies; it is not the fraction correct among all 120 subjects. That pooled fraction is \((34+21+42)/(40+30+50)=97/120=0.8083\), or about 80.8%. Both calculations are correct, but they answer different questions.
Choosing and Interpreting the Summary
Let \(M_k\) denote a performance metric calculated from predictions in held-out cohort \(k\). An equal-weight cohort average is
Each observed cohort contributes equally, regardless of size. This is often useful when the target is performance in a typical cohort among those represented. It can also make poor performance in a small cohort visible rather than allowing a large cohort to dominate the summary. It does not estimate performance in an abstract “typical” future cohort unless the observed cohorts are relevant to that target.
For accuracy, the pooled fraction correct weights each cohort by its size. If \(C_k\) is the number correctly classified in cohort \(k\), then \(M_k=C_k/n_k\), so
This identity follows by substituting \(C_k=n_kM_k\) into the pooled fraction. It shows that pooled accuracy is a size-weighted mean of cohort accuracies, not their equal-weight mean. For other metrics, such as sensitivity or area under the receiver operating characteristic curve, pooling predictions and outcomes may not equal a simple weighted average of cohort-specific values. State how any overall metric was calculated.
Worked Example: Equal-Weight and Pooled Sensitivity
Consider three hypothetical held-out cohorts with the following confusion counts. A true positive (TP) is a positive outcome predicted positive; a false negative (FN) is a positive outcome predicted negative; a true negative (TN) is a negative outcome predicted negative; and a false positive (FP) is a negative outcome predicted positive.
| Cohort | TP | FN | TN | FP |
|---|---|---|---|---|
| 1 | 8 | 2 | 8 | 2 |
| 2 | 3 | 1 | 4 | 2 |
| 3 | 1 | 4 | 14 | 1 |
Sensitivity is the proportion of positive-outcome subjects classified positive, calculated as \(\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})\). The three cohort sensitivities are \(8/(8+2)=0.800\), \(3/(3+1)=0.750\), and \(1/(1+4)=0.200\). Their equal-weight mean is \((0.800+0.750+0.200)/3=0.5833\).
If instead sensitivity is calculated after pooling the held-out predictions, there are \(8+3+1=12\) true positives and \(2+1+4=7\) false negatives. Thus pooled sensitivity is \(12/(12+7)=12/19=0.6316\). The difference arises because the equal-weight summary gives each cohort one-third of the weight, while pooled sensitivity weights cohorts according to their numbers of positive outcomes. The intended target should determine which summary is emphasized; reporting the cohort-specific values makes that choice transparent.
Diagnostics, Alternatives, and Limitations
Check the evaluation implementation before interpreting any summary. For each split, verify that the held-out cohort contributed no information to training, tuning, threshold selection, or data-dependent preprocessing. Confirm that all held-out subjects received predictions, that outcome labels and metric definitions are consistent, and that the reported cohort sizes agree with the prediction files. A table of the number of subjects, outcome counts, and metric by held-out cohort can reveal missing predictions or cohorts with too few positive or negative outcomes for a stable classification metric.
A particularly serious failure is to combine cohorts first, estimate a transformation using all subjects, and then perform LOCO on the transformed data. Even if the outcome labels were not used, the held-out predictors have influenced a data-dependent step. Instead, estimate that step using the training cohorts for each split and apply the resulting transformation to the held-out cohort. The same rule applies to feature selection and to choosing a classification threshold.
LOCO differs from random subject-level cross-validation because its test units are whole cohorts, not randomly selected subjects. Subject-level splits can be appropriate when the intended task is prediction for new subjects drawn from settings already represented in training, but they do not directly assess leaving an entire cohort unseen. A single external test cohort offers a direct assessment for that cohort, but provides no collection of held-out-cohort results and can be difficult to obtain. LOCO uses all observed cohorts for testing in turn, but the fits have overlapping training sets and do not constitute independent studies of performance.
LOCO also has limits. If there are very few cohorts, the cohort average is based on very few distinct test settings. If one cohort is unusually large or has many more training subjects than others, the training-set size and fitted model can vary substantially across splits. If a training split lacks one outcome class, some classifiers cannot be fitted as intended. These are design constraints to identify and report, not problems that can be fixed by silently changing the held-out cohort.
Worked Example: Keeping Threshold Selection Out of the Test Cohort
In a hypothetical LOCO split, cohort 3 is held out. Using only cohorts 1 and 2, the analyst fits the model and chooses a probability threshold of 0.60 through the prespecified training procedure. The threshold is then fixed before examining cohort 3’s outcomes. The held-out cohort has three positive and three negative outcomes, with predicted probabilities as follows: positive-outcome subjects receive scores 0.90, 0.70, and 0.40; negative-outcome subjects receive 0.80, 0.30, and 0.10.
Classify a subject as positive when the score is at least 0.60. The positive subjects scored 0.90 and 0.70 are true positives, while the one scoring 0.40 is a false negative. Among the negative subjects, 0.80 produces one false positive, and 0.30 and 0.10 produce two true negatives. Therefore, sensitivity is \(2/(2+1)=0.6667\), specificity is \(2/(2+1)=0.6667\), and accuracy is \((2+2)/6=0.6667\).
These metrics evaluate the threshold chosen without using cohort 3. Selecting a different threshold after looking at cohort 3’s outcomes and reporting the resulting metric as if it were an unbiased held-out assessment would reuse the test data for model selection. If threshold choice is part of the pipeline, it must be resolved within the training data for each LOCO split.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- With four cohorts, how many LOCO fits are made, and how many cohorts are used to train each fit?
- Two held-out cohorts have accuracies 0.90 on 10 subjects and 0.60 on 30 subjects. Calculate the equal-weight cohort mean and the pooled accuracy.
- Why is it leakage to estimate a data-dependent scaling transformation using all cohorts before running LOCO?
- What does an equal-weight mean of LOCO cohort sensitivities give equal weight to, and what does pooled sensitivity weight cohorts by?
Solutions. (1) Four fits; each trains on the other three cohorts. (2) The cohort mean is \((0.90+0.60)/2=0.75\). The pooled accuracy is \((0.90\times10+0.60\times30)/40=27/40=0.675\). (3) The held-out predictors influenced the estimated transformation, so the test cohort was not fully excluded from the fitted pipeline; estimate the transformation separately from each training set. (4) The equal-weight mean gives each cohort equal weight. Pooled sensitivity weights each cohort according to its number of positive-outcome subjects.