Donor-Held-Out Validation of Cell-Type Annotation

Why the Split Unit Matters

A cell-type annotation model is useful only to the extent that it works on the cells to which it will be applied. In many studies, the intended use is to annotate cells from a donor not used to fit or select the model. Validation must therefore mimic that use. If cells from the same donor appear in both training and test sets, the test can reward recognition of donor-specific profiles rather than recognition of cell types that generalize across donors.

Cells from one donor share biological and technical features. These may include genuine donor-specific expression, tissue condition, sample handling, sequencing depth, and processing effects. A model can exploit some of these patterns, even if they are not the intended basis for assigning cell types. A random split of individual cells can place near-replicates of these donor characteristics on both sides of the split. Holding out entire donors tests a more demanding and often more relevant target: performance on new biological donors drawn from the population of interest.

Let \(d(i)\) denote the donor from which cell \(i\) came, \(Y_i\) its reference cell-type label, and \(\hat{Y}_i\) the model's predicted label. A donor-held-out test set \(T\) is donor-disjoint from the training set \(R\): \(d(i)\ne d(j)\) for every \(i\in T\) and \(j\in R\). The model and any fitted preprocessing steps are learned using \(R\), then applied to \(T\).

The validation target should be stated before choosing the summary. If each cell in the intended application is equally important, pooled cell accuracy may be relevant. If the goal is performance for a typical new donor, average donor-level performance gives each donor equal weight. These quantities can differ substantially when donors contribute different numbers of cells.

What Donor-Held-Out Validation Estimates

A donor-held-out result estimates performance under a particular target setting: new donors, with the tissue, assay, label definitions, and other study conditions represented by the validation donors. It does not guarantee performance for a different tissue, laboratory, assay, or population. Those changes may alter the relationship between expression profiles and cell types.

The design relies on several conditions. First, the held-out donors should be reasonably representative of the donors to which the model will be applied. Second, reference labels should be sufficiently reliable for the intended evaluation; a model cannot be judged cleanly against labels that are systematically wrong or inconsistent. Third, the test donors must not influence model fitting, feature selection, tuning, or other decisions that are being evaluated. Finally, the number of independent donors—not merely the number of cells—must be adequate for the intended claim.

A marker panel specified in advance from independent biological knowledge can be applied to held-out donors without training on their labels. By contrast, selecting markers after inspecting which genes separate labels in the full dataset lets test information influence the model. Similarly, scaling parameters, dimensionality-reduction loadings, integration steps, or probability calibration fitted using all donors can compromise a strict new-donor evaluation. In each case, ask whether information from a held-out donor would have been available when the model was supposedly deployed.

Why Random Cell Splits Can Mislead

A cell-level random split usually places cells from the same donor in both training and test sets. To see how likely that overlap can be, suppose each cell is independently assigned to training with probability \(q=0.80\) and to testing with probability \(1-q=0.20\). Conditional on one particular cell being in the test set, suppose its donor contributed \(m=80\) cells in total. Each of the other 79 cells has probability 0.80 of being in training. The probability that at least one of them is in training is one minus the probability that none are:

Thus, under this independent assignment scheme, a test cell is virtually certain to have at least one same-donor cell in training. This calculation establishes a high probability of donor overlap; it does not prove that every overlapping split will inflate accuracy by a fixed amount. The degree of optimism depends on how strongly donor-specific information affects predictions and on the task.

A random cell split may still be suitable for a different target, such as predicting additional cells from donors already represented in training. It is not a substitute for testing on new donors when new-donor generalization is the question. The split should reflect the intended deployment setting, not just produce a convenient test set.

Designing a Donor-Level Evaluation

A practical evaluation begins by deciding what information the model may use and what population it should generalize to. Then allocate whole donors to training, validation, and test groups. Fit all learned steps using only the donors allowed for that step. For repeated model comparisons, use grouped cross-validation, with donors—not cells—as the groups.

When tuning model choices, an untouched test set is valuable: use training donors for fitting and validation donors for choices, then evaluate the selected procedure on test donors only. If data are limited, grouped cross-validation can rotate held-out donors, but decisions based on those same cross-validation results make the resulting score a model-selection estimate rather than an independent final test. A nested design can separate tuning from evaluation by performing donor-grouped tuning inside each outer training set and evaluating on the outer held-out donors.

The split also needs enough representation of relevant cell types. A donor-held-out fold may lack a cell type if that type is rare or absent in some donors. This is not a reason to put cells from the same donor on both sides. Instead, inspect donor-by-cell-type representation before finalizing folds, describe which classes can be evaluated, and avoid claiming performance for a class that the held-out data do not contain.

Diagnostics, Alternatives, and Limitations

Before interpreting a score, verify the design rather than relying on the name of the split procedure. Check that donor identifiers are disjoint across partitions, that no cell is duplicated across them, and that all fitted preprocessing and selection steps obey the stated boundary. Review donor-level results for a pattern in which one or two donors dominate the pooled score. Also check whether some held-out donors have cell types absent from training; low performance in that setting may reflect missing training support, not merely ordinary variation among familiar classes.

The main alternative, a random cell-level split, is easier to implement and may answer a known-donor prediction question, but it usually does not estimate generalization to unseen donors. A single donor holdout is transparent but can be highly dependent on which donor was chosen. Grouped \(K\)-fold cross-validation uses several held-out donor groups and can show variation across folds, though its estimates remain limited by the available donors and by any tuning performed on those results. An external cohort provides a stronger test of transport to a distinct data collection when it matches the intended application, but may differ in protocol or population in ways that complicate interpretation.

Donor-held-out validation is not a universal cure for distribution mismatch. If donors were all recruited under one narrow condition, holding them out does not establish performance in a different condition. Nor does the procedure by itself diagnose which cell types are confused or explain why predictions fail; those questions require examining prediction errors and relevant class-level behavior. The central contribution is a defensible separation between donors used to develop the method and donors used to assess its new-donor performance.