Label-Transfer Accuracy as an Integration Benchmark

What Label-Transfer Accuracy Measures

Integration methods are often used to align cells across batches or datasets. One way to assess whether an integrated representation retains information about cell identity is to use it for label transfer: labels from a reference dataset are used to predict labels for query cells. The benchmark then compares those predictions with independently established query labels. The resulting accuracy measures performance of the complete transfer procedure on that benchmark; it is not a direct measurement of integration quality alone.

The distinction matters because transfer also depends on the reference, the prediction rule, the label definitions, and the query cells. A method may appear accurate because the query contains mostly an easy, common cell type, or because reference information leaked into prediction. A useful benchmark therefore states which cells and labels are being evaluated, how transfer is performed, and how class frequencies and annotation uncertainty are handled.

This calculation assumes that the evaluation labels are a suitable benchmark for the target labels, that the transfer procedure has not used those labels to make its predictions, and that each evaluated cell is counted once. The labels must also have consistent meanings across reference and query. If one dataset uses a broad label and the other uses fine subtypes, a mismatch may reflect different definitions rather than an integration failure.

A clean design keeps query labels out of model fitting, feature selection, tuning, and transfer decisions. Reference labels may be used to train the transfer rule, but query labels should be held back for evaluation. If cells from the same donor or sample are dependent, cell-level accuracy still describes the observed cells, but treating every cell as independent can make uncertainty appear smaller than it is. Where uncertainty intervals are needed, resampling donors or samples is generally more defensible than resampling individual cells alone.

Class Imbalance and Balanced Accuracy

Overall accuracy weights each cell equally. Consequently, it weights a class in proportion to its frequency in the query set. That is appropriate when the target is the fraction of cells correctly labeled in a query population with that particular composition. It can be inadequate when the benchmark should give rare and common classes equal importance.

For class \(j\), its recall (also called sensitivity) is the proportion of cells with reference label \(j\) that were assigned label \(j\). Balanced accuracy is the unweighted mean of those class-specific recalls:

The row total in the denominator is the number of query cells in class \(j\). Balanced accuracy gives each represented class equal weight, regardless of how many cells it contributes. It is not a replacement for overall accuracy: the two answer different questions. If the target is performance on the observed cell mixture, report overall accuracy; if equal class importance is intended, also report balanced accuracy and the class-specific recalls.

A short justification makes the weighting explicit. Write \(n_j=\sum_k C_{jk}\) for the number of query cells in class \(j\). Overall accuracy can be rewritten as:

Thus, overall accuracy is a weighted mean of recalls, with class weights \(n_j/n\). Balanced accuracy replaces those observed class weights with equal weights \(1/K\). This explains why the metrics can diverge when class frequencies and recall differ.

Uncertain Reference Labels

A reference label is not always a certain truth. An annotation may be ambiguous, supported by limited markers, or assigned at a resolution coarser than the biological variation in the data. Reporting a single hard label as unquestionable ground truth can make a disagreement look like a transfer error even when the annotation itself is uncertain.

When annotation uncertainty is represented as a probability distribution, let \(q_{ij}\) be the probability or adjudicated weight that query cell \(i\) belongs to class \(j\), with \(q_{ij}\geq 0\) and \(\sum_{j=1}^{K}q_{ij}=1\). If the transferred label is \(\hat y_i\), an expected accuracy is the mean weight assigned by the annotation distribution to that transferred class:

This quantity is an expected match under the supplied label distributions, not a statement that those distributions are known objective probabilities. Its interpretation depends on how the uncertainty weights were constructed. If only hard labels are available, do not invent probability weights; instead, report sensitivity to defensible alternative annotations or separately evaluate a set of high-confidence labels.

If the transfer method produces class probabilities rather than only a winning label, accuracy discards the confidence information: predictions with probabilities 0.51 and 0.99 for the selected class count equally when both are correct. A probability-based score can be reported as a supplement when uncertainty in the predictions is itself relevant, but it should not be conflated with label accuracy. When the reference labels are uncertain too, both sources of uncertainty should be described rather than treating the transferred probabilities as if they resolved annotation ambiguity.

Query States Missing From the Reference

A label-transfer rule cannot reliably assign a reference class to a biological state that is not represented by any reference cell. In a closed-set procedure that must choose one of the available labels, it will still output a label, possibly with high confidence. That output does not demonstrate that the query state is represented or correctly identified.

Separate support from correctness. A query cell is supported for closed-set transfer when its independently established label is present among the reference classes. The supported fraction is the number of supported query cells divided by the total number evaluated. Accuracy conditional on support is the fraction correctly labeled among supported cells. These quantities describe different aspects of the benchmark and should be reported together when query states are missing from the reference.

Benchmark Design, Diagnostics, and Alternatives

A defensible comparison uses the same reference-query split, label definitions, and evaluation set for each integration method. The transfer procedure should also be specified: for example, the representation supplied to it, the prediction rule, and any rejection threshold. If these choices differ between methods, an accuracy difference cannot be attributed to integration alone. Where a query label helped select parameters or choose the best result, the evaluation is no longer an independent assessment of that choice.

Inspect the confusion matrix rather than relying only on one summary. It shows which classes are confused, whether errors are concentrated in one class, and whether a class has too few query cells for a stable recall estimate. Also check that every reported class has a nonzero denominator; recall for a class absent from the query is undefined, not zero. An absent class should not be silently included in the balanced-accuracy average.

Several failure modes deserve particular attention. If reference and query cells are not kept separate during fitting, apparent performance can be inflated by information leakage. If a broad reference label is compared with fine query labels, label granularity can create artificial errors. If class frequencies differ substantially, overall accuracy alone can conceal poor performance for less common classes. If labels are uncertain, a hard-label score can overstate the certainty of the benchmark. Finally, cell-level results may be dominated by a donor with many cells; summarize or resample at the donor or sample level when the scientific target is performance across donors or samples.

No single accuracy measure establishes that integration has preserved all relevant biology. High label-transfer accuracy shows that the chosen labels remain predictable under the specified transfer setup; it does not establish that every biological distinction is preserved, that batch effects have been removed appropriately, or that a transferred label is causally or biologically definitive. Interpret the metric alongside the benchmark target and the other evidence used to assess integration.

Common Mistake