Why Rare-State Preservation Needs Its Own Metrics
An integration method can align batches well overall while damaging a small but biologically important population. If rare cells are absorbed into a common state, an aggregate score may barely change: the many common cells dominate the calculation. Evaluating rare-state preservation therefore requires metrics whose denominators explicitly refer to the rare cells or to proposed matches involving them.
A benchmark needs a reference definition of the rare state that is not created solely by the integration result being evaluated. For example, the state may be defined by trusted labels, an independent assay, or known truth in a simulation. Labels should be sufficiently reliable for the intended evaluation, and the evaluation population should include the batches and biological variation relevant to the claim. The earlier material on defining benchmark targets is useful here: a metric is only meaningful relative to a clearly stated target.
Recovery asks what fraction of the true rare cells were retained or identified as rare; it is also called sensitivity or recall. Precision asks what fraction of cells called rare truly belong to that state. They answer different questions. A method can recover nearly all rare cells by assigning many common cells to the rare label, giving high recovery but low precision. If no cells are predicted rare, precision has a zero denominator and is undefined; it should not be silently reported as zero or one.
Worked Example: Aggregate Accuracy Hides Lost Rare Cells
Worked Example: Aggregate Accuracy Hides Lost Rare Cells
Suppose a hypothetical benchmark contains 1,000 cells: 20 are truly rare and 980 are common. Two integration pipelines are evaluated using the same binary rare-state prediction rule. Method A identifies 8 rare cells correctly and labels 10 common cells as rare. Method B identifies 18 rare cells correctly and labels 21 common cells as rare.
For Method A, \(TP=8\), \(FN=20-8=12\), \(FP=10\), and \(TN=980-10=970\). Its accuracy and recovery are
For Method B, \(TP=18\), \(FN=20-18=2\), \(FP=21\), and \(TN=980-21=959\). Thus
The accuracy scores differ by only 0.001, but Method B recovers 10 more rare cells: 18 rather than 8, or \(18/8=2.25\) times as many. The difference is consequential even though the aggregate scores look almost identical. Precision also matters: Method A has \(8/(8+10)=0.4444\), while Method B has \(18/(18+21)=0.4615\). In this example, B improves both rare-state recovery and precision, but this need not happen in every benchmark.
Why Accuracy Can Be Misleading
The masking effect follows directly from writing accuracy in terms of rare-state prevalence. Let \(\pi=R/N\) be the fraction of true rare cells, let \(r\) be rare-state recovery, and let \(s=TN/(TN+FP)\) be specificity, the fraction of common cells correctly classified as common. The numbers of correctly classified rare and common cells are \(rR\) and \(s(N-R)\), respectively. Dividing their sum by \(N\) gives
When the state is rare, \(\pi\) is small, so accuracy is driven mostly by specificity on common cells. In the example, \(\pi=20/1000=0.02\). Method A has specificity \(970/980=0.9898\), giving \(0.02(0.4)+0.98(0.9898)=0.978\). Method B has specificity \(959/980=0.9786\), giving \(0.02(0.9)+0.98(0.9786)=0.977\). The large difference in recovery contributes little to overall accuracy because rare cells are only a small part of the benchmark.
This is a descriptive identity, not an assumption that the cells were sampled randomly. It explains the arithmetic of accuracy for the benchmark at hand. It also shows why changing the rare-state prevalence between benchmarks changes accuracy, even if recovery and specificity remain fixed. Report the prevalence and the class-specific metrics rather than comparing accuracy scores in isolation.
Neighborhood Purity for Rare Cells
Label-based recovery evaluates predicted labels. Neighborhood purity instead evaluates whether rare cells remain near cells with the same known biological state in the integrated representation. This distinction matters when an integration method produces coordinates or a neighbor graph but no direct cell labels.
The choice of \(k\), distance measure, and whether the graph is directed must be fixed and reported. Neighbors should ordinarily exclude the cell itself. If a method yields variable neighborhood sizes, state whether the summary is an average of per-cell fractions or a pooled fraction of all neighbor links; these can differ because they weight cells differently. A label-defined neighborhood purity is not by itself a measure of batch mixing: a rare state could be biologically coherent but still segregated by batch.
Worked Example: Equal Mean Purity, Different Rare-Cell Coverage
Consider five hypothetical cells in the same rare state, each with \(k=4\) neighbors. Method X has rare-neighbor counts \(4,4,4,0,0\); Method Y has counts \(3,3,3,3,0\). The pooled rare-neighborhood purity for X is
For Y it is \((3+3+3+3+0)/20=12/20=0.6\) as well. Yet a simple coverage diagnostic—the fraction of rare cells with at least one rare neighbor—differs: X has \(3/5=0.6\), while Y has \(4/5=0.8\). The same mean purity can therefore describe quite different patterns. X has three cells with fully rare neighborhoods and two with none; Y has four cells with mixed neighborhoods and one with none. Inspect per-cell values or their distribution, not only the mean.
False Matches Across Batches
Some benchmarks evaluate candidate cross-batch correspondences or links rather than assigning every cell a rare-versus-common label. For such a benchmark, define a false match by reference biology: a proposed pair is false if its two cells do not belong to the same biological state under the benchmark’s matching criterion. The criterion may be stricter than sharing a broad cell-type label, so it must be specified before counting errors.
The terminology “false-match rate” is not uniform across all applications, so report this denominator explicitly. In a binary rare-state classification table, the false-positive rate is \(FP/(FP+TN)\): among truly common cells, the fraction incorrectly called rare. The false discovery proportion is \(FP/(TP+FP)\), which is one minus precision when the denominator is nonzero. These quantities answer different questions and should not be substituted for one another.
Worked Example: Overall False-Match Rate Conceals Rare-Cell Errors
A hypothetical integration proposes 100 cross-batch cell pairs. Reference labels show that 12 pairs are biologically discordant, so the false-match rate among proposed pairs is
Suppose 10 proposed pairs involve a rare-state query cell, and 4 of those are discordant. The rare-query false-match rate is \(4/10=0.4\). The other 90 pairs include the remaining \(12-4=8\) discordant pairs, giving \(8/90=0.08889\). The overall value of 0.12 blends these strata and makes the rare-query failure less visible. Report the overall rate together with the rare-involving rate and their denominators. If pairs share cells, they are also dependent observations; a naive interval treating all pairs as independent can understate uncertainty.
Benchmarking Workflow, Diagnostics, and Limitations
A useful rare-state benchmark makes the target, unit of analysis, and denominator explicit before comparing methods. The following sequence keeps different failure modes separate.
Define which cells count as rare using labels or known simulation truth independent of the integration result. Report the rare-cell count and its fraction of the benchmark.
For predicted labels, calculate recovery and precision, with specificity or false-positive rate for the non-rare class. For an integrated graph, calculate rare-cell neighborhood purity and inspect cell-level values. For candidate links, calculate false-match rates with explicit denominators.
Report performance by rare state and, where relevant, batch. Check confusion counts, empty denominators, isolated rare cells, and whether a high mean is driven by a few well-preserved cells.
Interpret rare recovery alongside precision and false matches, and interpret mean neighborhood purity alongside its distribution. Do not select a method from aggregate accuracy alone.
Useful diagnostics include a confusion table, the number of cells and links in every denominator, per-cell neighborhood-purity distributions, and a breakdown of errors by batch and state. A suspiciously perfect score may reflect label leakage—for example, using the same labels to build the evaluated representation and to define its success. Missing rare cells in a batch also matter: a cross-batch metric cannot establish successful matching to a state that is absent from that batch.
The main alternatives answer complementary questions. Overall accuracy gives a compact summary but can hide rare-class errors. Recovery emphasizes missed rare cells but ignores false rare assignments; precision emphasizes the reliability of rare calls but ignores rare cells that were missed. Neighborhood purity evaluates local structure but depends on graph construction and \(k\), and high purity does not guarantee that all rare cells are connected or recovered. False-match rates directly assess proposed links, but depend on the definition of a correct biological match and can be dominated by the composition of the proposed-pair set.
These metrics are benchmark summaries, not proof that an integration preserves every biologically meaningful distinction in new data. They depend on label quality, state definitions, sample composition, and the chosen decision rules. When benchmark cells or links are correlated, uncertainty calculations should account for that dependence rather than treating every cell or pair as independent. Finally, rare-state performance can be unstable when only a few reference cells are available: report counts alongside proportions so readers can see the evidence supporting each value.
Common Pitfall
Check Your Understanding
For each question, first identify the denominator being evaluated. Solutions follow each question.
- A benchmark has 15 true rare cells, of which 6 are predicted rare. What is recovery? Solution: \(TP=6\) and \(TP+FN=15\), so recovery is \(6/15=0.4\).
- Of those 9 predicted-rare cells, 6 are truly rare. What is precision, and how does it differ from recovery? Solution: Precision is \(6/9=0.6667\), the fraction of predicted-rare cells that are truly rare. Recovery uses all true rare cells as its denominator; it cannot be computed from these 9 predicted cells alone without knowing the total number of true rare cells.
- Four rare cells each have five neighbors, with rare-neighbor counts \(5,3,2,0\). Find pooled mean rare-neighborhood purity and the fraction with at least one rare neighbor. Solution: There are \(4\times5=20\) neighbor slots and \(5+3+2+0=10\) rare neighbors, so pooled purity is \(10/20=0.5\). Three of four cells have at least one rare neighbor, so coverage is \(3/4=0.75\).
- A method proposes 40 cross-batch pairs, 6 of which are biologically discordant. What false-match rate is being reported? Solution: Among proposed pairs, it is \(6/40=0.15\). This is not the classification false-positive rate, whose denominator would be all truly negative cell-level cases.
- Why can two methods have nearly equal accuracy but markedly different rare-state recovery? Solution: Accuracy is \(\pi r+(1-\pi)s\), where \(\pi\) is rare-state prevalence, \(r\) is recovery, and \(s\) is specificity. When \(\pi\) is small, the common-cell term \((1-\pi)s\) dominates, so a large change in \(r\) may have little effect on accuracy.