Diagnosing Uncertainty for Rare Cell Types

Why Rare Labels Need a Separate Uncertainty Check

A classifier can perform well overall and still struggle with a rare cell type. Overall accuracy may barely change when a small class is consistently missed, and a single cell can substantially change a rare class’s estimated performance. A useful diagnosis therefore asks two different questions: how uncertain is the estimate? and do the observed errors show a repeated pattern?

The key denominator for recall is the number of evaluated cells whose reference label is the rare type—not the total number of cells, and not the number of cells assigned that label by the classifier. Probability calibration asks a related but distinct question: among cells given a particular probability for the type, how often is that type actually the reference label? Both questions need enough relevant validation observations to support a stable estimate.

This is a conditional performance measure: it conditions on the reference label being \(c\). It is not the fraction of predicted-\(c\) cells that are correct; that is precision. A rare type in the data need not have imprecise recall if many validation cells of that type have been collected. Conversely, a reasonably common type can have an imprecise estimate if its validation count is small.

Uncertainty in Classwise Recall

A simple model treats the correctness indicator for each evaluated reference-\(c\) cell as a Bernoulli variable: it equals 1 if predicted correctly and 0 otherwise. If the classifier is fixed, the cells are representative of the target population of interest, and their outcomes are independent with a common probability \(r_c\), then \(x_c\) has a binomial distribution with size \(n_c\) and success probability \(r_c\). Under this model, \(\hat{r}_c\) estimates \(r_c\), with standard error approximately \(\sqrt{\hat{r}_c(1-\hat{r}_c)/n_c}\). That approximation can behave poorly for small counts or estimates near 0 or 1, so a Wilson interval is often preferable to a simple estimate-plus-or-minus-standard-error interval.

The Wilson interval is obtained by inverting the score test for a binomial proportion. For a candidate probability \(p\), the score-test acceptance condition is \(n(\hat{p}-p)^2\le z^2p(1-p)\). Expanding and collecting terms gives

The endpoints are the two roots of this quadratic. Completing the quadratic formula gives \(C-H\) and \(C+H\) above. Unlike a symmetric normal interval, these endpoints remain between 0 and 1 and need not be centered on \(\hat{p}\).

Calibration Requires Class-Specific Evidence Too

A classifier may output, for each cell \(i\) and type \(c\), a probability \(q_{ic}\) that the cell belongs to \(c\). Calibration for type \(c\) means that among cells assigned probabilities near \(q\) for that type, the fraction whose reference label is \(c\) is near \(q\). In notation, the ideal condition is \(P(Y_i=c\mid q_{ic}=q)=q\), where \(Y_i\) is the reference label. In finite data, this is examined over probability ranges or bins.

For rare types, a bin can contain very few validation cells, so its observed fraction correct can vary widely. Report the number of cells contributing to each class-and-probability bin alongside its average predicted probability and observed fraction. A reliability diagram without these counts can make a noisy point look as authoritative as a point supported by many observations. Binning also hides variation within a bin, so apparent agreement is not proof of perfect calibration.

Reference Counts and Validation Counts Answer Different Questions

The number of cells of type \(c\) in a reference atlas describes how much reference material is available for representing that label. It can flag limited support: a type represented by only a few cells may be difficult to characterize across its biological variation. But that atlas count is not automatically the denominator for validation recall. Recall must use independently evaluated cells with a reference label of \(c\). Likewise, the number of atlas cells does not show how well probabilities are calibrated on a target dataset.

Keep at least three counts conceptually separate: the number of reference-atlas cells of type \(c\), the number of validation cells with reference label \(c\), and the number of cells assigned label \(c\) by the classifier. They have different roles. A very small atlas count suggests limited representation; a small validation count yields imprecise performance estimates; a small predicted count may affect precision or the practical yield of that label. None can be substituted for another.

Diagnostics, Alternatives, and Limits

A useful rare-label report places the estimate, its denominator, and its uncertainty together. For each type, inspect the recall numerator and denominator, the interval, and the destinations of errors. For calibration, show class-specific bin counts as well as average predicted and observed probabilities. Compare results across donors or other independent sampling units when possible. A class with wide intervals and errors scattered among labels presents a different diagnostic picture from one with repeated errors to the same competing type, even if the point recalls happen to match.

The binomial interval is a practical alternative to reporting a point estimate alone, but its assumptions matter. If multiple cells come from the same donor, treating every cell as an independent Bernoulli observation can overstate the effective sample size. Donor-level summaries, resampling donors, or a model that accounts for clustering may be more suitable, depending on the design and estimand. If the reference labels themselves are uncertain or inconsistent, the calculation quantifies agreement with those labels, not error relative to an unquestionable biological truth.

Other summaries answer different questions. Overall accuracy weights types according to their frequency and can obscure rare-type failures. Precision focuses on the correctness of predicted labels rather than recovery of reference-labeled cells. A confusion matrix displays where errors go, while recall intervals quantify uncertainty in a classwise proportion. Reliability diagrams assess probability calibration, not whether the predicted class is biologically correct in every individual case. No single summary replaces the others.

A large reference count does not guarantee a representative reference, and a small count does not prove that predictions are wrong. Similarly, a low recall estimate with a wide interval is not evidence that recall is acceptable; it is evidence that the available data do not pin it down precisely. Treat absence of evidence for systematic misclassification separately from evidence of good performance.

A Practical Diagnostic Sequence

Common Mistake

• Wrong reasoning: “The atlas has only four cells of this rare type, so the recall denominator is four and its estimated recall is unreliable for that reason.”
• Correction: Atlas representation and validation evidence are distinct. Recall uses the independently evaluated cells with that reference label. A small atlas count is a warning about reference support; a small validation count is what directly limits precision of the recall estimate.
• Wrong reasoning: “The rare type has lower recall, so the classifier systematically confuses it with one particular neighboring type.”
• Correction: Low estimated recall alone does not identify the error pattern. Inspect the destinations of errors and their consistency, and account for the uncertainty in those observations.