Why a Reference Can Miss a Cell Type
A cell-type classifier can only choose among the labels it was given. If the query contains a genuine cell type that is absent from the reference, a classifier may still assign that cell to the closest available label. It may also divide its probability across several labels, producing low confidence. Neither outcome alone establishes that a cell type is missing: low confidence can reflect an ordinary boundary between related types, and a high-quality known type can look unusual because of technical or biological variation.
The practical task is therefore not to make a definitive “novel cell” call from one score. It is to identify cells for which several distinct kinds of evidence raise concern: the classifier is unsure, the cell lies outside the typical reference support, or its expression disagrees with the assigned label’s marker pattern. Quality checks help distinguish these candidates from cells whose weak signal is better explained by sparse or poor-quality measurements.
Three Complementary Signals
Let the reference contain labels \(1,\ldots,K\). For query cell \(i\), let \(\hat p_{ik}\) be the classifier’s predicted probability for reference label \(k\). These probabilities describe the classifier’s allocation among the available labels; they do not automatically include a probability for “unknown” or “novel.”
The maximum probability \(m_i\) summarizes how concentrated the prediction is, and \(\hat k_i\) is the top predicted label. A small \(m_i\) is a warning of uncertainty, not a direct estimate of the probability that the cell has a missing type. Because the probabilities sum to one over the known labels, even a cell unlike every reference cell must receive probability among those labels.
A second signal asks whether the cell lies near the reference cells for its proposed label in a representation used for annotation. Let \(z_i\) be the cell’s vector in that representation, \(\mu_k\) the reference center for label \(k\), and \(\Sigma_k\) a covariance estimate for that label. One possible class-specific distance is the Mahalanobis distance:
This distance accounts for the scale and correlation of the represented features. It requires a usable, invertible covariance estimate; in practice, the representation may need dimension reduction or regularization when there are too few reference cells. A simpler standardized Euclidean distance is another option. Whichever metric is chosen, “large” should be judged relative to distances for known reference cells, preferably within each label, rather than by an arbitrary universal cutoff. For instance, a preselected upper quantile of distances among reference cells can be used as an operational threshold. That threshold is a screening rule, not a guarantee about future cells.
The third signal is marker agreement. Earlier material on marker panels and marker evidence provides ways to assess whether a cell expresses expected positive markers and avoids markers that argue against a proposed label. For screening, record a simple, interpretable summary, such as the fraction of expected positive markers detected and whether strong negative-marker evidence is present. Marker evidence depends on assay depth: failure to detect a marker in a sparse cell is weaker evidence than observing a contradictory marker in a well-measured cell.
Why No Single Signal Is Enough
The signals answer different questions. A low maximum probability says that the classifier has not strongly preferred one available label. A large distance says that the cell is unusual in the chosen reference representation. Marker discordance asks whether the proposed biological identity fits the observed expression. These checks are complementary, but not statistically independent: the same expression features may influence the classifier, the representation, and the marker scores. Their evidence should not be multiplied as if independent likelihoods unless a suitable joint model has been specified.
There is a simple reason confidence cannot by itself detect novelty. A closed-set classifier returns \(K\) probabilities whose sum is one, even when the query cell comes from outside the represented classes. A model can be uncertain and spread probability across labels, or be highly confident because its decision rule extrapolates beyond the observed reference data. Thus, low confidence is neither necessary nor sufficient for a missing type.
The inequality follows because if every probability were less than \(1/K\), their sum would be less than one. It gives only a mathematical lower bound; it does not make a value near \(1/K\) evidence of novelty. Conversely, a large \(m_i\) does not show that the cell is well represented by the predicted class. A classifier’s probability is about its label assignment under the model, not a general measure of how typical the cell is.
A useful screening procedure checks several signals and then asks whether basic cell-quality measures are adequate. Relevant checks can include detected genes, total counts, mitochondrial-read proportion, and evidence of a doublet or contamination. These measures need context-specific reference ranges and should not be reduced to a single universal cutoff. A cell with few detected genes may lack marker signal and have an unstable prediction; that pattern is different from a well-measured cell with a coherent but unrepresented marker profile.
Worked Example: Several Signals Support a Novelty Review
Suppose a hypothetical query cell has predicted probabilities \(0.42\), \(0.35\), and \(0.23\) for three reference labels. In a two-dimensional representation, its nearest proposed-label center is \(\mu=(0,0)\), its position is \(z=(3.8,1.5)\), and the class covariance is the identity matrix. For this illustration, assume a screening rule flags maximum probabilities below \(0.60\) and distances above a reference-derived threshold of \(3.20\). The expected positive-marker panel for the proposed label has six markers, of which five are detected; no strong contradictory marker is observed. Quality measures are within the range seen for adequately measured cells in this hypothetical analysis.
The maximum probability is \(m_i=\max(0.42,0.35,0.23)=0.42\), below the illustrative \(0.60\) threshold. The squared Mahalanobis distance is \(3.8^2+1.5^2=14.44+2.25=16.69\), so \(d_i=\sqrt{16.69}=4.085\), above \(3.20\). The positive-marker fraction is \(5/6=0.833\).
The low maximum probability and large distance both raise concern, while the marker pattern is reasonably compatible with a coherent identity and the quality checks do not suggest poor measurement. Together, these observations justify flagging the cell or a group of similar cells for review as a possible missing type. They do not establish a new biological identity: the marker panel may be incomplete, the representation may be affected by technical variation, or the reference may fail to capture a known subtype.
Worked Comparisons: Quality and Confident Extrapolation
Worked Example: Low Confidence Explained by Weak Measurements
A second hypothetical cell has probabilities \(0.46\), \(0.39\), and \(0.15\). Its distance from the nearest reference center is \(2.10\), below the illustrative \(3.20\) threshold. Only two of six expected positive markers are detected, but the cell has 420 detected genes, compared with a typical value near 1,800 in the relevant reference cells. No strong contradictory marker is detected.
Here \(m_i=\max(0.46,0.39,0.15)=0.46\), which is below \(0.60\), but \(d_i=2.10<3.20\). The marker fraction is \(2/6=0.333\). The low probability and sparse marker detection are concerning, but both can arise from weak measurement. The distance is not unusually large, and 420 detected genes is far below the stated reference level. The appropriate interpretation is an uncertain, low-information cell requiring quality review, not a strong missing-type candidate. This conclusion could change if a coherent population of comparably measured cells showed the same unusual profile.
Worked Example: A Confident Label Outside Reference Support
A third hypothetical cell has probabilities \(0.91\), \(0.06\), and \(0.03\), so its assigned label appears highly confident. Its representation is \(z=(4.0,2.0)\), with nearest reference center \((0,0)\) and identity covariance. The same illustrative distance threshold, \(3.20\), applies. The cell has adequate detected genes and expresses only two of seven expected positive markers for the assigned label, while a marker associated with a different lineage is strongly expressed.
The maximum probability is \(m_i=0.91\), so the low-confidence rule does not flag it. However, \(d_i=\sqrt{4.0^2+2.0^2}=\sqrt{20}=4.472\), exceeding \(3.20\). Its expected-marker fraction is \(2/7=0.286\), and the strong contradictory marker adds concern. Since the cell is adequately measured, poor quality is a less persuasive explanation. This is a useful example of why screening only on maximum probability can miss a cell outside reference support. The cell should be reviewed, not automatically renamed: a technical artifact, an unusual state, or a reference mismatch could also produce this pattern.
Practical Screening and Failure Modes
A defensible workflow makes the screening choices explicit and interprets them together. The following sequence keeps the output appropriately cautious.
Record which cell types are represented and which expression features or embedding define the distances. A cell cannot be identified as absent from a label set without describing that set.
Calculate \(m_i\) and compare the cell’s distance with a reference-based threshold appropriate to the proposed label. Check that the reference has enough cells to estimate its distance distribution sensibly.
Ask whether observed positive and negative markers fit the assigned label, and whether sparse counts or other quality concerns could explain weak evidence.
Give greater priority to well-measured cells with multiple discordant signals, especially when similar cells recur with a consistent marker pattern. Preserve the original prediction and record the reasons for the flag.
Thresholds depend on the classifier, representation, label granularity, assay, and intended use. A threshold chosen after inspecting the same query cells can make the apparent evidence look stronger than it is. Prefer rules specified using reference data and state them as screening rules. A reference-derived distance threshold can also behave poorly for a rare or heterogeneous class: a small sample may yield an unstable estimate, while a broad class may have a wide distance distribution that masks unusual subgroups.
Several alternatives answer related but different questions. Selective classification with an abstain option withholds a label when a prediction is uncertain; it does not by itself distinguish a missing type from a difficult boundary case. A forced closed-set label always returns a reference label and can conceal cells outside reference support. Manual marker review adds biological context but depends on panel quality and expert judgment. Grouping similar flagged cells may reveal a recurring pattern, but a group is not automatically a distinct cell type. The right interpretation depends on the evidence and on the biological resolution being claimed.
- Low probability alone: may indicate ambiguity among represented labels rather than novelty.
- Large distance alone: may result from batch effects, an unusual known state, or an unstable representation.
- Marker absence in a sparse cell: is weak evidence of mismatch because the transcript may simply not have been detected.
- Marker conflict in a well-measured cell: merits inspection, but can also reflect doublets, contamination, or transitional biology.
- One unusual cell: is weaker evidence for a missing type than a reproducible set of adequately measured cells with a coherent profile.
The main limitation is that novelty is relative to the reference and the chosen representation. A type absent at fine resolution may be represented under a broader parent label. A true cell type may overlap the reference in expression space, and a technical artifact may look far away. Screening therefore prioritizes investigation; it does not replace biological interpretation or establish a new ontology label.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A classifier has four reference labels. What lower bound must the maximum predicted probability satisfy, and what does that bound not tell you?
- A cell has maximum probability \(0.38\), a distance below its label’s reference threshold, and very few detected genes. What is the most cautious interpretation?
- For a two-dimensional cell with identity covariance and displacement \((2,3)\) from the nearest class center, calculate its Mahalanobis distance.
- A well-measured cell has maximum probability \(0.95\), a distance above the reference threshold, and strong marker discordance. Should it be dismissed because its probability is high?
Solutions. (1) Since four probabilities sum to one, at least one is at least \(1/4=0.25\); this bound says nothing about whether the cell is typical of a reference label or novel. (2) Low confidence could reflect poor measurement, especially with few detected genes. The near-reference distance gives no positive support for an out-of-reference cell, so review quality and retain the result as uncertain rather than calling it novel. (3) The distance is \(\sqrt{2^2+3^2}=\sqrt{13}=3.606\). Whether that is large depends on the reference-derived threshold. (4) No. High maximum probability concerns the model’s preference among available labels. The large distance and marker conflict warrant review, while still leaving technical and biological alternatives to a missing type.