Cell-Type Annotation and Its Uncertainty · Tutorial 323 of 700

Cell-Type Annotation as a Statistical Decision

Learn to choose a cell-type annotation by weighing marker or reference evidence against the costs of incorrect, ambiguous, and withheld labels.

Intermediate 11 min read

What You'll Learn

Define cell-type labels, ambiguous labels, and abstention as distinct decision actions.
Calculate expected loss from candidate-type probabilities and a specified loss table.
See how asymmetric error costs can make the best decision differ from the most probable label.
Derive a decision threshold for choosing a label rather than abstaining.
Identify assumptions and diagnostics that matter when using marker or reference evidence.
Compare cost-sensitive decisions with forced labels and fixed confidence cutoffs.

Annotation Is a Decision, Not Just a Match

A cell-type annotation turns evidence into an action: a label is assigned to a cell or a group of cells. The evidence may include marker patterns or information from a reference. But the strongest apparent match does not automatically determine the best action. A mistaken specific label may be more damaging than an explicitly ambiguous label, while leaving a cell unassigned may also have a cost. Statistical decision theory makes these trade-offs explicit.

The decision is separate from how the evidence was produced. A marker result, a reference match, or a probability estimate is an input to the decision; none is itself a decision rule. In particular, a marker-test p-value is not the probability that a cell belongs to a type. Earlier material on post-clustering marker evidence is relevant here: evidence selected or estimated after clustering should be interpreted with that context in mind, not treated as automatically independent confirmation.

Definition: Let $T$ denote the cell's true type, $E$ the evidence available for annotation, and $a$ a possible annotation action. A loss function $L(a,t)$ assigns a numerical cost to taking action $a$ when the true type is $t$. Given evidence $E=e$, the expected loss of action $a$ is $$ R(a\mid e)=\sum_t L(a,t)P(T=t\mid E=e). $$ A minimum-expected-loss decision chooses an available action with the smallest $R(a\mid e)$.

The possible actions can include specific labels, an ambiguous label, and abstention. An ambiguous label records that the evidence supports a candidate set without resolving which member is correct; for example, “type A or type B.” It is still an annotation, but a less specific one. Abstention means withholding a type assignment, such as recording “unassigned.” These actions should not be treated as interchangeable: their consequences and costs can differ.

For an ambiguous action naming a set $S$ of candidate types, a loss table might charge a modest cost when the true type is in $S$, because the annotation is unresolved, and a larger cost when the true type is outside $S$. A project could instead specify different costs for different errors or uses. There is no universally correct numerical loss table: it should represent the intended use of the annotations. The following examples use hypothetical loss units, not biological measurements.

From Evidence to an Action

A decision rule requires a probability distribution over the possible true types given the evidence. Write $p_t=P(T=t\mid E=e)$, so $p_t\geq 0$ and $\sum_t p_t=1$ over the candidate types. These probabilities may come from an appropriate model or other justified assessment. They must not be invented by relabelling marker-test p-values, and they should not be used as probabilities unless their interpretation is warranted. The loss table is a separate input: it describes consequences, not biological evidence.

The basic calculation is to multiply each possible true type's loss by its probability and add the results. Choose the action with the lowest sum. A tie means that the stated model does not distinguish the tied actions; the tie can be reported or resolved using a prespecified secondary rule. It should not be resolved by silently changing the costs after seeing the desired label.

Worked Example: A High-Confidence Specific Label

Suppose hypothetical marker and reference evidence is summarized by $P(T=A\mid E=e)=0.90$ and $P(T=B\mid E=e)=0.10$. The available actions are to label the cell A, label it B, use the ambiguous label “A or B,” or abstain. Assume a wrong specific label costs 8 units, a correct specific label costs 0, the ambiguous label costs 1.5 units when either candidate is true, and abstention costs 2.5 units for either true type.

For action A, the only nonzero loss occurs when the true type is B: $$ R(A\mid e)=0(0.90)+8(0.10)=0.800. $$ For action B, the loss occurs when the true type is A: $$ R(B\mid e)=8(0.90)+0(0.10)=7.200. $$ The ambiguous action has expected loss $1.5(0.90)+1.5(0.10)=1.500$, and abstention has expected loss $2.5(0.90)+2.5(0.10)=2.500$. Thus A has the smallest expected loss, 0.800 units. Under these stated costs and probabilities, assigning A is preferred to either expressing ambiguity or withholding the label.

The calculation is conditional on the candidate set containing the true type and on the probability assessment being appropriate for this evidence. If those conditions are doubtful, the numerical minimum can give false reassurance: decision arithmetic cannot repair an incomplete set of candidate types or unreliable probabilities.

Why Error Costs Can Change the Preferred Label

A rule that always chooses the most probable type considers the probabilities but ignores what different mistakes cost. With asymmetric costs, that can be the wrong decision under the specified objective. For example, mistaking one rare or consequential cell type for another may have a different downstream cost from the reverse mistake. The appropriate costs depend on the use of the annotation and should be stated rather than assumed.

Worked Example: An Asymmetric Cost Favors the Less Probable Type

Suppose the evidence gives $P(T=A\mid E=e)=0.65$ and $P(T=B\mid E=e)=0.35$. A wrong A label when the truth is B costs 10 units, whereas a wrong B label when the truth is A costs 3 units. Correct specific labels cost 0. The ambiguous label “A or B” costs 2 units, and abstention costs 2.4 units, whichever type is true.

The expected loss of labeling A is $$ R(A\mid e)=0(0.65)+10(0.35)=3.500. $$ The expected loss of labeling B is $$ R(B\mid e)=3(0.65)+0(0.35)=1.950. $$ The ambiguous action costs $2(0.65)+2(0.35)=2.000$, and abstention costs $2.4(0.65)+2.4(0.35)=2.400$. Therefore, B is preferred, with expected loss 1.950 units. B is not the most probable type, but the cost of the two kinds of error is unequal. Under this loss table, the cost-sensitive decision is not the same as selecting the largest probability.

This decision is not a claim that B is biologically more likely than A. It is a choice of action under a specified loss function. If the consequences change, the preferred action may change even though the evidence probabilities stay fixed.

Ambiguous Labels and Abstention

Ambiguity and abstention are useful only when their costs are made explicit. An ambiguous label preserves some information by naming a candidate set, but leaves its members unresolved. Abstention withholds the assignment. Depending on the downstream purpose, either may be preferable. For example, an unresolved candidate set may be useful for exploratory summaries, while an unassigned record may be safer when a downstream analysis requires a specific type. These are decisions about annotation consequences, not claims about a universal ranking of the actions.

Worked Example: A Candidate Set Beats Both Specificity and Abstention

For a hypothetical cell, let the evidence probabilities be $P(T=A\mid E=e)=0.55$, $P(T=B\mid E=e)=0.35$, and $P(T=C\mid E=e)=0.10$. A wrong specific label costs 5 units, and a correct one costs 0. The ambiguous label “A or B” costs 1 unit when the true type is A or B, and 7 units when the true type is C. Abstention costs 2.2 units for every type.

The expected losses for the specific actions are $$ R(A\mid e)=5(0.35+0.10)=2.250,\qquad R(B\mid e)=5(0.55+0.10)=3.250, $$ and $$ R(C\mid e)=5(0.55+0.35)=4.500. $$ The ambiguous action costs $$ R(\{A,B\}\mid e)=1(0.55+0.35)+7(0.10)=1.600. $$ The expected loss of abstention is $2.2(0.55+0.35+0.10)=2.200$. The ambiguous label has the smallest expected loss, 1.600 units. It is preferred here because it covers 90% of the probability assigned to the candidate types and its specified cost for a true C is not large enough to outweigh that advantage.

The ambiguous label in this example expresses unresolved evidence among named candidates. It does not settle which biological label or level of detail should be used in a cell ontology; that is a distinct question. A loss table also needs to reflect the actual meaning of the ambiguous label. If “A or B” is not an acceptable output for the intended use, assigning it a small cost would misrepresent the decision problem.

A Decision Threshold and Its Assumptions

A simple two-type case shows how abstention creates a threshold. Suppose the only specific action is to label A, the correct-label loss is 0, the cost of labeling A when the truth is B is $C>0$, and abstention costs $r\geq 0$ regardless of the truth. Let $p=P(T=A\mid E=e)$. The expected loss of A is $(1-p)C$, while abstention has loss $r$. A is at least as good as abstention exactly when $$ (1-p)C\leq r \quad\Longleftrightarrow\quad p\geq 1-\frac{r}{C}. $$

Thus the threshold depends on the relative costs, not on a universal convention. This result assumes that the two types exhaust the possibilities, that the stated probabilities are meaningful, that correct A labels have zero loss, and that abstention has the same cost for either truth. If an ambiguous action is also available, its expected loss must be calculated too; the threshold alone no longer determines the best action.

For instance, if $C=8$ units and $r=2$ units, the threshold is $1-2/8=0.75$. With $p=0.70$, the expected loss of A is $0.30(8)=2.40$, which exceeds the abstention cost of 2.00; with $p=0.80$, it is $0.20(8)=1.60$, which is lower. Equality at $p=0.75$ gives a tie. The threshold is justified by comparing the two expected losses, not by treating 0.75 as a universal confidence standard.

The general minimum-expected-loss rule follows directly from the definition of expected loss. Conditional on the observed evidence, each action has a real-valued expected cost. If the set of available actions is finite, at least one action attains the smallest of these values. Choosing an action with the smallest value minimizes expected loss conditional on that evidence, among the actions and loss values specified. This conclusion does not say that the chosen label is certainly correct, nor that the loss table is objectively correct; it is a result conditional on the model inputs.

Checking the Decision and Its Limits

Before applying the rule, check whether its assumptions match the annotation task. The candidate types should be mutually exclusive for the calculation and should include the plausible true type. The probabilities should refer to the evidence actually used for the decision. The available actions should match what can be recorded downstream, and the loss table should describe those actions' consequences. If labels or evidence are based on clustered data, dependence among cells and selection of marker evidence may matter; treating every cell as independent evidence can make uncertainty look smaller than it is.

Useful diagnostics include recomputing decisions under plausible alternative costs, checking how often actions change under small probability perturbations, and reviewing ambiguous or abstained cases against their underlying evidence. If tiny changes in probabilities or costs cause many labels to switch, report that instability rather than presenting the labels as settled. Also inspect whether specific labels are disproportionately assigned in regions with weak or conflicting marker evidence. These checks diagnose fragility; they do not establish that the probability inputs are calibrated or that the candidate set is complete.

Approach	What it uses	Main limitation
Always choose the largest probability	Relative probabilities of candidate types	Ignores unequal error costs and does not naturally express abstention.
Fixed probability cutoff	A prespecified threshold for issuing a specific label	May ignore type-specific costs and the loss of an ambiguous label.
Minimum expected loss	Probabilities, available actions, and an explicit loss table	Depends on defensible probabilities and costs; it cannot fix a misspecified candidate set.

A fixed cutoff can still be a practical policy, but it is not automatically equivalent to a decision-theoretic rule. In the simple threshold derivation, a cutoff follows from particular assumptions and a particular ratio of costs. With multiple types, asymmetric errors, ambiguous actions, or type-dependent abstention costs, the full loss comparison is needed.

A common mistake is to say, “A has the largest probability, so A is the correct annotation,” or to interpret a high marker-test significance as a probability that the label is correct. The correction is to distinguish evidence from action: use probabilities with a valid interpretation, specify the relevant actions and costs, and compare their expected losses. The result is a preferred action under those assumptions, not proof of the cell's true identity.

Key takeaway: A defensible annotation rule states what evidence informs the decision, which actions are available, and what each action costs when each candidate type is true. Specific labels, ambiguous labels, and abstention can then be compared on the same expected-loss scale.

Check Your Understanding

Use the stated probabilities and loss values. Losses are in hypothetical units.

If a label has probability 0.80, does that alone determine whether it should be assigned? State two other inputs needed for a minimum-expected-loss decision.
There are two types, A and B. The probability of A is 0.70; a wrong specific label costs 6, and abstention costs 1.5 regardless of the truth. Assuming correct labels cost 0, compare labeling A with abstaining.
Suppose $P(A\mid e)=0.60$ and $P(B\mid e)=0.40$. A wrong A label costs 2 and a wrong B label costs 5; correct labels cost 0. Which specific label has lower expected loss?
Why is an ambiguous label “A or B” not the same action as abstention?

Solutions. (1) No. The decision also requires an available-action set and a loss function; the evidence probabilities must also be justified. (2) Labeling A has expected loss $6(0.30)=1.80$, while abstention costs 1.50, so abstention is preferred between these two actions. (3) Labeling A has expected loss $2(0.40)=0.80$; labeling B has expected loss $5(0.60)=3.00$. A is preferred between the specific labels. (4) “A or B” reports a candidate set and may incur a cost for unresolved specificity or for excluding the true type. Abstention withholds the type assignment. Their meanings and costs can therefore differ.

Back to Tutorials