Entropy and Margin Measures of Annotation Uncertainty

Why Summarize a Cell’s Label Probabilities?

A cell-type classifier can assign probabilities to several candidate labels, rather than returning only one label. For example, a cell might receive probabilities 0.45 for one type and 0.44 for another. Reporting only the most probable label hides how close the alternatives are. Two useful summaries are predictive entropy, which reflects the spread of probability across all candidate labels, and the top-two probability margin, which measures the lead of the most probable label over the runner-up.

These measures describe the classifier’s probability distribution for a cell. They do not, by themselves, establish whether the probabilities are accurate or whether the cell has a biologically ambiguous identity. In particular, interpreting a classifier’s probabilities requires care: a probability is a model output, and its empirical reliability is a separate question.

The probabilities must refer to the same candidate-label set and sum to one. If a classifier provides unnormalized scores, those scores cannot be used directly in these formulas; first obtain a probability distribution using the model’s specified procedure. The class-imbalance adjustment in the earlier material may change the probabilities, so calculate the summaries from the probabilities that are actually intended to guide the annotation.

What the Two Measures Capture

Entropy uses every label probability. It is low when most probability is concentrated on one label, and higher when probability is spread across several labels. For a fixed set of \(K\) labels, its minimum is 0, attained when one label has probability 1, and its maximum is \(\ln(K)\), attained when all labels have probability \(1/K\). Thus, entropy values depend in part on the number of candidate labels. A normalized version, \(H_i/\ln(K)\), ranges from 0 to 1 for \(K>1\), but it should be compared only when the label sets and their meanings are appropriately aligned.

The margin uses only the two largest probabilities. A large margin means the leading label is well ahead of the runner-up; a small margin means those two candidates are close. Its value lies between 0 and 1. A margin of 0 indicates a tie at the top, not necessarily maximum entropy: substantial probability might still be concentrated on the tied labels, or might be spread across many labels.

A large top probability alone is not the same as a large margin. If the leading probability is 0.60, a runner-up of 0.59 cannot occur in a normalized distribution, but a runner-up of 0.39 gives a margin of 0.21. Conversely, knowing the margin does not reveal whether the probability left over after the top two is concentrated on one label or dispersed among many. Entropy and margin are complementary summaries, not substitutes.

Why Entropy Has These Bounds

The bounds explain how to interpret an entropy value for a specified number of labels. Because every probability lies in \([0,1]\), each term \(-p_{ik}\ln(p_{ik})\) is nonnegative, with the zero-probability term defined by its limit as 0. Thus \(H_i\geq0\). If one probability is 1 and all the others are 0, every contribution is zero, so the minimum is attained.

For the upper bound, compare the distribution \(p_{ik}\) with the uniform probabilities \(u_k=1/K\). For every positive \(p_{ik}\), the inequality \(\ln(x)\leq x-1\), applied to \(x=u_k/p_{ik}\), gives \(p_{ik}\ln(u_k/p_{ik})\leq u_k-p_{ik}\). Summing over positive probabilities (with zero terms handled by a limit) yields \(\sum_k p_{ik}\ln(u_k/p_{ik})\leq0\), since \(\sum_k u_k=\sum_k p_{ik}=1\). Rearranging gives the entropy bound:

Equality in the upper bound requires the probabilities to be uniform across the \(K\) labels. For example, when there are four candidate labels, the maximum entropy is \(\ln(4)\approx1.386\) nats. An entropy of 1.0 nats means something different when there are two candidate labels than when there are ten; the range and label set provide essential context.

Worked Comparisons

Calculating and Checking the Measures

For each cell, the calculation can be organized as a short sequence. Keep enough precision during the calculation to avoid changing rankings or summary values through premature rounding.

Common warning signs include probabilities that do not sum to approximately 1, negative entries, or an entropy outside the bounds 0 to \(\ln(K)\). These often signal a calculation error, a failure to normalize scores, or an inconsistent label set. Rounding can make probabilities appear to sum to 0.99 or 1.01; where possible, calculate from unrounded values. Define \(0\ln(0)\) as 0 rather than attempting to evaluate \(\ln(0)\).

Comparison, Interpretation, and Limitations

A practical summary can report the predicted label, its probability, the margin, and entropy. This is more informative than presenting any one number without context. For example, low entropy and a large margin both suggest concentration on a leading prediction, while a small margin highlights a close competition among the top two. A high entropy with a moderate margin can occur when the remaining probability is broadly distributed.

These summaries have important limits. They do not prove that a prediction is correct, quantify the frequency of errors, or establish that a cell is biologically intermediate between types. Entropy and margin are functions of the supplied probability vector: if the model’s probabilities are poorly matched to observed outcomes, these measures may still be computed correctly but may not correspond well to actual prediction reliability. Likewise, comparisons across classifiers or datasets require attention to the candidate labels and probability-generation procedures. A larger entropy can reflect a genuinely broader model distribution, a different number of labels, or differences in the model outputs; the number alone does not identify the cause.

Common Mistakes

• Using only the predicted label. The top label suppresses information about competing labels. Report its probability or a complementary uncertainty summary when the probability distribution is relevant.
• Calling a small margin high entropy. A small margin indicates that the top two probabilities are close. It says nothing by itself about how the remaining probability is distributed.
• Calling zero margin maximum uncertainty. It means the top two probabilities tie. The distribution may still be far from uniform, as the worked example shows.
• Comparing entropy without checking \(K\). The maximum possible entropy is \(\ln(K)\), so values from different candidate-label sets need careful interpretation.
• Interpreting model uncertainty as biological ambiguity. The summaries describe the model’s probability vector, not a direct measurement of cell identity or a guarantee of prediction accuracy.