Separating Model Uncertainty From Cell Ambiguity

Two Different Sources of an Uncertain Label

A cell can receive an uncertain annotation for more than one reason. Plausible fitted models may disagree about its label, or different plausible labels may remain likely even when the fitted model changes very little. Those patterns call for different interpretations. A single probability vector can show that the prediction is uncertain overall, but cannot by itself distinguish these sources.

An ensemble makes the distinction visible by producing a probability vector for the same cell from each of several fitted models or resampled references. Variation across those vectors is evidence of model instability under the ensemble procedure. Uncertainty that remains within the vectors, even when they are similar to each other, is evidence of persistent ambiguity in the cell’s predictive evidence. Neither pattern alone establishes the biological cause of uncertainty.

The calculations below assume that all ensemble members predict the same cell against the same harmonized set of \(K\) labels. Probabilities must use the same label definitions and order, and each vector must sum to one. This comparability matters: disagreement caused only by differently named or defined labels is not model uncertainty about a common outcome.

The mean vector \(\bar{\mathbf p}_i\) summarizes the ensemble’s average probabilities. Its entropy is the cell’s total predictive entropy under that ensemble. The within-model component is large when individual models themselves spread probability across labels. The between-model component is large when model predictions differ from one another. A large between-model component can occur even when each model is individually confident.

The term “within-model” describes a property of the predicted probability vector, not a direct observation of a cell’s irreducible biological uncertainty. In particular, model probabilities should not be read as calibrated chances of the cell’s true type unless calibration and the relevant validation conditions support that interpretation. The entropy decomposition is an exact mathematical description of the ensemble predictions; its biological meaning depends on how the models and reference resamples were constructed.

Why the Entropy Decomposition Works

Imagine first selecting an ensemble member \(M\) at random, then drawing a predicted label \(Y\) according to that member’s probabilities for the cell. With equal ensemble weights, \(P(M=m)=1/M\), and \(P(Y=k\mid M=m)=p_{imk}\). The marginal probability of label \(k\) is therefore \(P(Y=k)=\bar p_{ik}\).

The entropy chain rule states that the uncertainty in \(Y\) can be written as uncertainty in \(Y\) given the selected model, plus the information about \(Y\) supplied by knowing the model. In this construction, the first term is average within-model entropy. The second is the between-model component:

Here \(H(Y)=H(\bar{\mathbf p}_i)\), \(H(Y\mid M)=M^{-1}\sum_m H(\mathbf p_{im})\), and \(I(Y;M)\) is the mutual information between the ensemble-member index and predicted label. Thus \(U_{\mathrm{between},i}=I(Y;M)\). This also shows why the between-model component cannot be negative: mutual information is nonnegative. It is zero when knowing which ensemble member was selected does not change the predictive distribution of the label.

The equality follows by expanding the joint entropy in two ways. Because \(P(M=m,Y=k)=M^{-1}p_{imk}\), the chain rule gives \(H(Y,M)=H(M)+H(Y\mid M)\). Reversing the order gives \(H(Y,M)=H(Y)+H(M\mid Y)\). Subtracting the second expression from the first yields \(H(Y)-H(Y\mid M)=H(M)-H(M\mid Y)=I(Y;M)\). The between-model quantity is also called the Jensen–Shannon divergence of the ensemble’s probability vectors from their mean, with equal weights.

If ensemble members have justified unequal weights \(w_m\), use \(w_m\geq0\), \(\sum_m w_m=1\), and replace each average over models with a weighted average. The decomposition still holds when model selection is represented by those weights. Arbitrary weights, however, do not automatically make the ensemble more representative of plausible fitted models.

Comparing the Components in Practice

The decomposition separates patterns that a single uncertainty score merges. For two labels, consider these two hypothetical ensembles. In the first, every model predicts \((0.50,0.50)\). In the second, half predict \((1,0)\) and half predict \((0,1)\). Both have the same mean prediction, \((0.50,0.50)\), and total entropy of 1 bit. But their components differ:

In the first pattern, uncertainty is present within each model’s predictions. In the second, uncertainty in the ensemble mean arises entirely because models disagree. An entropy or margin calculated only from the ensemble mean cannot distinguish them. Comparing it with average within-model entropy can.

A second useful diagnostic is the spread of each label’s predicted probability across models. For example, inspect the range or standard deviation of \(p_{imk}\) across \(m\), alongside the entropy decomposition. These summaries can reveal whether a large between-model component is driven by one unusual fit or is shared across many models. They do not replace the decomposition, and their scales differ from entropy. Report the ensemble construction and the number and type of fitted models or resamples so the source of variation is interpretable.

Assumptions, Diagnostics, and Failure Modes

The arithmetic decomposition requires probability vectors over a common set of labels and a defined weighting of ensemble members. Interpreting its components as a useful diagnostic additionally requires that the ensemble variation be meaningful. Members should represent plausible alternatives under the modeling or resampling question, rather than mostly duplicated fits or arbitrary perturbations. If predictions are badly calibrated, entropy remains a valid summary of the reported numbers but may not measure uncertainty about true labels well.

Inspect the individual prediction vectors as well as their summaries. A very large between-model component may be driven by one failed fit, a small class absent from a resample, unstable optimization, or inconsistent preprocessing. Verify that model fits converged, labels were aligned correctly, and the cell was processed consistently. If resampling references, specify the resampling unit and scheme: resampling individual cells when the intended uncertainty concerns independent donors, for example, may not represent the relevant source of variation.

An ensemble with low between-model disagreement is not proof of a reliable answer. All members can share the same misspecification, reference limitation, or uninformative evidence. Conversely, high disagreement does not prove that the cell is biologically ambiguous: it may reflect a fragile model fit or a poorly chosen ensemble. Calibration, held-out validation, and other evidence can inform interpretation, but this diagnostic does not replace them.

There is also a conceptual limit. Predictive entropy within a model is sometimes informally called “cell ambiguity” or “aleatoric uncertainty.” It is safer to call it within-model predictive ambiguity: without additional assumptions and evidence, it cannot establish irreducible biological ambiguity in the cell. A cell at a transitional state is one possible explanation, but the probabilities alone do not distinguish that from weak measurements, imperfect labels, or model limitations.

Common Mistake and Main Alternatives

Wrong reasoning: “The ensemble mean assigns 50% to each of two labels, so every model is uncertain about this cell.” That conclusion does not follow. The mean can arise from every model assigning \((0.50,0.50)\), or from half the models confidently assigning one label and half confidently assigning the other. Both have the same mean entropy; their within-model and between-model components are different.

The correction is to retain the member-level probability vectors and compare the entropy of their mean with their average entropy. A hard-label vote can also indicate disagreement, but it discards how confident each member is: predictions of \((0.51,0.49)\) and \((0.99,0.01)\) both vote for the first label. The entropy decomposition uses the full probabilities.

Other uncertainty summaries answer related but different questions. Entropy and margin applied to one probability vector summarize uncertainty within that vector, as covered in the earlier material on entropy and margin measures. Calibration checks whether predicted probabilities agree with observed frequencies in appropriate validation data; it does not itself separate ensemble disagreement from within-model entropy. Conformal prediction sets address set-valued coverage under their assumptions, rather than decomposing ensemble entropy. These approaches can complement the present diagnostic, but none makes its interpretation automatic.