Sensitivity to Cell-Type Annotation Uncertainty

Why Annotation Uncertainty Matters

A differential-abundance analysis treats each cell as belonging to a cell type, but some cells may be difficult to classify. A hard-label analysis assigns each cell one label and then proceeds as if that assignment were certain. When many cells have ambiguous labels, this can make the estimated abundance contrast depend on arbitrary choices near classification boundaries. The practical question is not only how uncertain the labels are, but whether plausible alternatives change the direction or strength of the condition contrast.

This tutorial treats annotation uncertainty as a sensitivity problem: keep the abundance-analysis design and model fixed, vary the plausible cell labels, and compare the resulting condition contrasts. The earlier material on posterior predictive checks concerns whether a fitted compositional model can reproduce relevant features of the observed data; label sensitivity asks a different question about whether the abundance conclusion is robust to uncertain cell identities.

These probabilities are useful for sensitivity analysis only to the extent that they represent plausible uncertainty. They might come from a probabilistic classifier, repeated annotation by experts, or a specified set of alternative labels. They are not automatically calibrated probabilities that the label is correct. A model that reports \(q_{ik}=0.8\) should be checked, when possible, against independent reference annotations or other evidence. Annotation probabilities also do not replace the biological replicates: in condition comparisons, the sample or donor generally remains the unit of replication, not the individual cell.

From Cell-Level Probabilities to Abundance Contrasts

For sample \(j\), let \(n_j\) be the number of cells and let \(q_{ijk}\) be the probability that cell \(i\) in that sample belongs to type \(k\). The expected count for type \(k\), given these probabilities, is the sum of its cell-level probabilities. Dividing by the fixed total number of cells gives the expected proportion.

The resulting counts can be fractional. For example, a count of 3.4 means that the cell-level probabilities sum to 3.4; it does not mean that a sample contains a fractional cell. The vector of expected counts still sums to \(n_j\), because each cell’s probabilities sum to one. The expected proportions therefore sum to one as well.

One can use these expected proportions in a fixed sample-level abundance analysis, or repeatedly draw plausible labels and refit that analysis. The first approach uses the average of the cell-level probabilities. The second represents uncertainty about which labels were assigned: for each draw, assign a type to each cell according to its full probability vector, recalculate the sample counts and proportions, and refit the same model. For multicategory labels, a draw must use the probabilities for all types. The probability of a target type alone is not enough to infer the most probable label.

A sensitivity analysis should preserve the original experimental design and covariate adjustment. If the original analysis compares donor-level proportions while accounting for pairing or other sample-level structure, each reassignment should be analyzed with that same structure. Changing both the annotation and the analysis method at once makes it difficult to tell what caused a changed result.

Multicategory Labels Require the Full Probability Vector

With more than two cell types, a target probability below 0.50 can still be the largest probability for that cell. For instance, a vector \((0.45,0.35,0.20)\) gives the target type the largest probability even though that probability is less than one-half. A hard assignment based on the most probable type must compare all entries in the vector. A cutoff on the target probability is a different decision rule and should be identified as such.

Reassignment, Refitting, and Evidence

A contrast should be recalculated under each plausible annotation scenario. A scenario may be a specified alternative set of hard labels, a changed decision rule, or one draw of labels from the full probability vectors. For each scenario, recalculate sample-level counts or proportions and refit the same condition model. Record the contrast estimate and its uncertainty or test statistic using that model. The purpose is to see whether the substantive conclusion depends on annotation, not to select whichever scenario gives the preferred result.

For a simple illustration of test strength, suppose an analysis uses independent donor-level proportions in two conditions and a two-sided Welch \(t\)-test. This procedure assumes independent donors within and across conditions, and uses an approximate \(t\) reference distribution for the difference in sample means; with very few donors, that approximation and the resulting evidence can be unstable. In a real study, the analysis should reflect its design, including pairing or covariates where appropriate. The following calculation compares annotation scenarios under the same illustrative test, rather than recommending it as a universal abundance model.

Diagnostics, Alternatives, and Limitations

Useful diagnostics begin with the label probabilities themselves. Inspect how many cells have a clear leading label, how many are close competitors, and whether uncertain cells are concentrated in particular samples or conditions. If one condition has systematically more ambiguous annotations, a condition contrast can partly reflect differential classification rather than a biological abundance shift. When reference labels are available, assess whether reported probabilities are reasonably calibrated and whether errors differ across conditions. Also verify that all cell-type probability vectors sum to one and that the same cell universe and sample denominators are used in each scenario.

Compare approaches according to the question and assumptions they address:

Repeated draws can show whether an effect’s sign or magnitude is sensitive to annotation, but the spread across draws is not automatically a complete measure of inferential uncertainty. It reflects only the label uncertainty represented by the draw mechanism. It does not automatically include donor-to-donor sampling variation, classifier estimation error, or systematic misclassification. Do not treat the repeated draws as new independent biological samples, and do not average their \(p\)-values as if that produced a valid combined test. Inference should use a method that accounts for the relevant sources of uncertainty.

A further limitation is that probabilities may omit plausible alternatives or share errors across cells. If a classifier systematically confuses two types, independent cell-by-cell draws from its reported probabilities may understate uncertainty. Use coherent scenarios or a dependence model when that structure is important, and report which sources of uncertainty were varied. A sensitivity analysis cannot correct a biased annotation system merely by repeating its assignments.

A useful report states the original annotation rule, how uncertainty was represented, which sample-level model was held fixed, and how the contrast changed across scenarios. Emphasize whether the direction was stable, whether the estimated magnitude changed materially, and whether evidence weakened or strengthened. Robustness across the tested scenarios supports the conclusion only with respect to those scenarios; it does not establish that all annotation error has been ruled out.