Probabilistic Classifiers for Cell-Type Labels

From Reference Labels to Class Probabilities

A cell-type classifier can return a single label, but that hides how the model arrived at its decision. A cell assigned “B cell” with probability 0.51 is a different case from one assigned “B cell” with probability 0.98, even though both hard labels are the same. This tutorial focuses on the probability vector behind that label and on two ways to estimate it: multinomial logistic regression and tree-based classifiers.

Suppose a reference data set contains \(n\) cells. For cell \(i\), let \(x_i\) denote its vector of measured features, such as expression values for selected genes or marker scores, and let \(Y_i\) denote its reference cell-type label. Assume the target labels are \(K\) mutually exclusive classes, numbered \(1,\ldots,K\). A classifier estimates \(P(Y_i=k\mid x_i)\) for each class \(k\). These are probabilities conditional on the features and the fitted model; they are not evidence that a label is biologically certain.

The probability vector is useful even when a downstream task ultimately requires one label. It preserves uncertainty among the candidate types, and makes it possible to distinguish a decisive prediction from a close competition. A probability is still a model output: its numerical value is most useful when the model's assumptions and training data are appropriate.

Multinomial Logistic Regression

Multinomial logistic regression models each class's log odds relative to a chosen baseline class. Take class \(K\) as the baseline. For each other class \(k\), define a linear score \(\eta_k(x)=\beta_{0k}+x^\mathsf{T}\beta_k\), where \(\beta_{0k}\) is an intercept and \(\beta_k\) is a vector of feature coefficients. The baseline score is set to zero. Applying the softmax transformation gives the class probabilities:

The probabilities are nonnegative and sum to one because their common denominator is the sum of the exponentiated scores, including the baseline's value \(\exp(0)=1\). For any nonbaseline class \(k\), the model implies \(\log\{P(Y=k\mid x)/P(Y=K\mid x)\}=\eta_k(x)\). Thus, holding the other features fixed, a one-unit increase in feature \(r\) changes the modeled log odds of class \(k\) versus the baseline by \(\beta_{rk}\). The corresponding odds are multiplied by \(\exp(\beta_{rk})\). This is a conditional association in the model, not a causal effect.

The model is usually fit by maximizing the conditional log likelihood of the reference labels. Let \(y_{ik}=1\) if cell \(i\) has class \(k\), and \(y_{ik}=0\) otherwise. One common objective is the log likelihood minus a penalty that discourages excessively large coefficients:

Here \(\lambda\geq 0\) controls the strength of an optional ridge penalty; the exact penalty and its tuning are modeling choices. It can stabilize estimates when features are numerous or correlated, but it does not make an unsuitable probability model correct.

Tree-Based Classifiers as Probability Models

A classification tree divides feature space through a sequence of rules, such as whether a marker score is above a threshold. Each terminal region, or leaf, contains the reference cells satisfying the rules along that path. A simple probability estimate for a new cell is the fraction of training cells in its leaf that belong to each class. A leaf containing only one observed class yields a probability of 1 for that class and 0 for the others, which can be an unstable estimate when the leaf is small.

One way to avoid zero estimates is to smooth the class counts. If a leaf contains \(n_k\) training cells of class \(k\), and the same positive pseudocount \(a\) is added to each of the \(K\) classes, the estimated probability is \((n_k+a)/(\sum_j n_j+Ka)\). This is a modeling choice, not a guarantee that the probabilities are well calibrated. In a random forest, many trees are fit using randomized training samples and/or candidate features; the forest's class probability is commonly the average of the trees' class-probability estimates. Averaging can reduce the instability of an individual tree, although it does not remove all model limitations.

Assumptions, Diagnostics, and Model Choice

Both approaches learn from labeled reference cells, so they require meaningful and consistently applied reference labels. The target classes should represent the categories to be predicted, and the features for new cells should be measured and processed in a way compatible with the training data. The fitted probabilities describe the training-defined task; if the reference omits a relevant cell type or labels are inconsistent, a classifier cannot correct that problem simply by returning probabilities.

Multinomial logistic regression assumes that the log odds of each nonbaseline class versus the baseline can be represented adequately by the specified linear predictor. The features themselves need not follow a normal distribution. Standard likelihood-based uncertainty calculations typically rely on independent observations and a suitable model; cells from the same donor may be correlated, so treating them as independent can overstate the effective information. Tree methods do not require linear log odds, but their estimates depend on how the feature space is partitioned. Deep trees can produce small leaves and unstable probabilities; shallow trees can miss useful structure.

Useful predictive checks include confirming that every output is between 0 and 1 and that the class probabilities sum to 1, then evaluating predictions on cells not used to fit the model. Log loss is one probability-sensitive summary: for \(n_{\text{test}}\) held-out cells with labels \(y_i\), it is the average negative log probability assigned to each observed label, \(-\sum_i\log\hat{p}_{y_i}(x_i)/n_{\text{test}}\); lower values are better when evaluated on the same test cases. A large difference between training and test performance suggests overfitting. Results that change markedly across reasonable fits, or predictions driven by a tiny leaf, also deserve caution. A large maximum probability alone does not establish that a probability estimate is trustworthy.

These methods are alternatives, not interchangeable probability mechanisms. Logistic regression is attractive when a roughly linear log-odds structure is plausible and a compact coefficient summary is useful. Trees are attractive when threshold-like or interaction patterns are expected. A forest can be more flexible than a single tree, at the cost of a less transparent decision structure. Comparing probability-sensitive performance on suitable held-out data is more informative than choosing a model solely because its hard labels look plausible.

Common Pitfall and Practical Takeaway

A frequent mistake is to report only the largest-probability class and then treat that label as certain. For example, “A was assigned” does not distinguish the vector \((0.51,0.30,0.19)\) from \((0.96,0.03,0.01)\). The correction is to retain the full probability vector when uncertainty matters and to describe it as the fitted model's estimate, not as a definitive biological fact. A second mistake is to interpret a logistic coefficient as a direct change in class probability: it changes log odds against the baseline, while the resulting probabilities depend on all class scores and features.

In practice, the probability model, reference labels, and evaluation design must be considered together. Probability outputs help communicate distinctions that hard labels suppress, but they remain subject to model misspecification, sampling variation, reference coverage, and differences between training and application data. A prediction should therefore be read as a quantified model assessment of the available evidence, not as an automatic resolution of biological ambiguity.