Conformal Prediction Sets for Cell-Type Labels

Why Return a Set of Cell-Type Labels?

A classifier usually returns one label for a cell, even when several labels have similar support. A prediction set makes that ambiguity explicit: it includes labels that are not ruled out by a calibration rule. The aim is not to guarantee that every set is short, or that every cell is classified correctly. Rather, under a stated exchangeability condition, the procedure controls the long-run rate at which a set misses the true label.

Conformal prediction uses held-out labeled cells to determine how unusual a candidate true label would have looked to the fitted model. It can use probabilities from a probabilistic classifier, but its coverage result does not require those probabilities to be calibrated. It does require that the calibration cells and a future cell be exchangeable for the score being used.

Split Conformal Sets

In split conformal prediction, the labeled data are separated into a proper training set and a calibration set. Fit the classifier using only the proper training set, then hold that fitted classifier fixed. For each calibration cell \(i=1,\ldots,m\), with features \(X_i\) and known label \(Y_i\), compute the score \(S_i=A(X_i,Y_i)\). Sort the scores as \(S_{(1)}\leq\cdots\leq S_{(m)}\).

For a target miscoverage level \(\alpha\), where \(0<\alpha<1\), set \(k=\lceil(m+1)(1-\alpha)\rceil\). If \(k\leq m\), use \(q=S_{(k)}\). If \(k=m+1\), define \(q=+\infty\), so every label in \(\mathcal{Y}\) is included. For a new cell with features \(x\), the prediction set is:

The cutoff uses the \(k\)-th ordered calibration score, not an ordinary sample quantile with an unspecified interpolation rule. The \(m+1\) in the rank calculation is essential: the future cell is one more score in the exchangeable collection.

Exchangeability and the Coverage Result

The standard split conformal guarantee assumes that, conditional on the proper training data and the resulting fixed scoring rule, the \(m\) calibration examples and the future example are exchangeable. Exchangeability means their joint distribution is unchanged by permuting their order. Independent, identically distributed sampling is a familiar sufficient condition, but exchangeability is the condition used by the argument. In a single-cell setting, one must consider the sampling unit and dependence structure: cells from related or shared sources need not behave like exchangeable independent observations just because they are separate rows in a data table.

The true label must also be in \(\mathcal{Y}\), and the calibration and future examples must follow the same distribution for the relevant features and labels. The score function must be fixed without using the calibration labels to fit or tune the model. These conditions are distinct from probability calibration: even a poorly calibrated \(\hat p_y(x)\) can be used to rank candidate labels, and split conformal coverage can still hold if exchangeability and the data-splitting requirements hold.

Here is the rank argument. Conditional on the proper training data, the \(m\) calibration scores and the future true-label score are exchangeable. If scores have no ties, the rank of the future score among the \(m+1\) scores is equally likely to be any of \(1,\ldots,m+1\). When \(k\leq m\), a future score no larger than \(S_{(k)}\) has rank at most \(k\), so its probability is \(k/(m+1)\). Since \(k=\lceil(m+1)(1-\alpha)\rceil\), this probability is at least \(1-\alpha\). Ties, handled by including scores equal to \(q\), can only make the event of inclusion more frequent than the no-tie rank calculation. If \(k=m+1\), \(q=+\infty\) includes every candidate label, giving coverage one.

The guarantee concerns whether the true label is present, not whether the set is useful in every other sense. A set containing all labels has perfect coverage but may provide little annotation information. Set size is therefore an important companion summary, interpreted alongside coverage and the biological task.

Reading Set Size and Checking the Procedure

The size \(|\widehat C(x)|\) is the number of candidate labels retained for a cell. A singleton set gives one candidate under the rule; a larger set indicates that several candidates passed the threshold. An empty set is mathematically possible when every candidate score exceeds \(q\). It indicates that none of the available labels met the rule, not that the cell has no biological identity. If the true label is outside \(\mathcal{Y}\), conformal prediction over \(\mathcal{Y}\) cannot add it.

Several checks help reveal when a conformal analysis may be uninformative or invalid. Verify that the classifier and any score choices were fixed before calibration, and that the calibration examples were not also used to tune the threshold. Inspect the distribution of set sizes: uniformly huge sets may be valid but not useful, while many empty sets warrant attention to the candidate label universe, score definition, and compatibility between calibration and prediction data. In a labeled evaluation sample, report empirical coverage and set sizes together, but do not treat either as a guarantee for a different population.

A major failure mode is distribution shift. If future cells differ systematically from calibration cells, their true-label scores need not be exchangeable with the calibration scores, and the rank argument no longer establishes the stated coverage. Similarly, dependence among cells can invalidate an exchangeability claim based on treating every cell as a separate exchangeable draw. These are design and sampling concerns, not problems that can be repaired merely by choosing a different numerical quantile.

How Conformal Sets Differ From Other Uncertainty Summaries

A probability classifier assigns scores or probabilities to labels; a conformal set applies a calibration-derived threshold to candidate-label nonconformity scores. Entropy and margin summarize how concentrated or separated the model's predictions are, but do not by themselves provide the finite-sample marginal coverage statement described here. A selective classifier can abstain from issuing one label when uncertainty is high; a conformal rule instead returns a set whose size may vary from cell to cell. These summaries answer different questions and need not agree.

Conformal coverage is not class-conditional coverage. For example, a method can achieve its marginal target while missing a larger fraction of a rare cell type, provided coverage on other types compensates. Nor does coverage measure biological correctness beyond whether the provided true label is included. Interpret the target level, set size, label universe, sampling assumptions, and evaluation population together.

Common Mistakes

• Using the calibration data to fit the model and then treating the same scores as held out. This breaks the fixed-score setup used by the split conformal argument. Separate model fitting from calibration, or use a conformal method whose design explicitly accounts for data reuse.
• Calling 90% coverage a 90% probability for each cell or each label. The standard guarantee is marginal over exchangeable future examples. It does not automatically hold conditionally on a particular cell type, donor, or feature profile.
• Interpreting a large set as a probability distribution over its labels. Membership is determined by a score threshold; the set is not a normalized distribution and does not say that its labels are equally likely.
• Assuming conformal prediction discovers labels absent from the candidate universe. It can only include labels in \(\mathcal{Y}\). A missing true label cannot be recovered by thresholding scores for other labels.
• Ignoring dependence or distribution shift. A numerical coverage target does not protect against failure of the exchangeability condition.