Deriving Probability Calibration for Cell-Type Predictions

Why Calibration Is Not the Same as Getting Labels Right

A cell-type classifier can assign a label and also report probabilities for its possible labels. For example, it might assign a cell to the CD4 T-cell type with probability 0.8. Classification accuracy asks whether the assigned label matches the observed cell type. Calibration asks a different question: among predictions made with a stated probability, does the corresponding event occur at that frequency? A classifier can have good accuracy but systematically overstate or understate its probabilities.

This distinction matters when probabilities are used to assess how strongly to trust annotations, compare predictions across cells, or decide whether a probability is informative enough for a downstream analysis. The earlier material on entropy and margin measures of annotation uncertainty summarizes aspects of the probability vector; calibration concerns whether those probabilities agree with observed outcomes.

Definition: For cell $i$, let $Y_i$ be its observed reference cell type, and let $\hat p_{ik}$ be the classifier's predicted probability that the cell belongs to type $k$. The probabilities satisfy $\hat p_{ik}\geq 0$ and $\sum_{k=1}^{K}\hat p_{ik}=1$. The predicted label is the type with largest predicted probability, with a fixed rule used to break ties. A predictor is calibrated when the probability it reports agrees with the corresponding event frequency in the target population, under the relevant conditioning information.

Calibration is a property of predictions in relation to outcomes in a specified population, not a guarantee about an individual cell. A probability of 0.8 does not mean that exactly 80% of the time that particular cell's label is correct. It means that, among a sufficiently large and comparable set of predictions to which the calibration statement applies, the relevant event occurs about 80% of the time.

Three Calibration Statements

There are several related definitions. They differ in how much of the predicted probability information is used when forming groups of predictions. Let $\boldsymbol{\hat p}_i=(\hat p_{i1},\ldots,\hat p_{iK})$ denote the entire probability vector for cell $i$.

Definition: Full-vector calibration holds when, for every cell type $k$, the conditional probability of observing type $k$, given the full predicted vector, equals the corresponding predicted probability: $$ P(Y_i=k\mid \boldsymbol{\hat p}_i)=\hat p_{ik}. $$ Classwise calibration holds when, for every type $k$ and probability value $q$, predictions with $\hat p_{ik}=q$ have type-$k$ frequency $q$: $$ P(Y_i=k\mid \hat p_{ik}=q)=q. $$ Top-label calibration holds when predictions whose largest probability is $q$ are correct with frequency $q$. If $\hat k_i$ is the predicted label and $C_i=\max_k\hat p_{ik}$, this is $P(Y_i=\hat k_i\mid C_i=q)=q$.

Full-vector calibration is the strongest of these statements: it conditions on the complete probability vector. It implies classwise calibration, because predictions with $\hat p_{ik}=q$ can be averaged over their full vectors. Classwise calibration and top-label calibration each summarize a different aspect of the predictions; neither should be silently substituted for full-vector calibration. In particular, top-label calibration assesses correctness of the assigned label, not the calibration of every probability in the vector.

These are population statements. In an actual validation set, probabilities take many possible values and there may be few cells with exactly the same value. Calibration is therefore assessed approximately, using a finite set of observations and an explicit grouping or smoothing strategy. The result is evidence about calibration, not proof that a predictor is calibrated for every cell type, probability value, or future population.

Deriving the Link to Accuracy

For a labeled validation cell, define $Z_i=1$ if the predicted label is correct and $Z_i=0$ otherwise. The accuracy in a validation sample of $n$ cells is the mean of these indicators. By contrast, $C_i=\max_k\hat p_{ik}$ is the predicted confidence in the assigned label. Accuracy summarizes realized correct labels; confidence summarizes predicted probabilities.

Result: Under top-label calibration, average correctness equals average top-label confidence in the target population: $E[Z_i]=E[C_i]$. Full-vector calibration also implies this result. The equality is an average consequence of calibration, not a sufficient condition for calibration.

To see the result under full-vector calibration, condition first on the probability vector. With a fixed tie-breaking rule, the predicted label $\hat k_i$ is determined by that vector. Thus the conditional probability of a correct prediction is the probability of the selected class:

E[Z_i\mid\boldsymbol{\hat p}_i] =P(Y_i=\hat k_i\mid\boldsymbol{\hat p}_i) =\hat p_{i\hat k_i} =C_i.

Taking expectations of both sides gives $E[Z_i]=E[C_i]$. Under top-label calibration, the same conclusion follows by conditioning on $C_i$: $E[Z_i\mid C_i]=C_i$, and hence $E[Z_i]=E[C_i]$. But equal averages do not require equal conditional frequencies. Overprediction in one confidence range can cancel underprediction in another. The examples below show why average confidence alone cannot establish calibration.

Worked Example: A Group With 80% Confidence

Suppose a hypothetical validation set contains 10 cells predicted to be CD4 T cells, each with top-label confidence 0.8. Eight have the reference label CD4 T cell, and two have a different reference label. The observed accuracy for this group is

\frac{8}{10}=0.8=80\%.

The group's observed correctness frequency agrees with its stated confidence, so it is consistent with top-label calibration at 0.8. Its accuracy is also 80%. This finite group does not prove population calibration: another comparable set could have a different observed fraction because of sampling variability.

Worked Example: High Accuracy Does Not Ensure Calibration

Consider a different hypothetical group of 10 cells, each assigned a top-label probability of 0.9. Seven predictions are correct. The group's accuracy is

\frac{7}{10}=0.70=70\%.

The observed correctness frequency is 0.70, while the stated confidence is 0.90, a difference of 0.20, or 20 percentage points. The predictions are overconfident in this group. Their accuracy is still 70%; calibration identifies a mismatch that the accuracy number alone does not describe.

Worked Example: Equal Average Confidence Can Hide Miscalibration

Compare two hypothetical sets of 20 cells. In Set A, 10 cells have confidence 0.9 and 9 are correct; the other 10 have confidence 0.6 and 6 are correct. Its accuracy and mean confidence are

\text{accuracy}_A=\frac{9+6}{20}=0.75, \qquad \overline C_A=\frac{10(0.9)+10(0.6)}{20}=0.75.

Each confidence group also matches its observed frequency: $9/10=0.9$ and $6/10=0.6$. In Set B, again 10 cells have confidence 0.9 and 10 have confidence 0.6, but 8 and 7 cells, respectively, are correct. Its averages are

\text{accuracy}_B=\frac{8+7}{20}=0.75, \qquad \overline C_B=\frac{10(0.9)+10(0.6)}{20}=0.75.

Set B has the same accuracy and mean confidence as Set A, but its group frequencies are $8/10=0.8$ at confidence 0.9 and $7/10=0.7$ at confidence 0.6. It overstates confidence in the first group and understates it in the second. Agreement of the two overall averages is therefore not enough to demonstrate calibration.

What Must Be in Place to Assess Calibration?

Calibration requires predicted probabilities and outcomes that can be compared meaningfully. The following conditions define the scope of a useful assessment.

Usable reference labels. The observed cell types must be defined consistently with the label set used by the classifier. Ambiguous or inconsistent reference labels can make apparent prediction errors difficult to interpret.
A relevant target population. Validation cells should represent the population for which the calibration claim is intended. Calibration in one tissue, donor mix, or assay setting does not automatically transfer to another.
Probabilities on the stated label set. The output must refer to probabilities for the same set of possible labels, with nonnegative entries summing to one. A score or an unnormalized marker-evidence value is not a predicted probability merely because it lies between zero and one.
Enough observations for the question. Calibration is conditional on probability information, so sparse types or narrow probability ranges can leave too few validation cells to estimate observed frequencies reliably.

A simple empirical check compares predicted probabilities with observed frequencies in groups of similar predictions. This is an approximation to the population definitions, not a new definition of calibration. Grouping too broadly can conceal local over- and underprediction; grouping too narrowly produces unstable frequencies. Check the number of cells and the number of observed outcomes in each group, and examine results by cell type as well as overall. A pooled summary can conceal poor calibration for a rare type.

Cells from the same donor or sample may share biological and technical features. If the goal is inference about performance across donors, treating every cell as independent can make uncertainty appear smaller than it is. The calibration definition still refers to a target population, but the validation design and uncertainty calculation must respect the sampling structure.

Comparing Calibration With Other Performance Summaries

Summary	Question answered	What it does not establish
Classification accuracy	What fraction of assigned labels match the observed labels?	Whether reported probabilities agree with outcome frequencies
Top-label calibration	Among predictions with a given top probability, how often is the assigned label correct?	Whether every class probability is calibrated given the full vector
Classwise calibration	Among predictions assigning a given probability to type $k$, how often is type $k$ observed?	Whether the complete probability vector is calibrated
Full-vector calibration	Given the full predicted vector, do observed type frequencies match its entries?	Whether the assigned labels are accurate in every finite sample

Accuracy depends only on whether the largest-probability label is correct. Changing the numerical probabilities while leaving the predicted labels unchanged leaves accuracy unchanged, even though it can substantially alter calibration. Conversely, a calibrated classifier need not have high accuracy: it may assign modest probabilities that accurately reflect a difficult prediction task. Calibration and accuracy are complementary, not competing definitions of performance.

Common Pitfalls and Limitations

Equating average confidence with calibration. The wrong reasoning is: “Mean confidence is 0.75 and accuracy is 0.75, so the predictions are calibrated.” The correction is that calibration concerns conditional frequencies; different confidence groups can have offsetting errors.
Treating one validation fraction as a population truth. A group with 8 correct out of 10 has observed accuracy 0.8, but that estimate is uncertain. Small samples can depart from the population frequency by chance.
Calling accuracy a probability assessment. Accuracy reports the proportion correct in a sample. It does not say whether a reported probability of 0.6 or 0.9 is trustworthy.
Ignoring which calibration definition is used. A statement about top-label confidence does not establish that all class probabilities are calibrated given the full vector. Report the type of calibration claim being assessed.
Assuming calibration travels across settings. Changes in cell composition, reference labels, donors, or assay conditions can change the relationship between predictions and outcomes. A result on one validation population is not a universal guarantee.

Calibration is also limited by the quality and scope of the labels used as outcomes. If reference labels do not represent the biological distinctions of interest, agreement with them cannot establish that the predictions capture every biologically meaningful cell state. Calibration describes probability agreement relative to the specified outcome and population.

Check Your Understanding

Answer each question, then compare with the solutions below.

In a group of 20 cells with top-label confidence 0.7, 14 predictions are correct. What are the group's observed accuracy and confidence gap?
Why does equality of overall accuracy and mean top-label confidence not prove calibration?
Which is the more detailed condition: classwise calibration or full-vector calibration? State what information each conditions on.
A classifier's predicted labels remain unchanged, but its reported probabilities become more extreme. What happens to accuracy, and what may happen to calibration?

Solutions. (1) Accuracy is $14/20=0.70$, matching the stated confidence; the observed confidence gap is $0.70-0.70=0$. (2) Errors in different confidence groups can cancel in the overall averages, so conditional frequencies may still differ from stated probabilities. (3) Full-vector calibration is more detailed: it conditions on the entire probability vector; classwise calibration conditions on one type's probability. (4) Accuracy is unchanged because the assigned labels are unchanged, while calibration may improve or worsen depending on whether the more extreme probabilities better match observed frequencies.