Selective Classification With an Abstain Option

Why Allow a Classifier to Abstain?

A cell-type classifier ordinarily assigns every cell one label, even when its evidence is weak or competing labels have similar support. An abstain option lets the procedure withhold a label for some cells rather than treating every prediction as equally trustworthy. This can be useful when downstream analyses need a set of relatively reliable labels, provided that the withheld cells are reported rather than silently discarded.

The decision is governed by a threshold on a confidence score. The key evaluation question is not simply whether a threshold is “high,” but what happens to coverage—the fraction of cells receiving a label—and to the selective error rate—the fraction of assigned labels that are wrong. These quantities describe a trade-off: stricter selection usually means fewer assigned labels, but it does not guarantee fewer errors among those retained.

Definition and Evaluation Setup

For cell \(i\), suppose a classifier supplies probabilities \(\hat p_{ik}\) for each candidate cell type \(k\). It predicts the label with the largest probability, denoted \(\hat Y_i\), and define its confidence score as the largest predicted probability, \(q_i=\max_k\hat p_{ik}\). For a threshold \(\tau\), the classifier assigns \(\hat Y_i\) if \(q_i\geq\tau\), and abstains otherwise. This tutorial focuses on that maximum-probability rule; the score must be defined consistently for all cells being evaluated.

Definition: Given \(n\) evaluated cells with reference labels \(Y_i\), the accepted set at threshold \(\tau\) is \(A_\tau=\{i:q_i\geq\tau\}\). Coverage is \(C(\tau)=|A_\tau|/n\). If \(|A_\tau|>0\), the selective error rate is \(R(\tau)=\sum_{i\in A_\tau} I(\hat Y_i\ne Y_i)/|A_\tau|\), where \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. Selective accuracy is \(1-R(\tau)\). If no cells are accepted, coverage is zero and selective error is undefined.

Evaluation requires cells with reference labels that can be compared with predictions. Ideally, the predictions are generated without using those evaluation labels to fit the classifier or choose its threshold. The evaluated cells should also represent the setting in which the annotation procedure will be used. If cells from the same donor are dependent, treating every cell as an independent observation can make uncertainty about performance appear smaller than it is; results should be examined at an appropriate biological sampling level as well.

A confidence score is a ranking or model output, not automatically a guarantee of correctness. Even if probabilities are well calibrated overall, a particular subset selected by a threshold need not have the exact error rate one would infer from its scores. The measured selective error on labeled evaluation data is therefore central to assessing the actual threshold.

Worked Example: Computing Coverage and Selective Error

Suppose a hypothetical validation set contains eight cells. Each row gives the predicted label, its confidence \(q_i\), and whether the predicted label matches the reference label.

Cell	Predicted label	Confidence	Correct?
1	T cell	0.96	Yes
2	B cell	0.91	Yes
3	Monocyte	0.84	No
4	T cell	0.81	Yes
5	B cell	0.77	No
6	Monocyte	0.72	Yes
7	T cell	0.66	No
8	B cell	0.60	Yes

At \(\tau=0.80\), cells 1–4 are accepted, so \(|A_\tau|=4\). Three of those four labels are correct, and one is wrong. Thus

C(0.80)=\frac{4}{8}=0.50,\qquad R(0.80)=\frac{1}{4}=0.25.

The procedure labels 50% of the validation cells and makes an error on 25% of the labeled cells in this set. It does not mean that 25% of all eight cells were mislabeled: one of eight cells was an accepted error, while four cells were abstentions.

How the Coverage–Error Curve Behaves

A threshold summarizes a particular operating choice. To compare choices, calculate coverage and selective error at a range of thresholds. With a fixed set of scores, only thresholds that change the accepted set produce a new result. Tied confidence scores enter or leave together, so the curve may have steps rather than changing smoothly.

Result: If \(\tau_2>\tau_1\), then \(A_{\tau_2}\subseteq A_{\tau_1}\), because any score at least \(\tau_2\) is also at least \(\tau_1\). Therefore \(C(\tau_2)\leq C(\tau_1)\). There is no corresponding general rule that \(R(\tau_2)\leq R(\tau_1)\): removing cells can increase, decrease, or leave unchanged the fraction of errors among those retained.

The set inclusion justifies the coverage result directly. Since both coverages use the same total \(n\), and the higher-threshold accepted set cannot be larger, its coverage cannot be larger. Selective error, by contrast, has a changing numerator and denominator. Removing correctly classified cells can raise the error fraction; removing incorrectly classified cells can lower it. Confidence is useful only insofar as it tends to rank correct predictions above incorrect ones in the evaluation setting.

Worked Example: A Higher Threshold Can Raise Error

Use the eight hypothetical observations in the table above. At threshold \(0.60\), all eight cells are accepted. There are three incorrect predictions, so coverage is \(8/8=1.00\) and selective error is \(3/8=0.375\). At threshold \(0.80\), the first four cells are accepted and one is incorrect, giving coverage \(4/8=0.50\) and error \(1/4=0.25\).

Now raise the threshold to \(0.85\). Only cells 1 and 2 are accepted, and both are correct, so coverage is \(2/8=0.25\) and error is \(0/2=0\). But at threshold \(0.84\), cells 1–3 are accepted: cell 3 is incorrect, so error is \(1/3\approx0.333\), greater than the error at threshold \(0.80\). The small increase from \(0.80\) to \(0.84\) removed a correct cell while retaining the incorrect one. Thus stricter selection reduced coverage but increased selective error in this comparison.

These values are empirical proportions for this hypothetical set, not a guarantee about future cells. In particular, zero observed errors among two accepted cells is weak evidence that the future error rate is zero.

Choosing a Threshold for a Stated Goal

Threshold selection should begin with the practical objective. One common rule is to choose the threshold giving the greatest coverage among those whose validation selective error is at or below a prespecified target. Another is to choose a threshold that produces a required minimum coverage, then report its observed error. These rules answer different questions; the chosen rule and the resulting coverage and error should both be reported.

The validation set is used to compare candidate thresholds. If the same labels are repeatedly used to select the best-looking threshold and to claim its performance, the reported error may be optimistic: the selection process favors thresholds whose observed errors happened to be low. A separate test set, or a validation design that accounts for threshold selection, is needed for a less biased final performance assessment. The target should be set for the application, not chosen after inspecting the most favorable result.

Worked Example: Meeting an Error Target While Retaining Cells

Suppose a hypothetical labeled validation set has 20 cells. The sorted confidence scores and correctness outcomes show that at threshold \(0.70\), 12 cells are accepted and 3 are incorrect; at \(0.80\), 8 are accepted and 1 is incorrect; and at \(0.90\), 4 are accepted and none are incorrect. Assume the prespecified goal is an observed selective error no greater than 0.15, and the rule is to maximize validation coverage among the listed thresholds meeting that goal.

At \(0.70\), \(R=3/12=0.25\), which exceeds 0.15, so this threshold does not meet the goal. At \(0.80\), \(R=1/8=0.125\), which is at most 0.15; coverage is \(8/20=0.40\). At \(0.90\), \(R=0/4=0\), and coverage is \(4/20=0.20\). The \(0.80\) threshold has the greater coverage among the listed thresholds that meet the observed-error target, so it is selected by the stated rule.

The conclusion is limited: it says that the observed validation error at \(0.80\) was 12.5%, not that the population error is known to be at most 15%. Only eight cells contribute to that estimate, and the threshold was evaluated on a finite sample. The selected threshold should be assessed on independent data before its performance is treated as established.

Diagnostics, Alternatives, and Limitations

A useful report gives several points on the coverage–error curve, not just one threshold. Check the number of accepted cells alongside every error estimate, because a low error based on very few cells is unstable. Also inspect whether the accepted cells are concentrated in particular cell types, confidence ranges, or other relevant groups. An acceptable overall error can hide poor performance in a subgroup. Reliability diagrams, covered in the earlier material on probability calibration and reliability diagrams, help assess probability calibration; they do not replace direct evaluation of selective error at the chosen threshold.

The maximum-probability threshold is not the only possible abstention rule. A rule based on the gap between the two largest class probabilities, for example, asks whether the leading label is sufficiently separated from its competitor, rather than whether its probability alone is large. Such scores can select different cells and must be evaluated with the same coverage and selective-error calculations. Class-specific thresholds are another option when operating requirements differ by label, but they require enough evaluation data per class and make the overall decision rule more complex.

A threshold cannot repair a poorly specified candidate label set or establish that every biological cell type is represented among the available labels. Nor can it make predictions reliable under a changed data-generating setting simply by being numerically high. The abstain rule addresses uncertainty in assigning a supplied label; its effectiveness depends on the score’s relationship to correctness in the application setting. Report the threshold, score definition, evaluation sample, coverage, error, and abstention count so that the trade-off is interpretable.

Mistake: “A higher threshold must lower the error rate.” It must not increase coverage, but selective error can rise if the newly excluded cells were correct and some incorrect cells remain.
Mistake: “Zero observed errors means the threshold is error-free.” Zero errors in a small accepted set is an observed result, not proof of zero future risk. Report the accepted sample size and validate independently.
Mistake: “The score is a probability, so the accepted fraction is automatically reliable.” A score alone does not establish performance. Compare predictions with reference labels and measure error among the accepted cells.

Define the rule.
Specify the confidence score, label prediction rule, and whether a score equal to the threshold is accepted.

Evaluate candidate thresholds.
On labeled evaluation data, calculate accepted count, coverage, and selective error at each threshold.

Choose and report transparently.
Use a prespecified application goal to select a threshold, then report the trade-off and assess the selected rule on data not used to tune it.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In a validation set of 50 cells, a threshold accepts 35 and 7 accepted predictions are wrong. Calculate coverage and selective error.
Explain why coverage cannot increase when the threshold is raised, while selective error can increase.
At a threshold, no cells are accepted. What is coverage, and is selective error defined?
Why can evaluating many thresholds on one validation set and reporting only the best observed error be misleading?

Solutions. 1. Coverage is \(35/50=0.70\), or 70%. Selective error is \(7/35=0.20\), or 20% of assigned labels. 2. A higher threshold creates a subset of the previously accepted cells, so the accepted count cannot grow. Error is a ratio among accepted cells; excluding correct cells while retaining incorrect ones can increase that ratio. 3. Coverage is 0. Selective error is undefined because its denominator, the number accepted, is zero. 4. Searching many thresholds can select one whose observed error is unusually low by chance. Its performance should be checked on independent data or with a validation design that accounts for threshold selection.