Reject Options and Abstention Costs

Why Allow a Classifier to Abstain?

A classifier is not always required to name a class. In a medical screening system, for example, an uncertain case might be referred for further review rather than assigned a diagnosis automatically. Abstention can reduce costly classification errors, but it is not free: it may require extra testing, delay a decision, or consume expert time. The decision therefore depends on comparing the cost of abstaining with the conditional cost of the best available class prediction.

We consider a fixed cost for abstaining and the usual zero-one classification loss: a correct class decision costs zero, and an incorrect one costs one. This makes the role of the rejection cost explicit. If the cost of a wrong decision were measured on another scale, the threshold would change accordingly.

Conditional Risk and the Rejection Threshold

The conditional risk of an action is its expected loss given the observed feature value. If the action is class \(j\), an error occurs exactly when \(Y\ne j\), so its conditional risk is \(1-\eta_j(x)\). The conditional risk of abstaining is \(c\), regardless of the unknown class.

Among class decisions, a class with the largest posterior probability has the smallest conditional risk. Write \(\eta_{\max}(x)=\max_{1\le k\le K}\eta_k(x)\). The best class decision has risk \(1-\eta_{\max}(x)\), so the comparison with abstention is between \(1-\eta_{\max}(x)\) and \(c\).

The threshold follows directly from the risk comparison:

Thus a common optimal rule is to predict a maximum-posterior class when \(\eta_{\max}(x)\ge 1-c\) and abstain otherwise. The weak inequality assigns threshold ties to classification, but assigning them to abstention is also optimal. If several classes share the maximum posterior probability, each is an equally good class action; a fixed tie-breaking convention, such as choosing the smallest index, makes the rule unambiguous.

Proof of the Conditional Decision Rule

The threshold is not a heuristic based on uncertainty; it is the result of minimizing conditional expected loss. The finite number of classes also ensures that the maximum posterior probability is attained.

For completeness, the conditional comparison also yields an optimal rule for the overall expected loss, not merely a good decision at an isolated feature value. Let \(d\) be a measurable rule taking values in \(\{1,\ldots,K,\bot\}\), and let \(L(d(X),Y)\) be its loss.

Proof. Define \(r(a,x)=E[L(a,Y)\mid X=x]\) for each action \(a\). A measurable minimizing rule exists: the conditional risks of the finite set of actions are measurable, and ties can be settled by a fixed ordering of the actions. Call such a rule \(d^*\). For any other measurable rule \(d\), pointwise minimization gives \(r(d^*(x),x)\le r(d(x),x)\) for almost every \(x\). By the tower property of conditional expectation and the fact that the loss is bounded,

Hence \(d^*\) minimizes expected loss. The boundedness needed for these expectations holds because the loss takes values in \([0,1]\). \(\square\)

Worked Decisions, Including a Threshold Tie

The threshold is governed by the largest posterior probability, not by the number of classes. For a fixed cost \(c\), the same threshold \(1-c\) applies for any finite \(K\); adding classes changes the posterior vector, but not the risk comparison.

A second useful calculation evaluates decisions across feature values. The pointwise rule chooses actions separately at each value of \(X\), while the total risk averages their conditional risks, weighted by how often those feature values occur.

Interpreting the Cost and Avoiding Common Errors

The cost \(c\) sets how much conditional classification error is worth avoiding by abstaining. A smaller \(c\) lowers the threshold \(1-c\), so abstention is chosen only when the best class probability is relatively low. A larger \(c\) raises the threshold, making abstention more attractive. This conclusion depends on the stated loss scale: the cost of an incorrect class prediction is one.

More generally, if every incorrect class prediction costs \(m>0\) and abstention costs \(r\ge0\), the best class risk is \(m(1-\eta_{\max}(x))\). For \(0\le r\le m\), comparing it with \(r\) gives the threshold \(\eta_{\max}(x)=1-r/m\). If \(r>m\), abstention is never better than the best class action, since \(m(1-\eta_{\max}(x))\le m<r\). This rescaling shows why a cost must be interpreted relative to the cost of an error.

With a general multiclass loss matrix, the best class action need not be the class with largest posterior probability. The relevant comparison is then between the minimum conditional risk over class actions and the rejection cost. The maximum-posterior threshold above relies on symmetric zero-one class loss; it should not be carried over unchanged to unequal error costs.

• Mistake: abstain whenever the largest posterior probability is below one half. That threshold is not universal. The correct threshold under the stated loss is \(1-c\), so it equals one half only when \(c=0.5\).
• Mistake: compare the rejection cost with the error probability of an arbitrary class. First choose a minimum-risk class, which under zero-one loss is a maximum-posterior class; compare \(c\) with \(1-\eta_{\max}(x)\).
• Mistake: treat equality as requiring abstention. At \(\eta_{\max}(x)=1-c\), a maximum-posterior class and abstention have equal risk. Either is optimal, and the implementation may choose a tie convention.
• Mistake: treat the threshold as a property of the number of classes. Under this loss, the threshold depends on \(c\), not directly on \(K\). The class count affects the posterior probabilities, not the algebra of the risk comparison.