Abstention Decisions With a Reject Cost

Why Allow a Decision to Abstain?

A classifier is often required to name a class even when the available evidence is weak. In some applications, however, an uncertain prediction is more costly than declining to decide: a laboratory test may be sent for review, or a case may be referred to a specialist. The statistical question is when abstention is preferable to making the best available class prediction.

The answer depends on the relative costs. Under zero-one classification loss, a class prediction has conditional risk equal to its posterior probability of being wrong. Abstention has a fixed risk, the reject cost. Comparing these risks gives an exact rule: predict the class with the largest posterior probability only when its risk of error is no greater than the cost of rejecting.

Actions, Loss, and Conditional Risk

Let the response \(Y\) take one of \(K\) class labels, \(1,\ldots,K\), where \(K\) is finite, and let \(X\) denote the observed features. An action is either a class prediction \(j\in\{1,\ldots,K\}\) or a reject action, denoted by \(R\). Fix a reject cost \(c\) with \(0\leq c\leq 1\), on the same scale as the unit cost of misclassification.

Since the loss of predicting class \(j\) is one exactly when \(Y\ne j\), and the loss of rejection is always \(c\), the conditional risks are

The best class action is therefore any class with the largest posterior probability. Define \(\eta_{\max}(x)=\max_{1\leq k\leq K}\eta_k(x)\). The smallest conditional risk among class predictions is \(1-\eta_{\max}(x)\). Rejection is preferable precisely when its cost is smaller than this quantity.

The Bayes Rule and Its Threshold

The comparison in the theorem follows directly from the conditional risks, but the rule also minimizes the overall expected loss, not just a pointwise quantity considered in isolation. The proof below makes that connection explicit.

Here is the full justification of the last step and of the threshold characterization. Let \(g\) be any measurable decision rule taking values in \(\{1,\ldots,K,R\}\). Because the action set is finite and the losses are bounded, conditional expectation gives

For every \(x\), \(r(g(x)\mid x)\) is at least the smallest conditional risk among all actions. Select a rule \(g^*\) that attains this minimum: among the finitely many actions, choose one with smallest conditional risk, breaking ties by a fixed ordering. The posterior probabilities are measurable, so the risks of all actions are measurable; a fixed tie-breaking rule consequently makes \(g^*\) measurable. Taking expectations yields

Thus \(g^*\) is a Bayes rule. To identify which action minimizes risk, the smallest class risk is \(1-\eta_{\max}(x)\), while the reject risk is \(c\). The class action is better exactly when \(1-\eta_{\max}(x)<c\), which is equivalent to \(\eta_{\max}(x)>1-c\). Rejection is better exactly when the inequality is reversed, and equality gives a tie. This proves the result. \(\square\)

Worked Risk Calculation Across Feature Values

How the Reject Cost Changes the Rule

The threshold \(1-c\) expresses the trade-off in posterior-probability units. A larger reject cost lowers the threshold, so a class prediction becomes worthwhile even when its posterior probability is less concentrated. A smaller reject cost raises the threshold and makes abstention more attractive. Formally, if \(c_1<c_2\), then \(1-c_1>1-c_2\): every feature value at which the best class strictly beats rejection under \(c_1\) also has the best class beating rejection under \(c_2\), though additional values may qualify under \(c_2\).

The endpoints are consistent with the loss definition. If \(c=0\), rejection has zero risk, and it is optimal unless the most probable class has posterior probability one; at that boundary, predicting that class also has zero risk. If \(c=1\), every class prediction has risk at most one and, because the finite posterior probabilities sum to one, the largest is positive. Thus a maximizing class has risk strictly below one, and rejection is not optimal.

A useful interpretation is that the rule does not reject because a probability is “low” in isolation. It rejects when even the best class prediction has error risk exceeding the specified reject cost. If misclassification were assigned a cost other than one, the threshold would need to be recalculated on that cost scale; the stated threshold uses unit misclassification loss.

Common Mistake: Predicting the Most Probable Class Automatically

A frequent error is to find the largest posterior probability and always predict its class, reasoning that it is the most likely label. That identifies the best class action, but it does not compare that action with abstention. For example, with posterior probabilities \((0.55,0.45)\), the most probable class has conditional error risk \(1-0.55=0.45\). If \(c=0.20\), rejecting costs less, so prediction of the most probable class is not Bayes-optimal. The correction is to calculate both \(1-\eta_{\max}(x)\) and \(c\), then choose the smaller risk; include both actions if the risks are equal.

A second subtlety is that a posterior tie among classes and a tie with rejection are different. If two classes share the largest posterior, they have the same class risk, but rejection is selected only if its risk is lower; if the risks are equal, all tied minimizing actions are Bayes actions. The decision is determined by minimizing conditional risk, not by an arbitrary preference for one type of action.