Bayes Decisions Under General Multiclass Costs

Why the Most Probable Class May Not Be the Best Decision

A classification decision can have consequences that are not equally serious. Missing a dangerous condition may be much worse than ordering an unnecessary follow-up test; confusing two common categories may be less costly than confusing either with a rare, high-risk category. In such settings, choosing the class with the largest probability is not generally the right objective. The relevant question is: which available action has the lowest expected loss, given the class probabilities?

Let $Y$ be the true class, taking values in $\{1,\ldots,K\}$, and let $X$ denote the observed features. Let $\mathcal A=\{1,\ldots,M\}$ be a finite set of available actions. An action could be a predicted class, but the decision rule is defined by the loss: action $a$ incurs loss $L(a,j)$ if the true class is $j$. The entries $L(a,j)$ make up a loss matrix, with rows indexed by actions and columns by true classes.

Definition: Write $\eta_j(x)=P(Y=j\mid X=x)$, for $j=1,\ldots,K$, using measurable versions of these conditional probabilities. The conditional risk of action $a$ at feature value $x$ is $$ r(a\mid x)=E[L(a,Y)\mid X=x]=\sum_{j=1}^{K}L(a,j)\eta_j(x). $$ A Bayes action at $x$ is any action in $\arg\min_{a\in\mathcal A}r(a\mid x)$. A measurable rule $g$ that chooses a Bayes action for almost every $x$ is a Bayes decision rule.

For the finite-class setting here, a loss matrix specifies all the losses, and the conditional probabilities specify how likely each true class is at $x$. Multiplying each possible loss by the probability of its corresponding class and summing gives the conditional risk. The Bayes decision minimizes that weighted average. If the actions are the class labels and $L(a,j)=\mathbf 1\{a\ne j\}$, this reduces to choosing a most-probable class. General loss matrices need not give that result.

Worked Decisions Under Different Costs

Worked Example: A Cost-Sensitive Three-Class Decision

Suppose there are three classes and three corresponding prediction actions. The loss matrix and the class probabilities at a particular $x$ are $$ L= \begin{pmatrix} 0&4&4\\ 1&0&1\\ 2&2&0 \end{pmatrix}, \qquad \bigl(\eta_1(x),\eta_2(x),\eta_3(x)\bigr)=(0.45,0.35,0.20). $$ The most-probable class is class 1, with probability $0.45$. To determine the Bayes decision, calculate each row's conditional risk: $$ \begin{aligned} r(1\mid x)&=0(0.45)+4(0.35)+4(0.20)=1.40+0.80=2.20,\\ r(2\mid x)&=1(0.45)+0(0.35)+1(0.20)=0.45+0.20=0.65,\\ r(3\mid x)&=2(0.45)+2(0.35)+0(0.20)=0.90+0.70=1.60. \end{aligned} $$ The minimum is $0.65$, so the Bayes action is class 2, not class 1. Although class 1 is most probable, choosing action 1 has substantial losses when the truth is class 2 or 3. The loss matrix, rather than the probabilities alone, determines the optimal action.

Worked Example: A Two-Class Decision Threshold

Let the possible classes and actions be 0 and 1. Suppose a false positive costs 1 and a false negative costs 3, while correct decisions cost 0: $$ L= \begin{pmatrix} 0&3\\ 1&0 \end{pmatrix}. $$ At a feature value $x$, let $p=P(Y=1\mid X=x)=0.40$, so $P(Y=0\mid X=x)=0.60$. The two risks are $$ r(0\mid x)=0(0.60)+3(0.40)=1.20, \qquad r(1\mid x)=1(0.60)+0(0.40)=0.60. $$ Thus the Bayes action is 1, even though class 0 is more probable.

More generally, if the false-positive cost is $c_{\mathrm{FP}}>0$ and the false-negative cost is $c_{\mathrm{FN}}>0$, then $$ r(1\mid x)=c_{\mathrm{FP}}(1-p), \qquad r(0\mid x)=c_{\mathrm{FN}}p. $$ Action 1 is optimal exactly when $c_{\mathrm{FP}}(1-p)\leq c_{\mathrm{FN}}p$, or, after rearranging, $$ p\geq \frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}}+c_{\mathrm{FN}}}. $$ For the stated costs, this threshold is $1/(1+3)=0.25$, and $0.40\geq0.25$. This confirms the calculation: the higher cost of a false negative makes action 1 preferable at a probability below one half.

Worked Example: Tied Bayes Actions

Consider three classes with probabilities $(0.50,0.30,0.20)$ and the loss matrix $$ L= \begin{pmatrix} 0&1&2\\ 1&0&1\\ 2&1&0 \end{pmatrix}. $$ The conditional risks are $$ \begin{aligned} r(1\mid x)&=0(0.50)+1(0.30)+2(0.20)=0.70,\\ r(2\mid x)&=1(0.50)+0(0.30)+1(0.20)=0.70,\\ r(3\mid x)&=2(0.50)+1(0.30)+0(0.20)=1.30. \end{aligned} $$ Both actions 1 and 2 minimize conditional risk. They are both Bayes actions at this $x$; no calculation justifies preferring one on the basis of conditional risk alone. A rule may use either action at this point, provided it makes a measurable choice when decisions are required across feature values.

Why Pointwise Minimization Minimizes Overall Risk

The conditional calculation is useful because it also solves the overall decision problem. A decision rule $g$ maps features to actions, and its risk is $R(g)=E[L(g(X),Y)]$. The next result shows that minimizing conditional risk at each feature value minimizes this expected loss among all measurable rules.

Theorem: Suppose $Y$ takes values in a finite set, the action set $\mathcal A$ is finite, and $L(a,j)$ is a finite real number for every action $a$ and class $j$. Let $X$ take values in a measurable space, and let $\eta_j(X)=E[\mathbf 1\{Y=j\}\mid X]$ be measurable versions of the class probabilities. Then a measurable rule $g^*$ that minimizes $r(a\mid X)$ over $a\in\mathcal A$ almost surely minimizes $R(g)$ over all measurable decision rules $g$. Its risk is $$ R(g^*)=E\left[\min_{a\in\mathcal A}r(a\mid X)\right]. $$

Proof. Since the action set is finite, fix an ordering of its actions and choose the first minimizer of $r(a\mid X)$. Each conditional risk is measurable, being a finite sum of measurable functions. The event that a particular action is the first minimizer is described by finitely many measurable comparisons of conditional risks. The resulting rule $g^*$ is therefore measurable. The losses are bounded in absolute value because there are finitely many finite matrix entries, so the expectations below exist. For any measurable rule $g$, conditional expectation gives $$ R(g)=E\bigl[E[L(g(X),Y)\mid X]\bigr]=E[r(g(X)\mid X)]. $$ By the definition of $g^*$, $r(g^*(X)\mid X)\leq r(g(X)\mid X)$ almost surely. Taking expectations proves $R(g^*)\leq R(g)$. Also, $r(g^*(X)\mid X)=\min_{a\in\mathcal A}r(a\mid X)$ almost surely, which gives the displayed expression for $R(g^*)$. $\square$

The theorem also clarifies the role of ties. A tie means that multiple actions attain the same minimum conditional risk at some $x$. Any measurable choice among the minimizers has the same conditional risk there and thus does not increase overall risk. At feature values with probability zero under the distribution of $X$, the choice cannot affect $R(g)$.

Excess Risk and the Cost of a Non-Bayes Choice

The difference between a rule's risk and the Bayes risk can be expressed pointwise. This identity makes precise how a decision rule loses performance: at each $x$, it incurs the gap between the risk of its chosen action and the minimum available risk.

Theorem: Under the assumptions of the preceding theorem, let $g^*$ be any measurable Bayes decision rule. For every measurable rule $g$, $$ R(g)-R(g^*)= E\left[r(g(X)\mid X)-\min_{a\in\mathcal A}r(a\mid X)\right]\geq 0. $$

Proof. Applying the conditional-expectation identity from the preceding result to $g$ and $g^*$ yields $$ R(g)-R(g^*)= E\left[r(g(X)\mid X)-r(g^*(X)\mid X)\right]. $$ Because $g^*$ is a Bayes rule, $r(g^*(X)\mid X)=\min_{a\in\mathcal A}r(a\mid X)$ almost surely. Substitution gives the claimed equality. The integrand is nonnegative at every $X$ where the conditional risks are defined, since it is a value minus the minimum over all actions. Its expectation is therefore nonnegative. $\square$

The identity has a useful implication: a rule is Bayes-optimal in overall risk if and only if it chooses a conditional-risk minimizer almost surely. To see the reverse direction, if $R(g)-R(g^*)=0$, then the nonnegative integrand in the identity has expectation zero, so it equals zero almost surely. Thus $g(X)$ minimizes conditional risk almost surely. This statement concerns expected loss under the specified distribution of $X$; it does not require the rule to be optimal at feature values that never occur.

Common Mistakes and Practical Interpretation

Choosing the most probable class automatically. That is justified under zero-one loss, but not for a general loss matrix. Calculate the conditional risk for each action.
Mixing up rows and columns. Here rows index the chosen action and columns index the true class. The risk for action $a$ is the dot product of row $a$ with the class-probability vector.
Comparing only the loss for the most likely class. Conditional risk averages losses over every possible true class, weighted by their probabilities. Less probable classes can matter greatly when their associated losses are large.
Treating a tie as a unique answer. If multiple actions attain the same minimum, each is Bayes-optimal at that feature value. Additional preferences require a criterion beyond the specified conditional loss.

In applied work, the loss matrix encodes the decision-maker's priorities, not merely the statistical uncertainty. It should therefore be specified and interpreted carefully. Once the costs and conditional class probabilities are given, the decision calculation is direct: evaluate each action's conditional risk, choose a minimizer, and use the same logic across feature values.

Key takeaway: A Bayes decision minimizes conditional expected loss, not necessarily the probability of classification error. The most-probable-class rule is one special case; changing the costs can change the optimal action even when the class probabilities stay fixed.

Check Your Understanding

Use the loss matrix and conditional probabilities, rather than the largest class probability alone, to answer these questions.

For two classes, with false-positive cost 2 and false-negative cost 6, derive the probability threshold above which action 1 is optimal.
In a three-class problem, what does it mean for two actions to tie as Bayes actions at a feature value?
Why does a finite action set help ensure that a Bayes decision rule can be chosen measurably?
Explain why a class with a smaller conditional probability can still have a large effect on an action's conditional risk.
What does the excess-risk identity imply if a rule has the same overall risk as a Bayes rule?