Multiclass Decisions With General Cost Matrices

Why the Most Probable Class May Not Be the Best Decision

In multiclass classification, it is tempting to choose the class with the largest posterior probability. That is optimal under zero-one loss, where every incorrect classification has the same cost. But many decisions do not treat all errors equally: confusing two similar categories may be less costly than confusing two very different ones. In those settings, the most probable class need not be the action with the smallest expected cost.

The relevant quantity is the conditional risk of an action: its expected loss given the observed features. A general cost matrix lets us calculate this risk for every possible action and choose one with minimum risk. The probabilities still matter, but the costs determine how they should be combined.

Cost Matrices and Conditional Risk

Let the true class \(Y\) take values in \(\{1,\ldots,K\}\), and suppose the available actions are also labels \(1,\ldots,K\). Write \(C_{jk}\) for the cost of taking action \(j\) when the true class is \(k\). Thus, rows index actions and columns index true classes. The cost of a correct decision need not be zero, and costs for different errors need not be equal.

Each row of the cost matrix is combined with the posterior probability vector. In vector notation, if \(C_j=(C_{j1},\ldots,C_{jK})\) is row \(j\), then \(R(j\mid x)=C_j\cdot\eta(x)\). The choice is therefore not necessarily the row corresponding to the largest component of \(\eta(x)\).

We assume throughout that the number of classes and actions is finite and that all entries of \(C\) are finite. These conditions ensure that every displayed conditional risk is finite. When choosing among tied minimizers, any measurable tie-breaking rule gives the same conditional risk.

Why Minimizing Conditional Risk Gives a Bayes Rule

The calculation above is pointwise: it is made separately at each \(x\). The following result explains why these pointwise choices also minimize the overall expected loss.

Proof. For each action \(j\), let \(A_j=\{X:g(X)=j\}\). Because the action set is finite and \(g\) is measurable, each \(A_j\) is measurable. Conditional on \(X\), the cost of the chosen action is \(C_{g(X),Y}\). On the event \(A_j\), this is \(C_{jY}\). Using the definition of conditional probability for the finite-valued variable \(Y\), $$ E[C_{jY}\mid X]=\sum_{k=1}^{K}C_{jk}P(Y=k\mid X) =\sum_{k=1}^{K}C_{jk}\eta_k(X)=R(j\mid X). $$ Thus, on \(A_j\), \(E[C_{g(X),Y}\mid X]=R(j\mid X)\), and consequently $$ E[C_{g(X),Y}\mid X]=R(g(X)\mid X). $$ The costs are bounded because there are finitely many finite matrix entries, so these conditional expectations and their expectations exist. Taking expectations gives \(\mathcal R(g)=E[R(g(X)\mid X)]\).

Now suppose \(g^*\) minimizes conditional risk almost surely. For any other measurable rule \(g\), the pointwise minimizing property gives \(R(g^*(X)\mid X)\le R(g(X)\mid X)\) almost surely. Taking expectations and applying the identity just proved yields \(\mathcal R(g^*)\le\mathcal R(g)\). Therefore \(g^*\) is a population-risk minimizer. \(\square\)

With finitely many actions, one can choose the smallest-index action among the minimizers. Its selection is measurable because each conditional risk is measurable and comparisons among finitely many measurable real-valued functions define measurable sets. This provides a definite Bayes rule even when ties occur.

Comparing Two Actions Directly

Sometimes it is useful to compare two rows without calculating every risk in full. The difference is a weighted sum of the differences between their costs.

Derivation. By the definition of conditional risk, $$ R(a\mid x)-R(b\mid x) =\sum_{k=1}^{K}C_{ak}\eta_k(x)-\sum_{k=1}^{K}C_{bk}\eta_k(x) =\sum_{k=1}^{K}(C_{ak}-C_{bk})\eta_k(x). $$ Since \(a\) has no greater risk precisely when \(R(a\mid x)-R(b\mid x)\le 0\), the stated comparison follows. \(\square\)

This expression makes the trade-off explicit. If action \(a\) is cheaper than \(b\) for some true classes but more expensive for others, the posterior probabilities determine which differences matter most.

Cost Changes That Do Not Affect the Decision

Some changes to the cost matrix alter the numerical risks without changing which action is best. For example, adding the same constant to every entry in a row adds that constant to the risk of that action. Adding the same constant to every row in one column adds a common amount to every action’s risk, weighted by the posterior probability of that true class. The latter cannot change the minimizing action.

Proof. Substitute \(\widetilde C_{jk}=C_{jk}+d_k\) into the definition of conditional risk: $$ \widetilde R(j\mid x) =\sum_{k=1}^{K}(C_{jk}+d_k)\eta_k(x) =R(j\mid x)+\sum_{k=1}^{K}d_k\eta_k(x). $$ The second term does not depend on \(j\). For any two actions \(a,b\), it therefore cancels in \(\widetilde R(a\mid x)-\widetilde R(b\mid x)\). All pairwise comparisons, and hence the minimizers, are unchanged. \(\square\)

Common Mistakes

• Choosing the largest posterior automatically. That rule minimizes expected cost under zero-one loss, not under every cost matrix. Calculate the row-wise conditional risks for the loss actually specified.
• Reversing rows and columns. Here, row \(j\) describes the costs of taking action \(j\), while column \(k\) describes the true class \(k\). Check this convention before multiplying by the posterior probabilities.
• Using only the cost of the most likely class. Conditional risk averages costs over all possible true classes, weighted by their posterior probabilities. Even a less probable class can matter greatly if its associated cost is large.
• Treating a tie as an error. If several actions attain the same minimum conditional risk, each is a Bayes action. A tie-breaking convention chooses among them without changing the minimum risk.