Unequal Misclassification Costs In Gaussian Classification

Why Misclassification Costs Change the Rule

A classifier assigns an observation to a class, but not all incorrect assignments need have the same consequences. In a clinical setting, missing a disease and sending a healthy patient for further testing can carry very different costs. A rule that minimizes the probability of error treats those two mistakes symmetrically; a rule that minimizes expected loss does not.

The relevant calculation is made after observing the feature vector. For each possible class assignment, combine the losses it would incur with the posterior probabilities of the true classes. The optimal assignment is the one with the smallest resulting conditional expected loss, called its conditional risk.

Definition: Let $Y\in\{1,\ldots,K\}$ be the true class, and let $a\in\{1,\ldots,K\}$ denote the class assigned by a classifier. A loss matrix has entries $L(a,j)$, the loss incurred by assigning class $a$ when the true class is $j$. Given features $X=x$, the conditional risk of assignment $a$ is $$ R(a\mid x)=\sum_{j=1}^K L(a,j)P(Y=j\mid X=x). $$ A minimum-expected-cost, or Bayes, classifier chooses an assignment with the smallest conditional risk.

The diagonal entries $L(j,j)$ are often set to zero, but the definition does not require this. In what follows, the binary examples use zero loss for correct classifications and positive losses for the two different kinds of error.

Why Minimizing Conditional Risk Is Optimal

A classifier is a measurable rule $g$ that maps observed features $x$ to an assignment $g(x)$. Its expected loss, or risk, is $E[L(g(X),Y)]$. Minimizing conditional risk at each observation minimizes this overall expected loss.

Theorem: Suppose there are finitely many possible assignments, the losses $L(a,j)$ are finite, and the posterior probabilities $P(Y=j\mid X)$ are measurable. A measurable rule $g^*$ that selects a minimum-risk assignment at each $x$ minimizes expected loss among all measurable classifiers.

Proof. For any classifier $g$, conditioning on $X$ gives $$ E[L(g(X),Y)]=E\left[\sum_{j=1}^K L(g(X),j)P(Y=j\mid X)\right] =E[R(g(X)\mid X)]. $$ At every $x$ where the conditional probabilities are defined, the choice $g^*(x)$ satisfies $R(g^*(x)\mid x)\leq R(g(x)\mid x)$. Therefore $$ E[R(g^*(X)\mid X)]\leq E[R(g(X)\mid X)]. $$ A measurable minimum-risk rule exists: enumerate the finite assignments and, in a tie, choose the first assignment attaining the minimum. Its selection regions are defined by comparisons of measurable risk functions, so the rule is measurable. The inequality proves that its expected loss is no greater than that of any other measurable classifier. $\square$

Worked Example: Comparing Conditional Risks Directly

There are two classes, and the loss matrix has zero diagonal entries, loss $L(1,2)=4$ for assigning class 1 when class 2 is true, and loss $L(2,1)=1$ for assigning class 2 when class 1 is true. At an observed feature vector, suppose $P(Y=1\mid X=x)=0.75$. Which class minimizes conditional risk?

The posterior probability of class 2 is $1-0.75=0.25$. If we assign class 1, the only loss occurs when class 2 is true: $$ R(1\mid x)=4(0.25)+0(0.75)=1. $$ If we assign class 2, the only loss occurs when class 1 is true: $$ R(2\mid x)=1(0.75)+0(0.25)=0.75. $$ Since $0.75<1$, assign class 2. The most probable class is class 1, but choosing it would have the higher expected loss because its associated error is more costly.

The Binary Posterior and Likelihood-Ratio Thresholds

Write $c_{12}=L(1,2)>0$ for the cost of assigning class 1 when the true class is 2, and $c_{21}=L(2,1)>0$ for the cost of assigning class 2 when the true class is 1. Assume $L(1,1)=L(2,2)=0$. Let $p_1=P(Y=1\mid X=x)$, so $P(Y=2\mid X=x)=1-p_1$.

Theorem: Under these binary loss assumptions, assigning class 1 minimizes conditional risk exactly when $$ P(Y=1\mid X=x)\geq \frac{c_{12}}{c_{12}+c_{21}}. $$ Equality means the two assignments have equal conditional risk. If the class priors $\pi_1,\pi_2$ are positive and the class-conditional densities $f_1(x),f_2(x)$ are positive, this is equivalent to $$ \frac{f_1(x)}{f_2(x)}\geq \frac{c_{12}\pi_2}{c_{21}\pi_1}. $$

Proof. The two conditional risks are $$ R(1\mid x)=c_{12}(1-p_1),\qquad R(2\mid x)=c_{21}p_1. $$ Choose class 1 exactly when its risk is no larger: $$ c_{12}(1-p_1)\leq c_{21}p_1 \quad\Longleftrightarrow\quad c_{12}\leq (c_{12}+c_{21})p_1 \quad\Longleftrightarrow\quad p_1\geq \frac{c_{12}}{c_{12}+c_{21}}. $$ For the likelihood-ratio form, Bayes' rule gives $$ p_1=\frac{\pi_1f_1(x)}{\pi_1f_1(x)+\pi_2f_2(x)}. $$ The denominator is positive under the stated assumptions. Substitute this expression into the posterior threshold and multiply by the positive denominator: $$ \pi_1f_1(x)(c_{12}+c_{21}) \geq c_{12}\{\pi_1f_1(x)+\pi_2f_2(x)\}. $$ Cancel the common $c_{12}\pi_1f_1(x)$ term. The result is $c_{21}\pi_1f_1(x)\geq c_{12}\pi_2f_2(x)$, which is equivalent to the stated likelihood-ratio threshold. $\square$

The likelihood-ratio form makes the role of priors and costs explicit. Equivalently, the class-1 decision is made when $$ \log\{\pi_1f_1(x)\}-\log\{\pi_2f_2(x)\}\geq \log(c_{12}/c_{21}). $$ Thus the usual comparison of prior-weighted class densities has a cost-dependent cutoff. For equal error costs this cutoff is zero; when assigning class 1 in error is more costly, $c_{12}/c_{21}$ is larger and the evidence required for class 1 increases.

Worked Example: A Posterior Threshold for a Screening Decision

Suppose a positive classification corresponds to class 1, and a false positive costs $c_{12}=5$ while a false negative costs $c_{21}=1$. The posterior threshold for assigning class 1 is $$ \frac{c_{12}}{c_{12}+c_{21}}=\frac{5}{6}\approx 0.833. $$ At a patient's feature vector, suppose the estimated posterior probability of class 1 is $0.80$. Assigning class 1 has conditional risk $5(1-0.80)=1$; assigning class 2 has conditional risk $1(0.80)=0.80$. The minimum-cost decision is class 2, despite the substantial posterior probability of class 1. If instead the posterior probability were $0.90$, the risks would be $5(0.10)=0.50$ for class 1 and $1(0.90)=0.90$ for class 2, so class 1 would be selected.

Applying the Threshold to Gaussian Class Models

For two Gaussian classes in $d$ dimensions, let $X\mid Y=k\sim N(\mu_k,\Sigma_k)$, where $\Sigma_k$ is positive definite, and let the prior probability of class $k$ be $\pi_k>0$. The density is $$ f_k(x)=(2\pi)^{-d/2}|\Sigma_k|^{-1/2} \exp\left\{-\frac12(x-\mu_k)^\top\Sigma_k^{-1}(x-\mu_k)\right\}. $$ Inserting these densities into the binary likelihood-ratio threshold gives the minimum-cost Gaussian rule: $$ \log\frac{f_1(x)}{f_2(x)} \geq \log\frac{c_{12}\pi_2}{c_{21}\pi_1}. $$ The left side is the log-density ratio; the right side combines the prior odds and the relative error costs. The calculation uses the Gaussian class densities and the cost rule, without requiring the two covariance matrices to be equal. When the covariance matrices differ, the log-density ratio contains quadratic terms in $x$.

Worked Example: Equal-Variance Gaussian Classes

Suppose $X\mid Y=1\sim N(2,1)$, $X\mid Y=2\sim N(0,1)$, with priors $\pi_1=0.3$ and $\pi_2=0.7$. Let $c_{12}=5$ and $c_{21}=1$. Find the threshold for assigning class 1.

The likelihood-ratio cutoff is $$ \frac{c_{12}\pi_2}{c_{21}\pi_1} =\frac{5(0.7)}{1(0.3)} =\frac{35}{3}. $$ The two normal densities have the same variance, so their common normalizing constant cancels. Their log-density ratio is $$ \log\frac{f_1(x)}{f_2(x)} =-\frac12(x-2)^2+\frac12x^2 =-\frac12(x^2-4x+4)+\frac12x^2 =2x-2. $$ Assign class 1 exactly when $$ 2x-2\geq \log(35/3), \qquad x\geq 1+\frac12\log(35/3)\approx 2.228. $$ For example, at $x=2.3$, the log-density ratio is $2(2.3)-2=2.6$, while $\log(35/3)\approx2.457$, so class 1 is selected. At $x=2$, the log-density ratio is $2$, below the cutoff, so class 2 is selected. The cost of a false positive pushes the class-1 threshold above the equal-cost, equal-prior-free density comparison point.

Worked Example: Unequal Gaussian Variances and a Quadratic Decision Region

Let $X\mid Y=1\sim N(2,1)$ and $X\mid Y=2\sim N(0,4)$, with equal priors. Set $c_{12}=2$ and $c_{21}=1$. Determine the values of $x$ for which class 1 minimizes conditional risk.

The likelihood-ratio cutoff is $c_{12}\pi_2/(c_{21}\pi_1)=2$, so the log-density ratio must be at least $\log 2$. The standard deviation in class 2 is 2, making its density's normalizing constant half that of class 1. Hence $$ \log\frac{f_1(x)}{f_2(x)} =\log 2-\frac12(x-2)^2+\frac{x^2}{8}. $$ Subtracting $\log 2$ from both sides of the decision inequality leaves $$ -\frac12(x-2)^2+\frac{x^2}{8}\geq 0. $$ Expand and multiply by the positive number 8: $$ -\frac38x^2+2x-2\geq0 \quad\Longleftrightarrow\quad -3x^2+16x-16\geq0. $$ The corresponding equation has roots $$ x=\frac{16\pm\sqrt{16^2-4(3)(16)}}{6} =\frac{16\pm8}{6}, $$ namely $4/3$ and $4$. Since the quadratic $-3x^2+16x-16$ opens downward, it is nonnegative between its roots. Assign class 1 for $4/3\leq x\leq4$, and class 2 outside that interval. The more concentrated class-1 distribution does not imply that all large observations favor class 1: in the far tail, class 2's larger variance makes its density relatively larger.

Using Loss Matrices Carefully

With more than two classes, the same minimum-risk principle applies: calculate $R(a\mid x)$ for each possible assignment $a$, then choose one attaining the minimum. A single posterior threshold generally does not describe the rule when there are several possible actions and a general loss matrix. In the binary zero-diagonal case, however, the two risks reduce to a comparison of one posterior probability with one cost-dependent threshold.

The rule minimizes expected loss under the specified loss matrix and the specified class probabilities. It does not automatically optimize every practical objective. For example, if an application imposes a required sensitivity, that is an additional constraint rather than a consequence of minimizing the loss matrix's expected value.

Common Mistake

Using the wrong cost in the threshold. If $c_{12}$ is the cost of assigning class 1 when class 2 is true, it multiplies the posterior probability of class 2 in $R(1\mid x)$. It therefore appears in the numerator of the class-1 posterior threshold. Reversing the cost labels reverses the threshold.
Choosing the most probable class automatically. This minimizes the probability of error only when the two error costs are equal. With unequal costs, compare conditional risks or use the adjusted posterior threshold.
Changing a prior when the stated change is a cost. Priors describe how common the classes are before observing $X$; losses describe the consequences of decisions. Both affect the Gaussian rule, but they enter for different reasons.

Key takeaway: To account for unequal misclassification costs, write the loss matrix first, calculate each assignment's conditional risk, and choose the smallest. For two classes with zero correct-classification loss, assign class 1 when its posterior probability is at least $c_{12}/(c_{12}+c_{21})$, or equivalently when its likelihood ratio exceeds the prior- and cost-adjusted cutoff.

Check Your Understanding

Use the loss definitions carefully, and compare conditional risks before deciding.

For two classes with $c_{12}=3$ and $c_{21}=2$, what posterior probability threshold is required to assign class 1?
If $P(Y=1\mid X=x)=0.6$, $c_{12}=1$, and $c_{21}=4$, calculate both conditional risks and identify the minimum-cost assignment.
How does increasing $c_{12}$, while holding $c_{21}$ fixed, change the class-1 posterior threshold? Explain using the risk comparison.
For positive priors and densities, write the likelihood-ratio cutoff for assigning class 1 in terms of $c_{12},c_{21},\pi_1,\pi_2$.
Why can unequal Gaussian variances produce a class-1 decision region that is bounded rather than extending to all large values of $x$?