Worked Binary Classification With Priors And Costs

Why Priors and Costs Both Matter

A classifier does more than decide which class makes an observation more likely. It must also account for how common each class is before observing the data and, when appropriate, how costly each kind of wrong decision is. These adjustments can change the chosen class even when one class has the larger posterior probability.

For a binary problem, label the classes 1 and 2. Write $\pi_k=P(Y=k)$ for the prior probability of class $k$, and $f_k(x)$ for the density of an observation $X=x$ under that class. The likelihood ratio $f_2(x)/f_1(x)$ compares how well the observation fits the two class models. Posterior odds also incorporate the prior odds $\pi_2/\pi_1$. A cost-sensitive decision then compares those posterior odds with a threshold determined by the two error costs.

Definition: Let $C_{a|k}$ be the cost of choosing class $a$ when the true class is $k$. Assume correct classifications have cost zero, so $C_{1|1}=C_{2|2}=0$, and assume the two misclassification costs $C_{1|2}$ and $C_{2|1}$ are positive. Given $X=x$, the conditional loss of choosing class $a$ is the expected cost under the posterior distribution of the true class.

From Posterior Odds to a Cost-Sensitive Rule

Bayes' rule gives the posterior odds in a useful product form. For observations where both class densities are positive,

\frac{P(Y=2\mid X=x)}{P(Y=1\mid X=x)} = \frac{f_2(x)}{f_1(x)}\frac{\pi_2}{\pi_1}.

The first factor is the likelihood ratio; the second is the prior odds. To choose a class under the stated costs, compare the conditional losses of the two possible decisions.

Theorem: Choose class 2 rather than class 1 exactly when $\frac{P(Y=2\mid X=x)}{P(Y=1\mid X=x)}>\frac{C_{2|1}}{C_{1|2}}$. Equivalently, choose class 2 exactly when $\frac{f_2(x)}{f_1(x)}>\frac{C_{2|1}\pi_1}{C_{1|2}\pi_2}$. At equality, the two decisions have equal conditional loss.

To prove the rule, abbreviate $p_k=P(Y=k\mid X=x)$. Choosing class 1 has conditional loss $C_{1|2}p_2$, because this decision incurs a cost only if class 2 is true. Choosing class 2 has conditional loss $C_{2|1}p_1$. Thus class 2 has smaller loss exactly when

C_{2|1}p_1<C_{1|2}p_2 \quad\Longleftrightarrow\quad \frac{p_2}{p_1}>\frac{C_{2|1}}{C_{1|2}}.

Substituting the posterior-odds identity gives the likelihood-ratio version. Replacing the strict inequality by equality gives the tie case. $\square$

The two thresholds answer different comparisons. The posterior-odds threshold compares the posterior odds directly with the cost ratio. The likelihood-ratio threshold also includes the prior odds, because the likelihood ratio alone does not reflect how prevalent the classes are.

The Threshold for Equal-Variance Gaussian Classes

Suppose the observation is one-dimensional and the class distributions are Gaussian with a shared variance: $X\mid Y=k\sim N(\mu_k,\sigma^2)$, where $\sigma^2>0$. If $\mu_2>\mu_1$, the log likelihood ratio increases with $x$, so the cost-sensitive rule has a single threshold: observations above it are assigned to class 2.

Result: For $\mu_2>\mu_1$, choose class 2 when $x>t$, where $$ t=\frac{\mu_1+\mu_2}{2} +\frac{\sigma^2}{\mu_2-\mu_1} \log\!\left(\frac{C_{2|1}\pi_1}{C_{1|2}\pi_2}\right). $$ Choose class 1 when $x<t$; at $x=t$, the conditional losses are equal.

For a complete derivation, the Gaussian density has the same normalizing constant in both classes, so it cancels in the ratio. Taking logarithms and expanding the squares gives

\begin{aligned} \log\frac{f_2(x)}{f_1(x)} &=\frac{(x-\mu_1)^2-(x-\mu_2)^2}{2\sigma^2}\\ &=\frac{\mu_2-\mu_1}{\sigma^2} \left(x-\frac{\mu_1+\mu_2}{2}\right). \end{aligned}

The cost-sensitive likelihood-ratio rule requires this log ratio to exceed $\log\!\left(\frac{C_{2|1}\pi_1}{C_{1|2}\pi_2}\right)$. Since $(\mu_2-\mu_1)/\sigma^2$ is positive, solving that inequality for $x$ gives the stated threshold and preserves the inequality direction. $\square$

The midpoint of the two means is the threshold only when the quantity inside the logarithm is 1. Unequal priors or unequal error costs generally move it. If $\mu_2<\mu_1$, the log likelihood ratio decreases with $x$, and solving the same inequality reverses the direction of the resulting one-sided decision.

Worked Calculations

Worked Example: A More Likely Class Is Not Always the Decision

Suppose $X\mid Y=1\sim N(0,1)$ and $X\mid Y=2\sim N(2,1)$, with priors $\pi_1=0.7$ and $\pi_2=0.3$. Let $C_{2|1}=4$ and $C_{1|2}=1$. Classify an observation $x=1.5$.

For these means and variance, the log likelihood ratio is $\log(f_2(x)/f_1(x))=2(x-1)$. At $x=1.5$, it equals $1$, so the likelihood ratio is $e$, approximately $2.718$. The posterior odds for class 2 over class 1 are therefore

\frac{P(Y=2\mid x)}{P(Y=1\mid x)} =e\frac{0.3}{0.7} \approx 1.165.

Thus $P(Y=2\mid x)=1.165/(1+1.165)\approx0.538$, and $P(Y=1\mid x)\approx0.462$. Class 2 has the larger posterior probability. But the cost-odds threshold is $C_{2|1}/C_{1|2}=4$, and $1.165<4$, so the cost-sensitive rule chooses class 1.

Checking the conditional losses confirms the decision: choosing class 1 costs $1(0.538)=0.538$ in posterior expected loss, while choosing class 2 costs $4(0.462)=1.848$. The less probable class is the lower-cost decision here.

Worked Example: Locating a Threshold Shifted by Priors and Costs

Let $X\mid Y=1\sim N(-1,4)$ and $X\mid Y=2\sim N(1,4)$, with $\pi_1=0.25$, $\pi_2=0.75$, $C_{2|1}=2$, and $C_{1|2}=5$. Find the threshold and classify $x=-4.2$ and $x=-3.8$.

The likelihood-ratio threshold is $\frac{C_{2|1}\pi_1}{C_{1|2}\pi_2} =\frac{2(0.25)}{5(0.75)}=\frac{2}{15}\approx0.1333$. Here $\mu_2-\mu_1=2$, the variance is 4, and the midpoint is 0. Hence

t=0+\frac{4}{2}\log\left(\frac{2}{15}\right) =2(-2.0149)\approx-4.0298.

Because $\mu_2>\mu_1$, choose class 2 above this threshold and class 1 below it. For a direct check, the log likelihood ratio is $x/2$. At $x=-4.2$, it is $-2.1$, giving a likelihood ratio of about $0.1225<0.1333$; choose class 1. At $x=-3.8$, it is $-1.9$, giving a likelihood ratio of about $0.1496>0.1333$; choose class 2. The threshold lies below the midpoint because class 2 has a larger prior and choosing class 1 when class 2 is true has the larger error cost.

Worked Example: Posterior Odds in a Two-Dimensional Gaussian Model

Let $X\in\mathbb{R}^2$. Suppose $X\mid Y=1\sim N_2((0,0),I)$ and $X\mid Y=2\sim N_2((2,1),I)$, where $I$ is the $2\times2$ identity matrix. Set $\pi_1=0.2$, $\pi_2=0.8$, $C_{2|1}=3$, and $C_{1|2}=2$. Classify $x=(1,2)$.

The shared covariance and Gaussian density formula give $\log(f_2(x)/f_1(x))=(2,1)^\mathsf{T}x-\frac12(2^2+1^2)$. At $x=(1,2)$, this is $2(1)+1(2)-2.5=1.5$. Thus the likelihood ratio is $e^{1.5}\approx4.4817$, and the posterior odds are

\frac{P(Y=2\mid x)}{P(Y=1\mid x)} =e^{1.5}\frac{0.8}{0.2} =4e^{1.5}\approx17.9268.

The posterior probability of class 2 is $17.9268/(1+17.9268)\approx0.9472$. The cost-odds threshold is $3/2=1.5$, so the rule chooses class 2. The conditional loss of choosing class 1 is $2(0.9472)\approx1.8944$; that of choosing class 2 is $3(1-0.9472)\approx0.1584$. The calculation verifies the decision using both posterior odds and conditional loss.

A Practical Decision Sequence and a Common Pitfall

Compute the likelihood ratio.
Compare the class densities at the observed value, or calculate their log ratio when that is simpler.

Include the prior odds.
Multiply the likelihood ratio by $\pi_2/\pi_1$ to obtain posterior odds for class 2 over class 1.

Apply the cost threshold.
Choose class 2 only when its posterior odds exceed $C_{2|1}/C_{1|2}$; equality gives a tie.

A common mistake is to compare the likelihood ratio directly with the cost ratio and stop. That omits the priors: the likelihood-ratio threshold is $\frac{C_{2|1}\pi_1}{C_{1|2}\pi_2}$, not merely $\frac{C_{2|1}}{C_{1|2}}$. The cost ratio alone is the threshold for posterior odds, after prior information has already been incorporated. Another mistake is to choose whichever class has posterior probability above one half automatically. That is valid only when the two misclassification costs are equal.

In practice, these calculations make explicit how prevalence and the consequences of different errors shape a classification boundary. In the one-dimensional equal-variance model, their combined effect is a shift from the midpoint between the means. In the multivariate shared-covariance model, the log likelihood ratio is linear in the observation, so changing the priors or costs changes the decision threshold for that linear score.

Check Your Understanding

Use posterior odds and conditional losses to justify each decision.

If the posterior odds for class 2 over class 1 are 3, and $C_{2|1}=6$, $C_{1|2}=2$, which class is chosen?
In the equal-variance Gaussian threshold formula, what happens to the threshold when $\pi_2$ increases while all other quantities stay fixed?
For the model in the first worked example, what posterior-odds value produces a tie between the two decisions?
Why is the likelihood-ratio threshold not generally equal to the cost ratio?
If $\mu_2<\mu_1$, does increasing $x$ make the likelihood ratio $f_2(x)/f_1(x)$ increase or decrease?