Bayes Decision Rules For Gaussian Class Models

From Gaussian Models to Classification Decisions

A Gaussian class model describes how measurements vary within each class, but it does not by itself specify what decision to make for a new observation. A decision also depends on how common the classes are before the measurement is seen. Bayes decision theory combines these ingredients: the class prior probabilities and the class-conditional densities.

This tutorial considers the zero-one loss: predicting the wrong class incurs loss one, and predicting the right class incurs loss zero. Under this loss, the optimal rule assigns an observation to a class with the largest posterior probability. The Gaussian assumption makes the required densities explicit; it does not change the decision-theoretic argument.

Definition: Let $Y$ be a class label taking values in $\{1,\ldots,K\}$, and let $X\in\mathbb{R}^p$ be the observed feature vector. Write $\pi_k=P(Y=k)$ for the prior probability of class $k$, with $\pi_k\geq 0$ and $\sum_{k=1}^K\pi_k=1$. Suppose $X\mid(Y=k)$ has density $f_k$. A Gaussian class model specifies $X\mid(Y=k)\sim N_p(\mu_k,\Sigma_k)$, where $\mu_k\in\mathbb{R}^p$ and $\Sigma_k$ is positive definite. Its density at $x$ is $$ f_k(x)=\frac{1}{(2\pi)^{p/2}|\Sigma_k|^{1/2}} \exp\left\{-\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)\right\}. $$

Positive definiteness ensures that $|\Sigma_k|>0$, the inverse $\Sigma_k^{-1}$ exists, and this expression is a valid density with respect to $p$-dimensional Lebesgue measure. The posterior probability $P(Y=k\mid X=x)$ describes class membership after observing the feature value $x$. For a point $x$ at which the marginal density is positive, Bayes' theorem gives

P(Y=k\mid X=x) = \frac{\pi_k f_k(x)} {\sum_{j=1}^{K}\pi_j f_j(x)}.

The numerator is the prior probability of class $k$ multiplied by the density of the observation under that class. The denominator adds these quantities over all classes. Thus, classification is not determined by which class density is largest alone: the prior probabilities matter as well.

Why the Posterior Maximizer Is the Bayes Rule

A decision rule $\delta$ assigns a class to each observed feature vector. We consider measurable rules $\delta:\mathbb{R}^p\to\{1,\ldots,K\}$, so the probabilities of their decisions are well-defined. Under zero-one loss, the loss for deciding $a$ when the true class is $y$ is $L(a,y)=\mathbf{1}\{a\ne y\}$.

Definition: The conditional risk of deciding class $a$ at $x$ is the posterior expected loss $$ R(a\mid x)=E[L(a,Y)\mid X=x]=1-P(Y=a\mid X=x). $$ A Bayes decision at $x$ minimizes this conditional risk over the available classes.

The equality follows because the loss is zero when $Y=a$ and one otherwise. The conditional risk is therefore the posterior probability of being wrong. This yields the first result directly.

Theorem: Under zero-one loss, any class maximizing $P(Y=k\mid X=x)$ minimizes the conditional risk at $x$. If the marginal density at $x$ is positive, this is equivalent to maximizing $\pi_k f_k(x)$. If several classes attain the maximum, each is a Bayes decision.

Proof. For each possible decision $a$,

R(a\mid x)=1-P(Y=a\mid X=x).

The value one is the same for every candidate class. Consequently, minimizing $R(a\mid x)$ is equivalent to maximizing $P(Y=a\mid X=x)$. By Bayes' theorem, the denominator $\sum_j\pi_j f_j(x)$ is also the same and is positive for all candidate classes at this $x$. Maximizing the posterior is therefore equivalent to maximizing its numerator, $\pi_a f_a(x)$. Any tie gives equal conditional risk, so every maximizing class is optimal. $\square$

The pointwise result also establishes optimality over whole-sample decision rules, not merely at one observation.

Theorem: Suppose there are finitely many classes and measurable class-conditional densities $f_k$, with priors $\pi_k$. A measurable rule that selects a maximizing class for $\pi_k f_k(x)$ wherever the marginal density is positive minimizes the overall probability of misclassification among all measurable rules.

Proof. Let $m(x)=\sum_{k=1}^K\pi_k f_k(x)$ be the marginal density of $X$. For any measurable rule $\delta$, the probability of a correct decision is

P\{\delta(X)=Y\} = \sum_{k=1}^K \int_{\{x:\delta(x)=k\}} \pi_k f_k(x)\,dx = \int_{\mathbb{R}^p}\pi_{\delta(x)} f_{\delta(x)}(x)\,dx.

The finite sum can be written as this integral because the decision regions partition the feature space; the integrands are nonnegative and integrable. At every $x$, a maximizing rule has $\pi_{\delta(x)}f_{\delta(x)}(x)\geq \pi_a f_a(x)$ for every alternative class $a$. Integrating this pointwise inequality shows that its probability of a correct decision is at least that of any other rule. Since the probability of error is one minus the probability of a correct decision, the maximizing rule minimizes error. For finitely many classes, choosing the smallest maximizing class index makes the rule measurable: each decision region is specified by finitely many comparisons of measurable functions. Values on a set where $m(x)=0$ have no effect on the error probability. $\square$

Worked Gaussian Classifications

The calculations below use the same sequence: evaluate each class density at the observed point, multiply by its prior, normalize to get posterior probabilities, and select a class with the largest posterior.

Worked Example: Unequal Priors in a Univariate Model

Suppose there are two classes with $P(Y=1)=0.4$, $P(Y=2)=0.6$, and conditional distributions $X\mid(Y=1)\sim N(0,1)$ and $X\mid(Y=2)\sim N(1,1)$. Classify an observation $x=0.8$.

For a normal distribution with variance one, the density at $x$ and mean $\mu$ is $(2\pi)^{-1/2}\exp\{-(x-\mu)^2/2\}$. Thus,

f_1(0.8) = \frac{1}{\sqrt{2\pi}}e^{-0.8^2/2} = \frac{1}{\sqrt{2\pi}}e^{-0.32} \approx 0.28969, \qquad f_2(0.8) = \frac{1}{\sqrt{2\pi}}e^{-(0.8-1)^2/2} = \frac{1}{\sqrt{2\pi}}e^{-0.02} \approx 0.39104.

The prior-weighted densities are

\pi_1 f_1(0.8)\approx 0.4(0.28969)=0.11588, \qquad \pi_2 f_2(0.8)\approx 0.6(0.39104)=0.23462.

Their sum is approximately $0.35050$. The posterior probabilities are therefore

P(Y=1\mid X=0.8)\approx\frac{0.11588}{0.35050}=0.3306, \qquad P(Y=2\mid X=0.8)\approx\frac{0.23462}{0.35050}=0.6694.

The Bayes decision is class 2. Although the observation is closer to the class 1 mean than the class 2 mean, class 2 has both a larger density at this point and a larger prior probability. The posterior comparison, rather than distance to a mean alone, determines the decision.

Worked Example: A Two-Dimensional Model With Different Covariances

Let $X\in\mathbb{R}^2$. Suppose $P(Y=1)=0.6$, $P(Y=2)=0.4$, and

X\mid(Y=1)\sim N_2\left( \begin{pmatrix}0\\0\end{pmatrix}, \begin{pmatrix}1&0\\0&1\end{pmatrix} \right), \qquad X\mid(Y=2)\sim N_2\left( \begin{pmatrix}1\\0\end{pmatrix}, \begin{pmatrix}4&0\\0&1\end{pmatrix} \right).

Classify $x=(0,0)^\mathsf{T}$. For class 1, the covariance determinant is $1$, and the quadratic form $(x-\mu_1)^\mathsf{T}\Sigma_1^{-1}(x-\mu_1)$ is zero. Hence

f_1(x)=\frac{1}{2\pi}\approx 0.15915, \qquad \pi_1 f_1(x)\approx 0.6(0.15915)=0.09549.

For class 2, $|\Sigma_2|=4$, $\Sigma_2^{-1}=\begin{pmatrix}1/4&0\\0&1\end{pmatrix}$, and $x-\mu_2=(-1,0)^\mathsf{T}$. The quadratic form is

(x-\mu_2)^\mathsf{T}\Sigma_2^{-1}(x-\mu_2) = (-1,0) \begin{pmatrix}1/4&0\\0&1\end{pmatrix} \begin{pmatrix}-1\\0\end{pmatrix} = \frac{1}{4}.

Therefore,

f_2(x) = \frac{1}{2\pi\sqrt{4}}e^{-(1/4)/2} = \frac{1}{4\pi}e^{-1/8} \approx 0.07023, \qquad \pi_2 f_2(x)\approx 0.4(0.07023)=0.02809.

The normalizing sum is approximately $0.09549+0.02809=0.12358$, giving

P(Y=1\mid X=x)\approx\frac{0.09549}{0.12358}=0.7726, \qquad P(Y=2\mid X=x)\approx\frac{0.02809}{0.12358}=0.2274.

The observation is assigned to class 1. The determinant and quadratic-form terms both enter the Gaussian density: using only the distance from $x$ to a class mean would omit information in the covariance matrix.

Worked Example: Comparing Three Gaussian Classes

Suppose $X$ is univariate, with prior probabilities $(0.2,0.5,0.3)$ and conditional distributions $N(-2,1)$, $N(0,1)$, and $N(2,1)$ for classes 1, 2, and 3, respectively. Classify $x=0$.

The standard normal density at zero is $0.39894$, while at either $-2$ or $2$ it is approximately $0.05399$. Thus the prior-weighted densities are

\pi_1 f_1(0)\approx 0.2(0.05399)=0.01080,\quad \pi_2 f_2(0)\approx 0.5(0.39894)=0.19947,\quad \pi_3 f_3(0)\approx 0.3(0.05399)=0.01620.

The sum is approximately $0.22647$. Normalizing each value gives posterior probabilities approximately $0.0477$, $0.8808$, and $0.0715$ for classes 1, 2, and 3. For instance, the class 2 value is $0.19947/0.22647\approx0.8808$; the other two are $0.01080/0.22647\approx0.0477$ and $0.01620/0.22647\approx0.0715$. They sum to one up to rounding. The Bayes decision is class 2.

Interpretation, Ties, and Common Pitfalls

The Bayes rule is a decision rule conditional on a specified probability model and loss function. In these examples, the model parameters and priors were given. In applications they may instead be estimated from training data, but substituting estimates creates a plug-in classifier; it does not make the rule above a guarantee of minimum error for the unknown true model.

If a class has prior probability zero, then its prior-weighted density is zero at every point and its posterior probability is zero wherever the posterior is defined. If all prior-weighted densities vanish at a point, the marginal density is zero there, so the posterior formula has a zero denominator. Such points have probability zero under the model and do not affect overall risk; a rule can be specified there by convention. Ties between positive maximum values can be resolved by any fixed measurable rule, such as choosing the smallest class index, without changing the conditional or overall risk.

Mistake: choose the class with the largest density and ignore its prior. This is wrong when priors differ. Compare $\pi_k f_k(x)$, or equivalently the posterior probabilities, not $f_k(x)$ alone.
Mistake: choose the closest mean. This disregards both covariance and prior information. The multivariate density depends on the covariance determinant and the covariance-weighted quadratic form, as well as the mean.
Mistake: treat posterior probabilities as class densities. A density is not itself a probability of class membership. To obtain the posterior, multiply each density by its prior and divide by the sum across classes.
Mistake: assume the rule remains optimal under a different loss. The maximum-posterior rule follows specifically from zero-one loss. If errors have unequal costs, conditional expected loss must be minimized using that loss instead.

Key takeaway: For Gaussian class models under zero-one loss, calculate each prior-weighted class density $\pi_k f_k(x)$. A class attaining the largest value also has the largest posterior probability and minimizes the probability of error conditional on the observed feature vector.

Check Your Understanding

Use the class priors, conditional densities, and zero-one loss to reason through each question.

Why does maximizing $P(Y=k\mid X=x)$ minimize conditional zero-one risk?
If two classes have equal densities at an observation but one has a larger prior probability, which class has the larger posterior probability?
For a Gaussian class model, identify the roles of the covariance determinant and the quadratic form in the density.
What is the classification consequence of a class having prior probability zero?
If two classes tie for the largest posterior probability, can either be selected as a Bayes decision? Explain.

Bayes Decision Rules For Gaussian Class Models

What You'll Learn