Class Priors And Intercepts In Gaussian Discrimination

Why Class Priors Affect Gaussian Classification

Two classes can have the same Gaussian feature distributions but occur at different rates in the population. A classifier that incorporates those rates may assign an observation to a more common class even when its feature values are not especially typical of that class. In Gaussian discrimination, this effect has a precise form: the logarithm of each class prior is added to its discriminant score.

The consequences depend on the covariance structure. With a common covariance matrix, the prior changes only the score intercept, so pairwise LDA boundaries keep their orientation and move. With class-specific covariance matrices, the prior still adds a constant to each score, but the existing QDA score differences are quadratic in the observation. A changed constant can move a quadratic boundary or, in some cases, remove it.

Definition: Let $C$ be a class label taking values $1,\ldots,K$, with prior probabilities $\pi_k=P(C=k)>0$ and $\sum_{k=1}^K\pi_k=1$. Let $X\in\mathbb{R}^p$ be the feature vector. In Gaussian classification, the class-conditional model is $X\mid C=k\sim N(\mu_k,\Sigma_k)$, where each $\Sigma_k$ is positive definite. A Gaussian discriminant score is a class-specific quantity whose largest value gives the maximum-posterior-probability classification.

Deriving the Prior-Adjusted Scores

For an observation $x$, Bayes' rule gives $P(C=k\mid X=x)$ proportional to $\pi_k f_k(x)$, where $f_k$ is the Gaussian density under class $k$. Since the logarithm is strictly increasing, maximizing the posterior probability is equivalent to maximizing $\log\pi_k+\log f_k(x)$. Terms that are the same for every class can be dropped without changing which score is largest.

Theorem: Under the Gaussian class-conditional models above, a score equivalent for maximum-posterior classification is $$ g_k(x)=\log\pi_k-\frac{1}{2}\log|\Sigma_k|-\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k). $$ That is, assigning $x$ to a class with the largest $g_k(x)$ maximizes $P(C=k\mid X=x)$.

Proof. The density of a $p$-dimensional Gaussian with mean $\mu_k$ and positive definite covariance $\Sigma_k$ is $$ f_k(x)=(2\pi)^{-p/2}|\Sigma_k|^{-1/2}\exp\left\{-\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)\right\}. $$ By Bayes' rule, the posterior probability is $$ P(C=k\mid X=x)=\frac{\pi_k f_k(x)}{\sum_{j=1}^K\pi_j f_j(x)}. $$ The denominator is positive and is the same for every candidate class $k$. Thus maximizing the posterior is equivalent to maximizing $\pi_k f_k(x)$. Taking logarithms gives $\log\pi_k+\log f_k(x)$. The density's term $-\frac{p}{2}\log(2\pi)$ is also common to all classes and can be omitted. The remaining terms are exactly $g_k(x)$, proving the claim. $\square$

In QDA, the matrices $\Sigma_k$ can differ across classes, so the log-determinant and quadratic terms are class-specific. In LDA, the covariance is common: $\Sigma_k=\Sigma$ for every $k$. Expanding the quadratic form in that case gives $$ g_k(x)=x^\mathsf{T}\Sigma^{-1}\mu_k-\frac{1}{2}\mu_k^\mathsf{T}\Sigma^{-1}\mu_k+\log\pi_k-\frac{1}{2}x^\mathsf{T}\Sigma^{-1}x-\frac{1}{2}\log|\Sigma|. $$ The last two terms do not depend on $k$, so an equivalent LDA score is $$ \delta_k(x)=x^\mathsf{T}\Sigma^{-1}\mu_k-\frac{1}{2}\mu_k^\mathsf{T}\Sigma^{-1}\mu_k+\log\pi_k. $$

Worked Example: A Prior Shift Moves a Two-Class LDA Boundary

Suppose $X$ is one-dimensional, with $X\mid C=1\sim N(0,1)$ and $X\mid C=2\sim N(2,1)$. First take equal priors, then change them to $\pi_1=0.2$ and $\pi_2=0.8$. Find the boundary under each choice and classify $x=0.5$.

The common variance is $1$, so the LDA scores, up to terms shared across classes, are $$ \delta_1(x)=\log\pi_1,\qquad \delta_2(x)=2x-\frac{1}{2}(2)^2+\log\pi_2=2x-2+\log\pi_2. $$ With equal priors, the log-prior terms cancel. Equating the scores gives $0=2x-2$, so the boundary is $x=1$. At $x=0.5$, the scores are $0$ and $2(0.5)-2=-1$; the observation is assigned to class 1.

With $(\pi_1,\pi_2)=(0.2,0.8)$, the boundary solves $$ \log(0.2)=2x-2+\log(0.8), \qquad x=1+\frac{1}{2}\log\left(\frac{0.2}{0.8}\right) =1+\frac{1}{2}\log(0.25)\approx 0.307. $$ At $x=0.5$, the scores are $\delta_1=\log(0.2)\approx-1.609$ and $\delta_2=-1+\log(0.8)\approx-1.223$. Class 2 now has the larger score. The likelihood models did not change; the increased prior probability of class 2 moved the boundary left and changed this classification.

How Priors Shift LDA Intercepts

For two LDA classes $k$ and $l$, subtracting their scores makes the prior's effect especially clear: $$ \delta_k(x)-\delta_l(x) =(\mu_k-\mu_l)^\mathsf{T}\Sigma^{-1}x -\frac{1}{2}\left(\mu_k^\mathsf{T}\Sigma^{-1}\mu_k-\mu_l^\mathsf{T}\Sigma^{-1}\mu_l\right) +\log\left(\frac{\pi_k}{\pi_l}\right). $$ The pairwise decision boundary is where this difference is zero. The first term determines the direction of the boundary; the prior odds $\pi_k/\pi_l$ occur only in the constant term.

Result: If $\mu_k\ne\mu_l$, changing the class priors in an LDA model with fixed means and common positive definite covariance leaves the normal vector $\Sigma^{-1}(\mu_k-\mu_l)$ unchanged. It translates the pairwise boundary parallel to itself. The signed change in its displacement, measured along that normal vector, is determined by the change in the log prior odds.

Derivation. Write $n=\Sigma^{-1}(\mu_k-\mu_l)$. The boundary equation has the form $n^\mathsf{T}x=c$, where $$ c=\frac{1}{2}\left(\mu_k^\mathsf{T}\Sigma^{-1}\mu_k-\mu_l^\mathsf{T}\Sigma^{-1}\mu_l\right)-\log\left(\frac{\pi_k}{\pi_l}\right). $$ The means and covariance determine $n$, so changing the priors leaves $n$ fixed. If the log prior odds change by $r$, then $c$ changes by $-r$. The corresponding displacement vector between the two parallel hyperplanes is $(-r)n/(n^\mathsf{T}n)$, because its inner product with $n$ is $-r$. Thus the direction of the boundary is unchanged, while its position shifts according to the log prior odds. $\square$

Worked Example: A Prior Change in Three-Class LDA

Let $X\in\mathbb{R}^2$, let the common covariance be the identity matrix, and take means $\mu_1=(0,0)^\mathsf{T}$, $\mu_2=(2,0)^\mathsf{T}$, and $\mu_3=(0,2)^\mathsf{T}$. Classify $x=(0.8,0.7)^\mathsf{T}$, first with equal priors and then with $(\pi_1,\pi_2,\pi_3)=(0.1,0.6,0.3)$.

For this covariance, $\delta_k(x)=x^\mathsf{T}\mu_k-\frac{1}{2}\mu_k^\mathsf{T}\mu_k+\log\pi_k$. With equal priors, the log-prior term is common and can be omitted. The three scores are $$ \delta_1=0,\qquad \delta_2=2(0.8)-2=-0.4,\qquad \delta_3=2(0.7)-2=-0.6. $$ The largest score is $\delta_1$, so the observation is assigned to class 1.

With the new priors, the scores are $$ \delta_1=\log(0.1)\approx-2.303,\qquad \delta_2=-0.4+\log(0.6)\approx-0.911,\qquad \delta_3=-0.6+\log(0.3)\approx-1.804. $$ The largest is now $\delta_2$, so the classification changes to class 2. Each class's score has gained its own constant $\log\pi_k$; the terms that depend on $x$ have not changed.

How Priors Affect QDA Scores and Boundaries

In QDA, each class has its own covariance matrix. The score $g_k(x)$ already contains class-specific quadratic and log-determinant terms. A change in priors adds $\log\pi_k$ to the score for class $k$, so the difference $g_k(x)-g_l(x)$ changes by the constant $\log(\pi_k/\pi_l)$. The quadratic terms themselves do not change when the Gaussian parameters are held fixed.

Unlike the LDA case, a QDA boundary need not be a hyperplane. Adding a constant to a quadratic score difference changes its zero set: the boundary can shift, change its shape or number of components, or cease to exist. A boundary may disappear when one class has the larger score everywhere. This follows from the form of the score difference; it does not mean that the class-conditional Gaussian distributions changed.

Worked Example: A QDA Prior Change Removes a Boundary

Consider two one-dimensional classes with $X\mid C=1\sim N(0,1)$ and $X\mid C=2\sim N(0,4)$. Compare equal priors with $(\pi_1,\pi_2)=(0.2,0.8)$, and examine the boundary and the classification at $x=0.5$.

The QDA scores are $$ g_1(x)=\log\pi_1-\frac{1}{2}x^2,\qquad g_2(x)=\log\pi_2-\frac{1}{2}\log 4-\frac{1}{8}x^2. $$ Therefore their difference is $$ g_2(x)-g_1(x)=\log\left(\frac{\pi_2}{\pi_1}\right)-\log 2+\frac{3}{8}x^2. $$ With equal priors this becomes $-\log 2+\frac{3}{8}x^2$. The boundary satisfies $x^2=\frac{8}{3}\log 2$, giving $x\approx-1.360$ and $x\approx1.360$. At $x=0.5$, the difference is $-\log 2+\frac{3}{8}(0.25)\approx-0.599$, so class 1 has the larger score.

With $(\pi_1,\pi_2)=(0.2,0.8)$, the difference is $$ \log 4-\log 2+\frac{3}{8}x^2=\log 2+\frac{3}{8}x^2. $$ It is positive for every real $x$, since $\log 2>0$ and $x^2\geq0$. There is no boundary: class 2 has the larger score everywhere. At $x=0.5$, the difference is $\log 2+0.09375\approx0.787$, so the classification has changed from class 1 to class 2.

Interpretation and Common Pitfalls

A prior represents the assumed probability of a class before observing its features. In the score, prior probabilities appear as log priors, not as probabilities multiplied directly into the linear or quadratic feature terms. The relative effect for two classes is governed by the log prior odds. If all priors are multiplied by the same positive constant and then normalized to sum to one, their relative log differences are unchanged, so the classification is unchanged.

The prior adjustment is meaningful only in relation to the model being used. In these calculations, the class-conditional Gaussian distributions are fixed while the priors change. In practice, priors might reflect population prevalence or a specified target population; changing the target population can change the appropriate priors even when the class-conditional models are retained.

Mistake: adding the prior probability instead of its logarithm. The Gaussian log-density is what is being combined with the prior. The score therefore includes $\log\pi_k$, not $\pi_k$.
Mistake: saying that priors change LDA slopes. With fixed means and common covariance, the prior changes only the score's constant term. It translates a pairwise boundary without changing its orientation.
Mistake: assuming every QDA boundary merely shifts in parallel. QDA score differences are generally quadratic, not linear. A constant change can alter the zero set substantially or eliminate it.
Mistake: treating a prior as a misclassification cost. A prior describes class prevalence before observing features. Cost-sensitive classification uses a different decision rule; the two concepts should not be conflated.

Key takeaway: For fixed Gaussian class models, each class receives the additive score adjustment $\log\pi_k$. In LDA this changes intercepts and translates pairwise linear boundaries; in QDA it changes the constant in each pairwise quadratic score difference and can alter or remove decision boundaries.

Check Your Understanding

Use the score formulas and show the comparisons that support each answer.

Why does maximizing the posterior probability give the same classification as maximizing the Gaussian score with $\log\pi_k$ included?
In two-class LDA, which part of the pairwise score difference changes when the prior odds change, and what does that imply about the boundary's orientation?
For two QDA classes, explain why changing the priors adds a constant to the pairwise score difference without changing its quadratic coefficients.
In the QDA worked example, why does the changed prior eliminate both boundary points rather than merely moving them?
Explain the difference between a class prior and a misclassification cost.