Linear Discriminant Analysis With Common Covariance

From Gaussian Class Models to a Shared-Covariance Rule

A classification rule must combine two kinds of information: how well an observation fits each class and how plausible each class is before observing it. Under a Gaussian model, the log density measures fit. When every class has the same covariance matrix, part of that log density is identical across classes and can be discarded when choosing the largest score. The remaining expression is the linear discriminant score used in linear discriminant analysis (LDA).

Suppose $X$ is a $p$-dimensional observation and the possible classes are indexed by $k=1,\ldots,G$. Write $\mu_k$ for the mean vector in class $k$, and let $\pi_k=P(Y=k)$ be its prior probability, where $Y$ denotes class membership. The common-covariance model assumes that, conditional on class $k$, $X$ has a multivariate normal distribution with mean $\mu_k$ and the same positive-definite covariance matrix $\Sigma$.

Definition: Under the common-covariance Gaussian model, $X\mid(Y=k)\sim N_p(\mu_k,\Sigma)$ for each $k$, with $\Sigma$ positive definite and $\pi_k>0$. The LDA discriminant score for class $k$ at observation $x$ is $$ \delta_k(x)=x^\mathsf{T}\Sigma^{-1}\mu_k-\frac{1}{2}\mu_k^\mathsf{T}\Sigma^{-1}\mu_k+\log\pi_k. $$ The rule assigns $x$ to a class with the largest score. If several scores tie, the rule has a tie and requires a tie-breaking convention.

The covariance matrix must be invertible for this score to be defined. The model also assumes that the same $\Sigma$ applies to every class; the consequence of allowing class-specific covariance matrices is a different discriminant rule.

Why the Score Has This Form

The Bayes decision rule for equal misclassification costs assigns an observation to the class with the largest posterior probability. By Bayes’ rule, that is equivalent to maximizing the class-conditional density multiplied by the class prior. The Gaussian log density makes the cancellation under a shared covariance explicit.

Result: If $X\mid(Y=k)\sim N_p(\mu_k,\Sigma)$ for all $k$, with common positive-definite $\Sigma$ and positive priors $\pi_k$, then maximizing $P(Y=k\mid X=x)$ over $k$ is equivalent to maximizing $\delta_k(x)$.

Proof. The multivariate normal density in class $k$ is $$ f_k(x)=(2\pi)^{-p/2}|\Sigma|^{-1/2} \exp\left\{-\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma^{-1}(x-\mu_k)\right\}. $$ Bayes’ rule gives $P(Y=k\mid X=x)$ as $\pi_k f_k(x)$ divided by a denominator that is the same for every candidate class. Thus the maximizing class also maximizes $\log\{\pi_k f_k(x)\}$. Taking this logarithm yields $$ \log\{\pi_k f_k(x)\} =\log\pi_k-\frac{p}{2}\log(2\pi)-\frac{1}{2}\log|\Sigma| -\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma^{-1}(x-\mu_k). $$ Expand the quadratic term: $$ -\frac{1}{2}(x-\mu_k)^\mathsf{T}\Sigma^{-1}(x-\mu_k) =-\frac{1}{2}x^\mathsf{T}\Sigma^{-1}x +x^\mathsf{T}\Sigma^{-1}\mu_k -\frac{1}{2}\mu_k^\mathsf{T}\Sigma^{-1}\mu_k. $$ The first quadratic term in $x$, the normalizing constant, and the determinant term do not depend on $k$. Removing these common terms leaves exactly $\delta_k(x)$, so the maximizing class is unchanged. $\square$

This derivation explains why an LDA score is not just a measure of closeness to a class mean: it includes both a term involving the observation and mean, and a term involving the mean alone. It also includes the prior probability. The score is a convenient quantity for comparing classes; its numerical value is not itself a posterior probability.

Worked Example: Comparing Two Classes in Two Dimensions

Suppose there are two classes with means $\mu_1=(0,0)^\mathsf{T}$ and $\mu_2=(2,1)^\mathsf{T}$, common covariance $$ \Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}, $$ and priors $\pi_1=0.4$ and $\pi_2=0.6$. Classify $x=(1.5,1)^\mathsf{T}$.

First calculate the inverse: $$ \Sigma^{-1}=\frac{1}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix}. $$ Because both scores contain the same terms depending only on $x$, compare $\delta_2(x)-\delta_1(x)$. Since $\mu_1$ is zero, the difference is $$ \delta_2(x)-\delta_1(x) =x^\mathsf{T}\Sigma^{-1}\mu_2-\frac{1}{2}\mu_2^\mathsf{T}\Sigma^{-1}\mu_2 +\log\left(\frac{0.6}{0.4}\right). $$ The matrix-vector product is $$ \Sigma^{-1}\mu_2 =\frac{1}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix} \begin{pmatrix}2\\1\end{pmatrix} =\begin{pmatrix}3/7\\2/7\end{pmatrix}. $$ Therefore $x^\mathsf{T}\Sigma^{-1}\mu_2=(1.5)(3/7)+(1)(2/7)=13/14$, while $\mu_2^\mathsf{T}\Sigma^{-1}\mu_2=(2)(3/7)+(1)(2/7)=8/7$. The score difference is $$ \delta_2(x)-\delta_1(x)=\frac{13}{14}-\frac{4}{7}+\log(1.5) =\frac{5}{14}+0.4055\approx 0.7626. $$ It is positive, so class 2 has the larger score and is selected.

Using Estimated Means and a Pooled Covariance

In applications, the class means and shared covariance are usually unknown. Suppose the training sample contains $n_k$ observations in class $k$, with $\sum_{k=1}^G n_k=n$. Let $\bar{x}_k$ be the sample mean in class $k$, and let $S_k$ be its usual sample covariance matrix, calculated with denominator $n_k-1$. A standard shared-covariance estimate is the within-class pooled covariance $$ S_p=\frac{\sum_{k=1}^{G}(n_k-1)S_k}{n-G}. $$ This estimate combines within-class variation; it does not combine the differences between the class means into the covariance estimate. For the formula to define an inverse, $S_p$ must be positive definite. Estimated priors can be supplied by the modelling context or, for example, set to $\hat{\pi}_k=n_k/n$.

Estimated LDA rule: Given estimated class means $\bar{x}_k$, a positive-definite pooled covariance estimate $S_p$, and positive prior estimates $\hat{\pi}_k$, calculate $$ \hat{\delta}_k(x)=x^\mathsf{T}S_p^{-1}\bar{x}_k-\frac{1}{2}\bar{x}_k^\mathsf{T}S_p^{-1}\bar{x}_k+\log\hat{\pi}_k. $$ Assign $x$ to a class with the largest estimated score.

The estimated rule has the same algebraic form as the population rule, with unknown quantities replaced by estimates. This substitution defines a plug-in classification rule; it does not make the estimates known without uncertainty. The validity of using one pooled covariance rests on the shared-covariance modelling assumption.

Worked Example: Classifying With Three Classes and Unequal Priors

Consider a one-dimensional model with common variance $\sigma^2=4$. The three class means are $-1,1,3$, and the respective priors are $0.2,0.5,0.3$. For an observation $x=1.2$, compute each score and identify the predicted class.

In one dimension the score is $\delta_k(x)=x\mu_k/4-\mu_k^2/8+\log\pi_k$. For class 1, $$ \delta_1(1.2)=\frac{(1.2)(-1)}{4}-\frac{1}{8}+\log(0.2) =-0.3-0.125-1.6094=-2.0344. $$ For class 2, $$ \delta_2(1.2)=\frac{(1.2)(1)}{4}-\frac{1}{8}+\log(0.5) =0.3-0.125-0.6931=-0.5181. $$ For class 3, $$ \delta_3(1.2)=\frac{(1.2)(3)}{4}-\frac{9}{8}+\log(0.3) =0.9-1.125-1.2040=-1.4290. $$ The largest score is $-0.5181$, so the observation is assigned to class 2. Class 2 is selected even though the observation is not exactly at its mean: the rule compares all three scores, including the prior contributions.

A Fully Worked Training-Sample Calculation

The next example shows how sample summaries enter the estimated rule. It uses two classes with three observations each. Writing out the sample means and covariance matrices makes the estimation step and the final score comparison visible.

Worked Example: Pooling Sample Covariances Before Classification

For class A, suppose the three training observations are $(1,c),(-1,c),(0,-2c)$, where $c=\sqrt{2/3}$. For class B, the observations are $(2+\sqrt{3},1+c),(2-\sqrt{3},1+c),(2,1-2c)$. Use equal priors and classify $x=(1.2,0.8)^\mathsf{T}$.

The sample means are $\bar{x}_A=(0,0)^\mathsf{T}$ and $\bar{x}_B=(2,1)^\mathsf{T}$, since each coordinate’s deviations from its stated mean sum to zero. In class A, the deviations in the first coordinate are $1,-1,0$, and in the second they are $c,c,-2c$. Each set of deviations sums to zero, and their cross-products sum to $c-c+0=0$. Since the sample covariance denominator is $3-1=2$, the covariance is $$ S_A=\begin{pmatrix} (1^2+(-1)^2)/2&0\\ 0&(c^2+c^2+(-2c)^2)/2 \end{pmatrix} =\begin{pmatrix}1&0\\0&2\end{pmatrix}, $$ using $c^2=2/3$. For class B, the deviations are $(\sqrt{3},c),(-\sqrt{3},c),(0,-2c)$. Their cross-products also sum to zero, and hence $$ S_B=\begin{pmatrix} (3+3)/2&0\\ 0&(c^2+c^2+4c^2)/2 \end{pmatrix} =\begin{pmatrix}3&0\\0&2\end{pmatrix}. $$ With $n_A=n_B=3$, the pooled covariance is $$ S_p=\frac{2S_A+2S_B}{4} =\begin{pmatrix}2&0\\0&2\end{pmatrix}, \qquad S_p^{-1}=\begin{pmatrix}1/2&0\\0&1/2\end{pmatrix}. $$ The equal priors make their log terms cancel in the score difference. Because $\bar{x}_A=0$, $$ \hat{\delta}_B(x)-\hat{\delta}_A(x) =x^\mathsf{T}S_p^{-1}\bar{x}_B -\frac{1}{2}\bar{x}_B^\mathsf{T}S_p^{-1}\bar{x}_B. $$ The first term is $(1.2)(1)+(0.8)(0.5)=1.6$. The quadratic term before multiplication by $1/2$ is $(2)(1)+(1)(0.5)=2.5$, so the score difference is $1.6-1.25=0.35$. It is positive, and the estimated rule assigns the observation to class B.

What the Comparison Does—and Does Not—Say

For any pair of classes $k$ and $j$, their score difference is $$ \delta_k(x)-\delta_j(x) =x^\mathsf{T}\Sigma^{-1}(\mu_k-\mu_j) -\frac{1}{2}\left(\mu_k^\mathsf{T}\Sigma^{-1}\mu_k-\mu_j^\mathsf{T}\Sigma^{-1}\mu_j\right) +\log\left(\frac{\pi_k}{\pi_j}\right). $$ This expression follows by subtracting the two score definitions. It shows that the shared covariance allows the common quadratic term in $x$ to cancel. With estimated quantities the same subtraction applies to $\hat{\delta}_k(x)-\hat{\delta}_j(x)$, provided $S_p$ is invertible. The comparison depends on the observation, class means, covariance, and priors—not on a separate class-specific covariance term.

The classification decision uses only which score is largest. Adding the same number to every class score at a given $x$ leaves that decision unchanged. However, changing a class-specific term, such as its prior contribution, can change the selected class. This is why it is safe to remove terms common to every class but not to omit $\log\pi_k$ when the priors differ.

A useful caution is that an estimated score is not a calibrated probability. To obtain posterior probabilities, one must transform and normalize the class scores appropriately; simply interpreting a larger score as a particular probability is incorrect. The scores here establish an ordering for classification under the stated model.

Common Mistakes

Using only the distance to the class mean. The LDA score also includes the mean-dependent quadratic term and the prior. Those terms can affect which class wins.
Dropping unequal priors. The log-prior terms cancel only when they are the same for all classes. If priors differ, leaving them out changes the rule.
Using a different covariance in each score. The common-covariance derivation depends on using the same $\Sigma$ in every class density. Replacing it with class-specific matrices changes the model and prevents this cancellation.
Inverting a singular pooled estimate. The stated score requires an inverse. If $S_p$ is singular, the formula cannot be applied as written; a different estimation strategy or regularization is needed.
Calling the score a posterior probability. A score is used to rank classes. Its raw value need not lie between zero and one and is not itself a probability.

Key takeaway: Under a Gaussian model with one positive-definite covariance shared across classes, expand the log density, remove terms common to all classes, and compare the resulting discriminant scores. In practice, substitute estimated means, a positive-definite pooled covariance, and the chosen class priors.

Check Your Understanding

Use the score definition and cancellation argument to answer these questions.

Which terms in the common-covariance Gaussian log density can be removed when comparing candidate classes, and why?
In the estimated LDA rule, what must be true of $S_p$ for the displayed inverse to exist?
If two classes have different prior probabilities, what happens if their log-prior terms are omitted?
Why is the estimated discriminant score not itself a posterior probability?
For two classes, what does a positive value of $\hat{\delta}_2(x)-\hat{\delta}_1(x)$ imply about the estimated classification?