Worked Multiclass Comparison Of LDA And QDA

Comparing Multiclass Gaussian Classifiers

LDA and QDA can use the same class means and prior probabilities yet assign an observation to different classes. The distinction is their covariance assumption: LDA uses one covariance matrix shared by every class, whereas QDA allows a separate covariance matrix for each class. A worked comparison is useful because a classification depends on the combined effects of distance from a class mean, covariance shape and scale, and prior probability.

Let \(X\) be a \(p\)-dimensional observation and \(G\in\{1,\ldots,K\}\) its class. Assume \(P(G=k)=\pi_k>0\), with \(\sum_{k=1}^K\pi_k=1\). Under the Gaussian model, \(X\mid G=k\sim N_p(\mu_k,\Sigma_k)\), where each \(\Sigma_k\) is positive definite. The equal-cost classification rule assigns \(x\) to the class with the largest posterior probability \(P(G=k\mid X=x)\).

Definition: The QDA discriminant score for class \(k\), with terms common to all classes removed, is \[ \delta_k^{Q}(x)=\log\pi_k-\frac12\log|\Sigma_k|-\frac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k). \] QDA assigns \(x\) to a class maximizing \(\delta_k^{Q}(x)\). In LDA, all classes use a common positive-definite covariance \(\Sigma\), so the score is \[ \delta_k^{L}(x)=\log\pi_k-\frac12(x-\mu_k)^\mathsf{T}\Sigma^{-1}(x-\mu_k). \] The omitted terms are common to every class and therefore cannot affect which score is largest.

The quadratic form in each score measures squared Mahalanobis distance from \(x\) to the class mean. In QDA, that distance uses the class-specific covariance, and the term \(-\tfrac12\log|\Sigma_k|\) accounts for the Gaussian density’s scale. Priors contribute through \(\log\pi_k\). Comparing only distances, or only priors, can therefore give the wrong assignment.

Why the Scores Give the Classification and Posterior

The score formula follows directly from the multivariate normal density. Writing the density of \(X\mid G=k\) at \(x\) and applying Bayes’ rule gives a class probability proportional to the prior times that density.

Result: Under the stated Gaussian model, \[ P(G=k\mid X=x)=\frac{\exp\{\delta_k^{Q}(x)\}}{\sum_{j=1}^K\exp\{\delta_j^{Q}(x)\}}. \] With shared covariance, \(\delta_k^Q\) can be replaced by \(\delta_k^L\). Thus maximizing a discriminant score is equivalent to maximizing the posterior probability.

To verify the result, the class-conditional density is \[ f_k(x)=(2\pi)^{-p/2}|\Sigma_k|^{-1/2} \exp\left\{-\frac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)\right\}. \] Bayes’ rule gives \(P(G=k\mid X=x)=\pi_k f_k(x)/\sum_j\pi_j f_j(x)\). The common factor \((2\pi)^{-p/2}\) cancels. The remaining numerator is \(\exp\{\delta_k^Q(x)\}\), and the denominator is the sum of these numerators. This proves the stated expression and shows why the largest score identifies the largest posterior.

A second useful check is that QDA contains LDA as the common-covariance case. If \(\Sigma_k=\Sigma\) for every \(k\), then the determinant term in QDA is common across classes. Its quadratic form expands as \[ (x-\mu_k)^\mathsf{T}\Sigma^{-1}(x-\mu_k) =x^\mathsf{T}\Sigma^{-1}x-2\mu_k^\mathsf{T}\Sigma^{-1}x+\mu_k^\mathsf{T}\Sigma^{-1}\mu_k. \] The first term is also common to every class. Removing these common terms leaves \[ \log\pi_k+\mu_k^\mathsf{T}\Sigma^{-1}x-\frac12\mu_k^\mathsf{T}\Sigma^{-1}\mu_k, \] which is the LDA score up to a class-independent term. Therefore both rules rank classes identically when their covariance matrices are equal. \(\square\)

Worked Calculations

Worked Example: Three-Class LDA Posterior Probabilities

Suppose \(X\) is one-dimensional, the class means are \(\mu_1=-2,\ \mu_2=0,\ \mu_3=2\), all classes have variance \(1\), and the priors are \((0.5,0.3,0.2)\). Classify \(x=0.5\) and calculate its posterior probabilities.

Because the variance is common, the class score is \(\log\pi_k-\tfrac12(x-\mu_k)^2\). The three squared deviations are \(6.25,\ 0.25,\ 2.25\). Thus \[ \delta_1^L=\log(0.5)-3.125\approx-3.8181,\quad \delta_2^L=\log(0.3)-0.125\approx-1.3290,\quad \delta_3^L=\log(0.2)-1.125\approx-2.7344. \] The largest score is \(\delta_2^L\), so the LDA assignment is class 2.

For the posterior, exponentiate the scores. The relative weights are \[ w_1=0.5e^{-3.125}\approx0.0219685,\quad w_2=0.3e^{-0.125}\approx0.2647491,\quad w_3=0.2e^{-1.125}\approx0.0649305. \] Their sum is approximately \(0.3516481\). Dividing each weight by this sum gives posterior probabilities approximately \((0.0625,\ 0.7529,\ 0.1846)\), which sum to \(1.0000\) to four decimal places. The largest posterior is for class 2, as the score comparison requires.

Worked Example: QDA Posterior Probabilities and the Scale Term

Consider three Gaussian classes with a common mean of \(0\), priors \((0.4,0.3,0.3)\), and standard deviations \(\sigma_1=0.1,\ \sigma_2=0.3/2.995,\ \sigma_3=0.3/1.997\). Classify the observation \(x=0\). The variances are \(\sigma_k^2\), and all are positive.

At \(x=0\), each class-specific exponential distance factor is \(1\). The Gaussian density for class \(k\), multiplied by its prior, is proportional to \(\pi_k/\sigma_k\). The relative weights are therefore exactly \[ w_1=0.4/0.1=4,\qquad w_2=0.3/(0.3/2.995)=2.995,\qquad w_3=0.3/(0.3/1.997)=1.997. \] The normalizing total is \(4+2.995+1.997=8.992\). Hence \[ P(G=1\mid x)=4/8.992\approx0.4448,\quad P(G=2\mid x)=2.995/8.992\approx0.3331,\quad P(G=3\mid x)=1.997/8.992\approx0.2221. \] These rounded probabilities sum to \(1.0000\), and the largest is for class 1. Since the means coincide, the assignment here comes entirely from the class priors and the density scale \(1/\sigma_k\); ignoring the covariance-dependent determinant term would miss that scale effect.

Worked Example: LDA and QDA Assign Different Classes

Let the three class means be \((-1,0,1)\), with priors \((0.4,0.4,0.2)\), and consider \(x=0\). For the LDA model, take the shared variance to be \(1\). For the QDA model, take class variances \((1,25,1)\). Compare the assignments under these two covariance specifications.

Under LDA, the relative weights are \(\pi_k\exp\{-(x-\mu_k)^2/2\}\). At \(x=0\), they are \[ 0.4e^{-1/2}\approx0.2426123,\qquad 0.4,\qquad 0.2e^{-1/2}\approx0.1213061. \] The total is approximately \(0.7639184\), giving posterior probabilities \((0.3176,\ 0.5236,\ 0.1588)\). LDA assigns the observation to class 2.

Under QDA, the relative weights are \(\pi_k\sigma_k^{-1}\exp\{-(x-\mu_k)^2/(2\sigma_k^2)\}\); the common normal-density factor cancels. The weights are \[ 0.4e^{-1/2}\approx0.2426123,\qquad 0.4/5=0.08,\qquad 0.2e^{-1/2}\approx0.1213061. \] They sum to approximately \(0.4439184\), so the posterior probabilities are approximately \((0.5465,\ 0.1802,\ 0.2733)\). QDA assigns the observation to class 1.

The middle class has a much larger variance in the QDA specification. Although \(x=0\) is exactly its mean, its density at that point is reduced by the scale factor \(1/5\). The outer classes are one unit from their means, but their smaller variances and prior weights together give class 1 the largest posterior. The two rules disagree because their covariance assumptions give different balances between distance and density scale.

How to Compare the Rules Reliably

For a specified observation, use the same class means and priors in both calculations, but use the covariance matrix appropriate to each model. A reliable sequence is to calculate the class scores, compare scores to identify the assignment, and then exponentiate and normalize if posterior probabilities are requested. The normalization is necessary for probabilities; it is unnecessary merely to identify the largest score.

Record the model ingredients.
Write down each \(\pi_k\), \(\mu_k\), and either the shared \(\Sigma\) or each class-specific \(\Sigma_k\). Check that priors are positive and sum to one.

Evaluate each score at the observation.
Include the log-prior, the covariance determinant term for QDA, and the appropriate quadratic distance term.

Compare and, if needed, normalize.
The largest score gives the assignment. For posterior probabilities, exponentiate scores and divide by their sum; subtracting the largest score before exponentiating can help avoid numerical underflow.

Common Pitfall and Interpretation

A common mistake is to assume that the class whose mean is closest in ordinary Euclidean distance must be selected. That reasoning ignores both the covariance geometry and the prior. Even a Mahalanobis-distance comparison alone is incomplete for QDA: classes with different covariance determinants have different density scales. The class assignment is determined by the full score, not by any one of its components.

LDA is appropriate for a model with a shared within-class covariance; QDA allows the class distributions to have different covariance structures. Comparing their assignments clarifies the practical effect of that modeling choice for a particular observation. It does not, by itself, establish which model will classify future observations better; that is a question about classification performance beyond the calculation of these rules.

Key takeaway: For multiclass Gaussian classification, compare complete discriminant scores. LDA uses one covariance matrix; QDA uses a covariance matrix and determinant term for each class. Convert scores to posterior probabilities by exponentiating and normalizing, and expect assignments to differ when the covariance assumptions alter the relative class densities.

Check Your Understanding

Use the score formulas and calculations above to answer the following.

Why can the largest QDA Mahalanobis distance not, by itself, determine which class has the largest posterior probability?
In the LDA example, why can the common Gaussian density constant be omitted when comparing classes?
For the QDA posterior example, verify that the relative weights sum to \(8.992\), then explain why the posterior for class 1 is largest.
In the disagreement example, which term most directly reduces the middle class’s QDA density at \(x=0\)?
What condition on the class covariance matrices makes QDA and LDA rank classes identically, assuming the same priors and means?