Quadratic Discriminant Analysis With Class-Specific Covariances

Why Class-Specific Covariances Change the Rule

When two Gaussian classes have different covariance matrices, their densities can differ in both location and spread. A point may be close to one class mean yet still be more likely under another class because that class’s covariance gives greater density in the point’s direction. The covariance determinant also matters: it controls the overall scale of the Gaussian density, and therefore contributes to every class comparison.

In linear discriminant analysis with a common covariance matrix, the shared quadratic part cancels when class scores are compared. With class-specific covariance matrices it generally does not cancel. This tutorial derives the resulting quadratic discriminant analysis (QDA) score and works through how its two covariance contributions—the inverse-covariance quadratic form and the log determinant—affect assignments.

Definition: Suppose an observation $X\in\mathbb{R}^p$, conditional on belonging to class $k$, has a multivariate normal distribution with mean vector $\mu_k$ and positive-definite covariance matrix $\Sigma_k$: $$ X\mid C=k\sim N_p(\mu_k,\Sigma_k). $$ The QDA score based on the class-conditional density is $$ \delta_k(x)=-\frac12\log|\Sigma_k|-\frac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k). $$ For equal class priors and equal misclassification costs, assign $x$ to a class with the largest score. If priors are unequal, the log prior probability is added to the score; the covariance-dependent terms retain the same form.

Positive definiteness ensures that $\Sigma_k^{-1}$ exists and $|\Sigma_k|>0$, so the inverse and logarithm in the score are defined. The quadratic form measures the observation’s displacement from the class mean relative to that class’s covariance. The determinant term adjusts for the overall scale of the density. Neither term should be dropped merely because the means are known.

Deriving the QDA Score

For a $p$-dimensional normal distribution with positive-definite covariance matrix, the density at $x$ is $$ f_k(x)=\frac{1}{(2\pi)^{p/2}|\Sigma_k|^{1/2}}\exp\left\{-\frac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)\right\}. $$

Taking the logarithm gives a convenient comparison scale. The term $-\frac{p}{2}\log(2\pi)$ is common to all classes and can be removed without changing which density is largest. The remaining terms are exactly $\delta_k(x)$.

Theorem: Under the class-conditional Gaussian model above, if all class priors are equal and misclassification costs are equal, maximizing the class-conditional density $f_k(x)$ is equivalent to maximizing $\delta_k(x)$. For any two classes $k$ and $l$, their score difference is $$ \delta_k(x)-\delta_l(x) =-\frac12\log\frac{|\Sigma_k|}{|\Sigma_l|} -\frac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k) +\frac12(x-\mu_l)^\mathsf{T}\Sigma_l^{-1}(x-\mu_l). $$

Proof. The Gaussian density is positive, and the logarithm is strictly increasing. Thus $f_k(x)>f_l(x)$ if and only if $\log f_k(x)>\log f_l(x)$. Taking the logarithm of the density yields $$ \log f_k(x)=-\frac{p}{2}\log(2\pi)-\frac12\log|\Sigma_k|-\frac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k). $$ The first term is identical for every class, so removing it leaves $\delta_k(x)$ and does not change the ordering. Subtracting the score for class $l$ from that for class $k$ gives the displayed difference. Equal priors and equal costs make choosing the largest class-conditional density the relevant decision rule. $\square$

The pairwise difference also shows why QDA has quadratic rather than generally linear comparisons. Expanding the quadratic forms gives the following result.

Result: The coefficient matrix of the quadratic part of $\delta_k(x)-\delta_l(x)$ is $$ -\frac12\left(\Sigma_k^{-1}-\Sigma_l^{-1}\right). $$ If the two precision matrices differ, the score difference generally includes quadratic terms. If they are equal, this quadratic part vanishes.

Derivation. Write $P_k=\Sigma_k^{-1}$. Since $P_k$ is symmetric, $$ (x-\mu_k)^\mathsf{T}P_k(x-\mu_k)=x^\mathsf{T}P_kx-2x^\mathsf{T}P_k\mu_k+\mu_k^\mathsf{T}P_k\mu_k. $$ Substitute this expansion for both quadratic forms in the pairwise score difference. The terms involving $x$ quadratically are $-\frac12x^\mathsf{T}P_kx+\frac12x^\mathsf{T}P_lx$, which combine to $-\frac12x^\mathsf{T}(P_k-P_l)x$. The remaining terms are at most linear in $x$, together with constants. This proves the stated coefficient and explains why distinct covariance matrices can make comparisons quadratic. $\square$

Worked Comparisons

Worked Example: A Determinant Creates a Radial Threshold

Consider two classes in two dimensions with equal priors and common mean $\mu_1=\mu_2=(0,0)^\mathsf{T}$. Let $\Sigma_1=I$ and $\Sigma_2=4I$. Write $r^2=x_1^2+x_2^2$. Determine the preferred class as a function of $r^2$.

For class 1, $|\Sigma_1|=1$ and $\Sigma_1^{-1}=I$, so $$ \delta_1(x)=-\frac12r^2. $$ For class 2, $|\Sigma_2|=|4I|=16$ and $\Sigma_2^{-1}=\frac14I$. Therefore $$ \delta_2(x)=-\frac12\log(16)-\frac12\left(\frac14r^2\right) =-\log(4)-\frac18r^2. $$ Equating the scores gives $$ -\frac12r^2=-\log(4)-\frac18r^2 \quad\Longrightarrow\quad r^2=\frac83\log(4). $$ The score difference is $\delta_1(x)-\delta_2(x)=\log(4)-\frac38r^2$. Thus class 1 wins when $r^2<\frac83\log(4)$, class 2 wins when $r^2>\frac83\log(4)$, and the scores tie at the threshold.

The determinant is essential here. If it were omitted, the scores would be $-r^2/2$ and $-r^2/8$. Class 2 would then win for every $r^2>0$, with a tie only at the origin. That comparison misses the actual threshold because it ignores the lower peak density associated with the more diffuse class.

Worked Example: Expanding Scores With Unequal Means and Covariances

Let $\mu_1=\mu_2=(\sqrt{2},0)^\mathsf{T}$, $\Sigma_1=I$, and $\Sigma_2=\operatorname{diag}(4,2)$. Assume equal priors. Compute $\delta_1(x)-\delta_2(x)$, then compare the classes at $x=(0,0)^\mathsf{T}$ and $x=(-2,0)^\mathsf{T}$.

The determinants are $|\Sigma_1|=1$ and $|\Sigma_2|=8$; the inverse of $\Sigma_2$ is $\operatorname{diag}(1/4,1/2)$. Expanding each score gives $$ \begin{aligned} \delta_1(x) &=-\frac12\left\{(x_1-\sqrt{2})^2+x_2^2\right\}\\ &=-\frac12x_1^2+\sqrt{2}x_1-1-\frac12x_2^2, \\[4pt] \delta_2(x) &=-\frac12\log(8)-\frac12\left\{\frac14(x_1-\sqrt{2})^2+\frac12x_2^2\right\}\\ &=-\frac12\log(8)-\frac18x_1^2+\frac{\sqrt{2}}4x_1-\frac14-\frac14x_2^2. \end{aligned} $$ Subtracting, including the constant terms, yields $$ \delta_1(x)-\delta_2(x) =\frac12\log(8)-\frac38x_1^2+\frac{3\sqrt{2}}4x_1-\frac34-\frac14x_2^2. $$ In particular, the constant from the squared mean terms is $-1+\frac14=-\frac34$, not $-\frac54$.

At the origin, the difference is $\frac12\log(8)-\frac34\approx 0.290$, which is positive, so class 1 has the larger score. At $(-2,0)^\mathsf{T}$, it is $$ \frac12\log(8)-\frac38(4)-\frac{3\sqrt{2}}2-\frac34 \approx 1.040-1.500-2.121-0.750=-3.332. $$ The difference is negative, so class 2 has the larger score at this point. The assignment changes because both the quadratic displacement terms and the determinant contribution enter the comparison.

Worked Example: A Non-Diagonal Covariance Produces a Cross-Product

Take zero means and equal priors, with $\Sigma_1=I$ and $$ \Sigma_2=\begin{pmatrix}2&1\\1&2\end{pmatrix}. $$ Compare the scores at the origin and at $(3,0)^\mathsf{T}$.

The determinant of $\Sigma_2$ is $3$, and $$ \Sigma_2^{-1}=\frac13\begin{pmatrix}2&-1\\-1&2\end{pmatrix}. $$ Consequently, $$ \begin{aligned} \delta_1(x)-\delta_2(x) &=\frac12\log(3)-\frac12(x_1^2+x_2^2) +\frac16(2x_1^2-2x_1x_2+2x_2^2)\\ &=\frac12\log(3)-\frac16x_1^2-\frac13x_1x_2-\frac16x_2^2. \end{aligned} $$ At the origin this is $\frac12\log(3)\approx0.549>0$, so class 1 wins. At $(3,0)^\mathsf{T}$, it is $\frac12\log(3)-\frac{9}{6}\approx-0.951$, so class 2 wins. The cross-product term comes from the off-diagonal entries of the inverse covariance matrix; in this particular point comparison it is zero because $x_2=0$.

What the Two Covariance Terms Do

The inverse covariance matrix determines how displacement from a class mean is measured. A direction with relatively large variance contributes less to the quadratic distance, while a direction with relatively small variance contributes more. When a covariance has off-diagonal entries, its inverse can also produce cross-products such as $x_1x_2$, as in the third example.

The determinant term is not a separate preference that always favors one class. A larger determinant contributes a more negative $-\frac12\log|\Sigma_k|$, reflecting a lower overall density scale, but the quadratic term may favor that same class for observations far from its mean or in directions where its covariance is relatively large. The assignment is determined by the complete score, not by either term in isolation.

For computation, it is usually safer to compare the full scores or their pairwise difference than to reason informally from the means alone. A useful sequence is to calculate each determinant and inverse, form each quadratic form, retain all constants, and only then compare scores. In numerical work, direct linear solves and log-determinant routines are often preferable to explicitly forming matrix inverses, but the mathematical score is the same.

Common Pitfall: Dropping the Determinant

A tempting but incorrect argument is: “The quadratic form measures closeness to the mean, so I can assign the observation using only that term.” This fails whenever covariance determinants differ. In the first example, omitting the determinant makes class 2 win at every nonzero point, whereas the complete QDA rule favors class 1 near the common mean and switches to class 2 only beyond the calculated threshold. The correction is to retain $-\frac12\log|\Sigma_k|$ in every class score, including when the means coincide.

A second check is to keep track of score direction: the rule selects the largest score, not the smallest. The quadratic form itself is nonnegative, but it appears with a negative sign in the log-density score. Confusing the distance with the score can reverse a comparison.

Key takeaway: QDA compares class-specific log densities. Each score combines a determinant contribution, $-\frac12\log|\Sigma_k|$, with a covariance-adjusted quadratic contribution, $-\frac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)$. When covariance matrices differ, retaining both terms is essential.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

Why does taking logarithms of Gaussian class densities preserve the assignment that maximizes density?
For $\Sigma_1=I$ and $\Sigma_2=4I$ in two dimensions with a common zero mean, what threshold in $r^2$ separates the preferred classes?
In a pairwise score difference, which matrix determines the coefficient of the quadratic part?
Why can an off-diagonal covariance entry lead to a cross-product in a QDA score comparison?
What incorrect assignment pattern results in the first example if the determinant term is omitted?