Determinant Penalties In Plug-In QDA

Why Covariance Volume Enters a QDA Score

Quadratic discriminant analysis (QDA) allows each class to have its own covariance matrix. That flexibility affects classification in two ways: the covariance determines the quadratic distance from an observation to a class mean, and its determinant contributes a separate adjustment to the class score. The second part is easy to overlook. It matters because a class with a more spread-out distribution has its probability density distributed over a larger region.

In plug-in QDA, unknown class parameters are replaced by estimates from training data. We focus here on how the estimated covariance determinant affects the resulting classification, while keeping the quadratic-distance and prior contributions visible for comparison.

Definition: Suppose there are \(K\) classes, each modeled by a \(p\)-dimensional Gaussian distribution. For class \(k\), let \(\hat{\mu}_k\) be its estimated mean, \(\hat{\Sigma}_k\) its estimated covariance matrix, and \(\hat{\pi}_k\) its estimated prior probability. Assume \(\hat{\pi}_k>0\) and that \(\hat{\Sigma}_k\) is symmetric positive definite. The plug-in QDA score, with a term common to all classes omitted, is

\hat{\delta}_k(x) = \log \hat{\pi}_k -\frac{1}{2}\log|\hat{\Sigma}_k| -\frac{1}{2}(x-\hat{\mu}_k)^\mathsf{T} \hat{\Sigma}_k^{-1}(x-\hat{\mu}_k).

The rule assigns \(x\) to a class with the largest score. The term \(-\frac{1}{2}\log|\hat{\Sigma}_k|\) is the log-determinant adjustment; the final term is minus one-half the squared Mahalanobis distance to the estimated class mean.

The score follows by evaluating the Gaussian log density at \(x\), adding the log prior, and replacing the unknown parameters by their estimates. The omitted term is \(-\frac{p}{2}\log(2\pi)\), which is the same for every class when all classes use the same \(p\)-dimensional feature vector. It therefore cannot affect the ranking.

A larger covariance determinant corresponds to a larger covariance volume. Its log-determinant adjustment is consequently more negative, all else equal. This is a relative comparison: the determinant is not itself a probability or a penalty that can be interpreted independently of the other score terms.

Determinant as a Volume Adjustment

For a positive number \(c\), consider the set of points whose squared Mahalanobis distance from \(\mu\) under a positive definite covariance matrix \(\Sigma\) is at most \(c\). This set is an ellipsoid. Its volume is proportional to \(|\Sigma|^{1/2}\), so the determinant records its volume scale, not its orientation or every detail of its shape.

Result: Let \(\Sigma\) be a \(p\times p\) symmetric positive definite matrix, and let \(B_p\) be the unit ball in \(\mathbb{R}^p\), with volume \(V_p\). The ellipsoid

E_c=\{x:(x-\mu)^\mathsf{T}\Sigma^{-1}(x-\mu)\leq c\}

has volume \(V_p c^{p/2}|\Sigma|^{1/2}\).

Proof. By the spectral theorem, a positive definite \(\Sigma\) has a symmetric positive definite square root \(\Sigma^{1/2}\). Make the change of variables \(x=\mu+\sqrt{c}\,\Sigma^{1/2}u\). Then

(x-\mu)^\mathsf{T}\Sigma^{-1}(x-\mu) =c\,u^\mathsf{T}u, \qquad dx=c^{p/2}|\Sigma^{1/2}|\,du =c^{p/2}|\Sigma|^{1/2}\,du.

Thus \(x\in E_c\) exactly when \(u\in B_p\). Integrating the Jacobian over the unit ball gives \(\operatorname{Vol}(E_c)=c^{p/2}|\Sigma|^{1/2}V_p\), as claimed. \(\square\)

This volume connection explains the density normalization. If two classes have equal priors and an observation is equally far from their means in their respective Mahalanobis metrics, the class with the larger covariance volume has the lower density at that observation. QDA does not, however, classify by volume alone: the quadratic distances and priors remain part of the score.

Worked Example: Comparing Determinant Adjustments at the Means

In two dimensions, suppose two classes have equal priors and both estimated means are \((0,0)^\mathsf{T}\). Their estimated covariance matrices are

\hat{\Sigma}_1= \begin{pmatrix}4&1\\1&2\end{pmatrix}, \qquad \hat{\Sigma}_2= \begin{pmatrix}2&0\\0&2\end{pmatrix}.

At \(x=(0,0)^\mathsf{T}\), both quadratic distances are zero. The determinants are

|\hat{\Sigma}_1|=4(2)-1(1)=7, \qquad |\hat{\Sigma}_2|=2(2)-0=4.

Since the priors are equal, their log-prior contributions cancel. The remaining scores are

\hat{\delta}_1(x)=-\frac12\log 7\approx-0.973, \qquad \hat{\delta}_2(x)=-\frac12\log 4\approx-0.693.

Class 2 has the larger score and is selected. The quadratic distances provide no preference here; the smaller determinant gives class 2 the less negative log-determinant adjustment.

How the Determinant Can Change a Classification

For any two classes \(a\) and \(b\), subtracting their scores separates the contributions that determine the ranking:

\hat{\delta}_a(x)-\hat{\delta}_b(x) = \log\frac{\hat{\pi}_a}{\hat{\pi}_b} -\frac12\log\frac{|\hat{\Sigma}_a|}{|\hat{\Sigma}_b|} -\frac12\left(\hat{q}_a(x)-\hat{q}_b(x)\right),

where \(\hat{q}_k(x)=(x-\hat{\mu}_k)^\mathsf{T}\hat{\Sigma}_k^{-1}(x-\hat{\mu}_k)\). A positive difference favors class \(a\); a negative difference favors class \(b\). This form makes clear that a small advantage in quadratic distance can be outweighed by a larger determinant adjustment or a prior difference.

Result: If the priors are equal, class \(a\) outranks class \(b\) exactly when

\hat{q}_a(x)-\hat{q}_b(x) < \log\frac{|\hat{\Sigma}_b|}{|\hat{\Sigma}_a|}.

Proof. With equal priors, the log-prior difference is zero. Starting from \(\hat{\delta}_a(x)-\hat{\delta}_b(x)>0\), multiply by \(-2\), reversing the inequality:

\log\frac{|\hat{\Sigma}_a|}{|\hat{\Sigma}_b|} +\hat{q}_a(x)-\hat{q}_b(x)<0.

Rearranging gives the stated condition. Equality corresponds to a tie. \(\square\)

Worked Example: A Determinant Term Reverses the Distance Ranking

Let both classes have prior \(1/2\), let \(x=(0,0)^\mathsf{T}\), and use the following estimated means and covariances:

\hat{\mu}_1=(\sqrt{2.1},0)^\mathsf{T},\quad \hat{\Sigma}_1=I_2, \qquad \hat{\mu}_2=(\sqrt{8},0)^\mathsf{T},\quad \hat{\Sigma}_2=4I_2.

The determinants are \(1\) and \(16\). Since \((cI_2)^{-1}=c^{-1}I_2\), the quadratic distances are

\hat{q}_1(x)=2.1, \qquad \hat{q}_2(x)=\frac{8}{4}=2.0.

Distance alone would rank class 2 ahead because \(2.0<2.1\). But the full scores, omitting the equal log priors, are

\hat{\delta}_1(x)=-\frac12\log 1-\frac12(2.1)=-1.05, \qquad \hat{\delta}_2(x)=-\frac12\log 16-\frac12(2.0) =-\log 4-1\approx-2.386.

The full QDA rule selects class 1. Its quadratic distance is only \(0.1\) larger, while the determinant ratio contributes \(\frac12\log 16=\log 4\approx1.386\) in class 1's favor. This example shows why ranking classes by distance alone does not implement QDA.

What the Determinant Does Not Tell You

Equal determinants do not imply equal covariance matrices. Matrices can have the same volume scale while spreading variation in different directions. In that situation, determinant adjustments tie, but the quadratic distances can differ substantially. The determinant is therefore a volume summary; the inverse covariance in the quadratic term accounts for directional scaling.

Worked Example: Equal Volume, Different Quadratic Distances

Let two equally likely classes have means \((0,0)^\mathsf{T}\) and covariance estimates

\hat{\Sigma}_1= \begin{pmatrix}9&0\\0&1/9\end{pmatrix}, \qquad \hat{\Sigma}_2=I_2.

Both determinants equal one, since \(9(1/9)=1\). At \(x=(3,0)^\mathsf{T}\), the quadratic distances are

\hat{q}_1(x)=\frac{3^2}{9}+\frac{0^2}{1/9}=1, \qquad \hat{q}_2(x)=3^2+0^2=9.

The log-determinant terms and log priors tie, so the scores are \(\hat{\delta}_1(x)=-1/2\) and \(\hat{\delta}_2(x)=-9/2\). Class 1 is selected because its covariance allows more variation in the first coordinate, even though its overall volume matches that of class 2.

A change in the units of a feature changes covariance determinants. For instance, expressing one coordinate in different units rescales the corresponding covariance row and column. If the same invertible change of coordinates is applied to every class, each determinant receives the same multiplicative factor, so the common change in log determinant cancels in pairwise comparisons. Comparing determinants computed under inconsistent units, however, has no coherent interpretation.

Common Mistake and Practical Checks

Ignoring the determinant because the distance is smaller. The QDA score contains both terms. Compare complete scores, or use the pairwise score difference, before assigning a class.
Treating a larger determinant as automatically decisive. It contributes a relative adjustment, but a sufficiently better quadratic fit or a larger prior can outweigh it.
Calling the determinant the covariance shape. It summarizes volume scale. Covariances with the same determinant can have different directional spreads, as the previous calculation illustrates.
Using a singular covariance estimate without comment. The displayed plug-in score requires an inverse and a log determinant, so the covariance estimate must be positive definite. If it is singular, this formula is not defined; a suitable estimation or regularization choice must be made before applying it.

A reliable calculation keeps three contributions visible: \(\log\hat{\pi}_k\), \(-\frac12\log|\hat{\Sigma}_k|\), and \(-\frac12\hat{q}_k(x)\). This prevents the determinant adjustment from disappearing inside a numerical score and makes it possible to explain why a class won.

Check Your Understanding

For each question, distinguish the determinant contribution from the other parts of the QDA score.

For two equally likely classes with equal quadratic distances, which class has the larger score if its estimated covariance determinant is smaller?
Why does a covariance determinant describe volume scale but not the full shape of a covariance matrix?
With equal priors, state the condition under which class \(a\) outranks class \(b\) using their quadratic distances and determinants.
Can a class with a larger quadratic distance still be selected by QDA? Explain which other score contributions could make this happen.
What condition on an estimated covariance matrix is needed for the plug-in QDA score as written, and why?