Gaussian Discriminant Scores From Log Densities

From Gaussian Densities to Class Scores

A Gaussian classification rule compares how well each candidate class accounts for an observed vector. The density supplies a measure of fit, while the class prior probability accounts for how common the class is before seeing the observation. Taking logarithms turns the product of these quantities into a sum whose separate contributions are easier to inspect.

Suppose \(C\) takes values \(1,\ldots,K\), and the observed feature vector is \(x\in\mathbb{R}^p\). Assume the prior probabilities are \(\pi_k=P(C=k)>0\), with \(\sum_{k=1}^K\pi_k=1\), and that \(X\mid C=k\sim N_p(\mu_k,\Sigma_k)\). Each \(\mu_k\) is a \(p\)-vector and each \(\Sigma_k\) is a symmetric positive-definite \(p\times p\) matrix. Positive definiteness ensures that the Gaussian density exists, its determinant is positive, and its inverse is defined.

The full log of the joint class-and-observation density is \(\log\{\pi_k f_k(x)\}\), where \(f_k\) is the \(N_p(\mu_k,\Sigma_k)\) density. It includes the additional term \(-\frac p2\log(2\pi)\). Since the dimension \(p\) is the same for every class, that term does not affect which score is largest. The distinction between a full log joint density and the reduced score matters when interpreting values: reduced scores are useful for comparison, but are not themselves log posterior probabilities.

Deriving and Reading the Score

The multivariate normal density is \(f_k(x)=(2\pi)^{-p/2}|\Sigma_k|^{-1/2}\exp\{-\frac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)\}\). Taking its logarithm and adding the log prior gives

The quadratic form is nonnegative. It measures discrepancy from the class mean after adjusting for the covariance scale and correlation. A large value lowers the score. The determinant term comes from the density’s normalizing factor: a larger covariance determinant contributes a larger penalty, all else equal. The prior contributes \(\log\pi_k\), so a more probable class receives a higher score before the feature vector is considered.

The score can also be expanded as a polynomial in \(x\). Because \(\Sigma_k^{-1}\) is symmetric, \((x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k) =x^\mathsf{T}\Sigma_k^{-1}x-2x^\mathsf{T}\Sigma_k^{-1}\mu_k+\mu_k^\mathsf{T}\Sigma_k^{-1}\mu_k\). Substitution yields

This form makes the roles of the terms explicit. The first is quadratic in the observation, the second is linear in it, and the remaining terms are constant with respect to \(x\) for a fixed class. Across classes, however, those constants need not agree. In particular, neither the determinant nor the prior may be dropped merely because it does not vary with \(x\).

Proof. By Bayes’ rule, for each class \(k\), \(P(C=k\mid X=x)=\pi_k f_k(x)\big/\sum_{j=1}^K\pi_j f_j(x)\). The denominator is positive and does not depend on \(k\), so it cannot change the maximizing class. The logarithm is strictly increasing, hence maximizing the posterior is equivalent to maximizing \(\log\{\pi_k f_k(x)\}\). From the density expansion, this log joint density equals \(\delta_k(x)-\frac p2\log(2\pi)\). The subtracted term is also independent of \(k\), proving the claim. \(\square\)

Why the Determinant and Prior Matter

The quadratic distance alone does not represent the Gaussian density. A class with a more dispersed distribution can place an observation relatively close in its own covariance-scaled geometry, but its density also has a normalizing factor involving \(|\Sigma_k|\). Likewise, the likelihood alone ignores the prior probability of encountering the class. A complete comparison retains every term that varies across classes.

Common Mistakes and a Reliable Comparison

A useful way to organize a calculation is to evaluate the five terms in the expanded score for each class, or to calculate the three terms in the centered form. Keep one form consistently; the expanded and centered forms are algebraically identical.

A frequent mistake is to discard every term that is constant in \(x\). That reasoning confuses “constant as \(x\) varies” with “the same for all classes.” The determinant, prior, and mean-dependent terms are constant in \(x\), but generally differ by class, so removing them can change the assignment. Only terms that are common to every class at the observation being evaluated can be dropped without changing the maximizing class.

Another mistake is to compare raw quadratic distances when covariance matrices differ. The relevant distance is scaled by the class-specific inverse covariance, and even that is only one part of the score. The determinant and prior complete the comparison. When all class parameters are specified, retaining these contributions makes the classification calculation transparent and checkable.