Why Small Calculation Errors Matter
LDA and QDA decisions can depend on several terms that are easy to drop or misplace. A missing prior may reverse an LDA decision; a missing determinant may make QDA favor a class whose density is actually lower at the observation. Using a covariance matrix where its inverse belongs, or changing a covariance divisor without recalculating the score, can also change the answer. Finally, even correct scores lead to the wrong classification if the comparison is reversed.
The safest check is to write down the score being used, identify which terms are common to all classes, and compare the resulting class scores explicitly. Throughout, the examples use equal misclassification costs, so the assigned class is the one with the largest posterior probability. With unequal costs, the decision rule must also account for those costs.
Score Formulas to Audit
Suppose \(X\) is a \(p\)-dimensional feature vector and the possible classes are indexed by \(k\). Under the Gaussian model, class \(k\) has prior probability \(\pi_k>0\), mean vector \(\mu_k\), and positive-definite covariance matrix \(\Sigma_k\). The corresponding precision matrix is \(\Sigma_k^{-1}\). LDA assumes a common covariance \(\Sigma_k=\Sigma\); QDA allows a different covariance in each class.
In these formulas, \(\log|\Sigma_k|\) is the log determinant of the covariance, and the quadratic form uses the precision matrix, not the covariance matrix. In LDA, the shared covariance determinant and the term involving \(x^\mathsf{T}\Sigma^{-1}x\) have been omitted because they are identical across classes. A prior term is not generally common across classes, so it cannot be dropped just because it is inconvenient.
Why the Formulas Work
A multivariate normal density with mean \(\mu_k\) and positive-definite covariance \(\Sigma_k\) is \(f_k(x)=(2\pi)^{-p/2}|\Sigma_k|^{-1/2}\exp\{-\tfrac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)\}\). Bayes' rule gives posterior class probabilities proportional to \(\pi_k f_k(x)\). Taking logarithms preserves their ordering because the logarithm is strictly increasing.
For QDA, the log of \(\pi_k f_k(x)\) is \(-\tfrac p2\log(2\pi)+\log\pi_k-\tfrac12\log|\Sigma_k|-\tfrac12(x-\mu_k)^\mathsf{T}\Sigma_k^{-1}(x-\mu_k)\). The first term is common across classes, and deleting it leaves exactly \(q_k(x)\). This proves that maximizing the stated QDA score gives the same class as maximizing the posterior probability.
For LDA, expand the quadratic term using the shared covariance: \((x-\mu_k)^\mathsf{T}\Sigma^{-1}(x-\mu_k)=x^\mathsf{T}\Sigma^{-1}x-2x^\mathsf{T}\Sigma^{-1}\mu_k+\mu_k^\mathsf{T}\Sigma^{-1}\mu_k\). In the log density, the first term on the right is common across classes. Removing it, along with the shared normalizing constant and shared log determinant, leaves \(\delta_k(x)\). Thus the LDA score also preserves the posterior ordering. In both derivations, only terms that are truly identical for every class may be discarded. \(\square\)
Worked Checks for Common Errors
Worked Example: Omitting a Prior Reverses an LDA Decision
Consider one-dimensional LDA with \(\mu_1=0\), \(\mu_2=2\), common variance \(\sigma^2=1\), and priors \(\pi_1=0.2\), \(\pi_2=0.8\). Classify \(x=0.4\).
The LDA scores are \(\delta_k(x)=x\mu_k/\sigma^2-\mu_k^2/(2\sigma^2)+\log\pi_k\). Therefore, \(\delta_1(0.4)=0+\log(0.2)\approx-1.6094\), while \(\delta_2(0.4)=0.4(2)-2^2/2+\log(0.8)=0.8-2+\log(0.8)\approx-1.4231\). Since \(-1.4231>-1.6094\), the correct classification is class 2.
If the priors are omitted, the scores become 0 for class 1 and \(-1.2\) for class 2, so the incorrect calculation assigns class 1. The observation is closer to class 1's mean, but closeness alone does not determine the posterior decision when the prior probabilities differ. Here the larger prior for class 2 changes the result.
Worked Example: Omitting the QDA Determinant Changes the Winner
Suppose both classes have mean 0, with variances \(\sigma_1^2=1\) and \(\sigma_2^2=4\), and equal priors. At \(x=1\), compare the QDA scores, omitting terms common to both classes.
For a one-dimensional class with variance \(\sigma_k^2\), the score is \(-\tfrac12\log\sigma_k^2-\tfrac12(x-\mu_k)^2/\sigma_k^2+\log\pi_k\). The equal log-prior terms cancel in the comparison. Thus \(q_1(1)=-\tfrac12\log(1)-\tfrac12(1)^2=-0.5\), and \(q_2(1)=-\tfrac12\log(4)-\tfrac12(1)^2/4\approx-0.6931-0.125=-0.8181\). The larger score is \(-0.5\), so the correct classification is class 1.
If the determinant term is dropped, the scores are \(-0.5\) and \(-0.125\). That calculation selects class 2, reversing the decision. The determinant is part of the normalizing constant of the Gaussian density: a more dispersed distribution has a lower peak density, and that information matters even when the quadratic terms are similar.
Worked Example: Using the Covariance Instead of the Precision
In two-dimensional LDA, take equal priors, \(\Sigma=\begin{pmatrix}9&0\\0&1\end{pmatrix}\), \(\mu_1=(0,0)^\mathsf{T}\), \(\mu_2=(1,2)^\mathsf{T}\), and \(x=(1,0.5)^\mathsf{T}\). Compare the two scores; the prior terms cancel.
The precision matrix is \(\Sigma^{-1}=\begin{pmatrix}1/9&0\\0&1\end{pmatrix}\). Since \(\mu_1=0\), \(\delta_1(x)=0\). For class 2, \(x^\mathsf{T}\Sigma^{-1}\mu_2=1/9+1=10/9\), and \(\mu_2^\mathsf{T}\Sigma^{-1}\mu_2=1/9+4=37/9\). Therefore \(\delta_2(x)=10/9-\tfrac12(37/9)=-17/18\approx-0.9444\). The correct decision is class 1.
A common substitution error is to use \(\Sigma\) in place of \(\Sigma^{-1}\) in both quadratic-score terms. It gives \(x^\mathsf{T}\Sigma\mu_2=9+1=10\) and \(\mu_2^\mathsf{T}\Sigma\mu_2=9+4=13\), hence the erroneous score \(10-\tfrac12(13)=3.5\). This would select class 2. The substantial change is not a rounding issue: covariance and precision encode different quantities, and the Gaussian quadratic form requires precision.
Worked Example: Changing a Covariance Divisor Changes the Decision
Suppose two fitted LDA class means are \(\mu_1=0\) and \(\mu_2=2\), the stated priors are \(\pi_1=0.25\) and \(\pi_2=0.75\), and \(x=0.5\). A pooled within-class sum of squares is \(Q=6\) for \(N=8\) observations in \(K=2\) classes. Compare the pooled covariance estimates using divisors \(N-K\) and \(N\).
The estimates are \(s^2_{N-K}=6/(8-2)=1\) and \(s^2_N=6/8=0.75\). For these two means, subtracting class 1's LDA score from class 2's gives \(\delta_2(x)-\delta_1(x)=\frac{2}{s^2}(x-1)+\log(0.75/0.25)\). Here \(\log(3)\approx1.0986\). With divisor \(N-K\), the difference is \(2(-0.5)+1.0986=0.0986\), so class 2 wins. With divisor \(N\), it is \((2/0.75)(-0.5)+1.0986\approx-0.2347\), so class 1 wins.
Each divisor defines a different estimate, and the example does not say that one convention is universally correct. The error is changing conventions partway through a calculation, or reporting a decision for one covariance estimate while using another in the score. State the estimator and its divisor, then use that same estimate consistently in every required term.
A Reliable Comparison Procedure
For a numerical classification, compute one score per class and choose the largest. Do not select the smallest score merely because the expression contains a negative quadratic term: the complete score, including priors and any class-specific determinant, is what must be compared. A simple audit separates the calculation into these steps.
Decide whether the calculation is LDA or QDA, and write its full score before simplifying.
Confirm the class means, priors, covariance matrices, and any stated covariance divisors.
Use \(\Sigma^{-1}\) in the quadratic form and \(\log|\Sigma_k|\) in QDA. Do not add class-specific determinant terms to the LDA comparison when the covariance is shared.
Evaluate every class score at the same \(x\), then choose the largest score under equal misclassification costs.
Common Mistakes and Why They Matter
- Dropping a prior as if it were a common constant. Priors can differ by class. In the first worked example, omitting them reversed the classification.
- Dropping QDA's determinant term. The determinant varies across classes when their covariance matrices differ, so it affects the density and may affect the winner.
- Replacing precision by covariance. The Gaussian distance is measured with \(\Sigma_k^{-1}\), not \(\Sigma_k\). A matrix that is appropriate for scaling variation is not interchangeable with its inverse.
- Mixing covariance divisors. A divisor changes the covariance estimate; even a scalar change to a shared LDA covariance can change the balance between the mean-separation term and the log-prior term.
- Choosing the smallest score. Under the score convention defined here, the largest score corresponds to the largest posterior probability. A negative quadratic component does not change that comparison rule.
These checks matter because a classification is a comparison, not an isolated calculation. A score that is numerically plausible can still be wrong if one class has been given an omitted term, a mismatched covariance estimate, or a different convention. When scores are close, verifying every term is especially important.
Check Your Understanding
Use the score definitions and calculation checks above to answer these questions.
- Which LDA terms can be removed from every class score without changing the classification, and what condition makes their removal valid?
- Why does the QDA score include a log determinant, and under what covariance condition could that determinant term be common across classes?
- In the covariance-versus-precision example, which class has the larger correct score, and which matrix belongs in the quadratic form?
- How can changing a pooled covariance divisor change an LDA decision even when the class means and priors remain fixed?
- Under equal misclassification costs and the score convention used here, should the selected class have the largest or smallest score?