Mahalanobis Geometry Of LDA

Why LDA Has a Distance Geometry

In linear discriminant analysis, each class has its own mean but shares a covariance matrix. The shared covariance determines how differences between observations should be measured: a change along a high-variance direction is less informative than the same-sized change along a low-variance direction. The resulting distance is the Mahalanobis distance. By transforming the coordinates to remove covariance, this distance becomes ordinary Euclidean distance, and the class means become centroids in the transformed space.

The geometric interpretation is exact under the Gaussian model. With equal class priors and equal misclassification costs, LDA assigns an observation to the class whose transformed centroid is nearest. With unequal priors, the same distance comparison is adjusted by a class-specific prior term; the nearest centroid need not win.

Positive definiteness matters: it ensures that \(\Sigma^{-1}\) exists and that the whitening transformation is invertible. One standard choice is the symmetric inverse square root \(W=\Sigma^{-1/2}\). Other valid choices can rotate or reflect the whitened coordinates, but they preserve the distances that determine classification.

From Gaussian Scores to Mahalanobis Distance

Suppose class \(k\) has prior probability \(\pi_k>0\), with \(\sum_k\pi_k=1\), and classification uses equal misclassification costs. The Bayes rule assigns \(x\) to a class maximizing \(\pi_k f_k(x)\). In the common-covariance Gaussian model, terms shared by all classes can be removed from the log of this quantity. The resulting LDA score is

The same score can be written using the squared Mahalanobis distance from \(x\) to the class mean, \(d_\Sigma^2(x,\mu_k)=(x-\mu_k)^T\Sigma^{-1}(x-\mu_k)\).

Proof. Expand the squared distance:

The first term, \(-\frac{1}{2}x^T\Sigma^{-1}x\), does not depend on \(k\). The remaining terms are exactly \(\delta_k(x)\). Therefore maximizing \(\delta_k(x)\) is equivalent to maximizing \(-\frac{1}{2}d_\Sigma^2(x,\mu_k)+\log\pi_k\), or, after multiplying by \(-1\), minimizing \(\frac{1}{2}d_\Sigma^2(x,\mu_k)-\log\pi_k\). When priors are equal, the prior term is also constant across classes, leaving only the distance comparison. \(\square\)

This theorem separates two ideas that are sometimes conflated. The shared covariance defines the distance scale; the priors can shift the decision in favor of a class even when it is farther away. Thus “nearest centroid” is the exact rule for equal priors, not a description of every prior-adjusted LDA decision.

Whitening Turns the Distance Into Euclidean Length

Let \(W\) be any whitening matrix. Since \(W\Sigma W^T=I_p\), its inverse satisfies \(W^TW=\Sigma^{-1}\). For the transformed observation \(z=Wx\) and transformed class centroid \(m_k=W\mu_k\), it follows that

Consequently, under equal priors, LDA in the original coordinates and nearest-centroid classification in whitened coordinates make the same choice. The word “centroid” here means the transformed class mean \(W\mu_k\); whitening transforms the observation and every class mean using the same matrix.

Proof. Define \(Q=W_2W_1^{-1}\). From \(W_1\Sigma W_1^T=I_p\), we have \(\Sigma=W_1^{-1}W_1^{-T}\). Therefore

Because \(Q\) is square, \(QQ^T=I_p\) implies that \(Q\) is orthogonal. Also \(W_2=QW_1\) by definition. Orthogonal transformations preserve Euclidean length: for any vector \(v\), \(\|Qv\|^2=v^TQ^TQv=v^Tv=\|v\|^2\). Applying this to each difference between a transformed observation and a transformed centroid proves the distance claim. \(\square\)

In the original coordinates, equal Mahalanobis distances form covariance-shaped contours: the sets \(x\) satisfying \((x-\mu)^T\Sigma^{-1}(x-\mu)=c\), for \(c>0\), are ellipsoids centered at \(\mu\). Whitening maps these ellipsoids to Euclidean spheres. This explains why the covariance changes the geometry of “near” without changing the underlying classification rule.

Worked Classification Examples

Interpreting the Geometry and Avoiding a Common Error

A Mahalanobis distance is a Euclidean distance after rescaling and mixing coordinates according to the covariance. If two coordinates are correlated, moving in the direction of their shared variation can be less unusual than moving by the same raw amount in a direction that departs from that pattern. Whitening accounts for this dependence rather than treating each coordinate as an independent scale adjustment.

For two class centroids with equal priors, the points equidistant from them in whitened coordinates satisfy \(\|z-m_1\|^2=\|z-m_2\|^2\). Expanding both squares cancels \(z^Tz\), leaving an equation linear in \(z\). The same equal-distance condition in the original coordinates is \(d_\Sigma^2(x,\mu_1)=d_\Sigma^2(x,\mu_2)\). This pairwise comparison is one way to see why the shared covariance is compatible with linear LDA scores.

The geometric view is useful for diagnosis as well as calculation. It indicates which directions contribute strongly to separation, clarifies why covariance pooling matters, and provides a transparent way to check a computed LDA decision. In practice, fitted class means, covariance matrices, and priors replace their population counterparts; the same formulas apply when the estimated common covariance is positive definite. If it is singular, the inverse-based distances and whitening transformation used here are not defined without additional methods or assumptions.