Preserving LDA Decisions In The Discriminant Subspace

Why a Lower-Dimensional Representation Can Preserve LDA Decisions

A data set may have many predictors but only a few distinct directions in which class means differ. In that situation, LDA decisions need not depend on every coordinate separately. The key is to retain the directions that carry the class-mean differences in the geometry induced by the shared covariance matrix. Projecting onto the resulting discriminant subspace can reduce dimension while preserving every comparison between LDA scores.

This is a precise claim about the LDA score comparisons, not a claim that projection preserves all information about the predictors or every possible analysis. We will construct the subspace, derive the reduced scores, and prove that the full and reduced score differences are identical. The proof also identifies the zero-dimensional case, where no inverse of a zero-dimensional covariance matrix is needed.

The Discriminant Subspace and Projected LDA

Suppose there are \(K\) classes and \(p\) predictors. In class \(k\), assume the predictor vector has a multivariate normal distribution \(N(\mu_k,\Sigma)\), where the class means \(\mu_k\in\mathbb{R}^p\) may differ and the shared covariance matrix \(\Sigma\) is positive definite. Let the prior probabilities satisfy \(\pi_k>0\) and \(\sum_{k=1}^K\pi_k=1\). The LDA score for a predictor vector \(x\in\mathbb{R}^p\), omitting a term common to all classes, is

The LDA decision selects a class with the largest score. The class-mean differences determine the parts of these scores that vary across classes. Define the prior-weighted mean \(\bar{\mu}=\sum_{k=1}^K\pi_k\mu_k\), the between-class mean-difference space \(\mathcal{M}=\operatorname{span}\{\mu_k-\bar{\mu}:1\leq k\leq K\}\), and the discriminant subspace \(\mathcal{S}=\Sigma^{-1}\mathcal{M}\). Equivalently, \(\mathcal{S}\) is spanned by the vectors \(\Sigma^{-1}(\mu_k-\bar{\mu})\). Since \(\bar{\mu}\) is a weighted average of the class means, \(\mathcal{M}\) is also the span of all pairwise differences \(\mu_k-\mu_l\).

Let \(U\) be a \(p\times r\) matrix whose columns form an orthonormal basis for \(\mathcal{S}\), when \(r>0\). The projected predictor is \(z=U'x\). Under class \(k\), its mean is \(U'\mu_k\), and its covariance matrix is \(C=U'\Sigma U\). This matrix is positive definite: for any nonzero \(a\in\mathbb{R}^r\), \(a'Ca=(Ua)'\Sigma(Ua)>0\), because \(Ua\neq 0\) and \(\Sigma\) is positive definite. Thus the reduced LDA score is well defined:

This is the ordinary shared-covariance LDA score formed from the projected class distributions. If \(r=0\), the projected coordinate has no entries and \(C^{-1}\) must not be used. In that case all class means are equal, and we define the zero-dimensional projected scores directly as \(\delta_k^{(S)}=\log\pi_k\).

Exact Preservation of Score Comparisons

Proof. First consider \(r>0\), and set \(H=UC^{-1}U'\). The matrix \(H\) is symmetric because \(C\) is symmetric. For any \(d\in\mathcal{M}\), the vector \(\Sigma^{-1}d\) belongs to \(\mathcal{S}\), so there is an \(a\in\mathbb{R}^r\) such that \(\Sigma^{-1}d=Ua\). Therefore \(d=\Sigma Ua\) and \(U'd=U'\Sigma Ua=Ca\). It follows that

Now fix classes \(k,l\), and write \(d=\mu_k-\mu_l\), which belongs to \(\mathcal{M}\). The difference of the full linear score terms is \(x'\Sigma^{-1}d\). The difference of the projected linear terms is \(x'Hd\), because \(z=U'x\). These are equal by \(Hd=\Sigma^{-1}d\).

For the quadratic terms, use symmetry of \(H\) and \(\Sigma^{-1}\):

Thus the difference of the quadratic contributions, including their factor of \(-\tfrac12\), also agrees. The prior contribution is \(\log\pi_k-\log\pi_l\) in both score systems. Adding the linear, quadratic, and prior contributions proves the asserted equality for \(r>0\).

If \(r=0\), then \(\mathcal{M}=\{0\}\), so \(\mu_k=\mu_l\) for every pair. In the full scores the linear terms and quadratic terms therefore cancel in every pairwise difference, leaving \(\log\pi_k-\log\pi_l\). The defined zero-dimensional scores have exactly this difference. This proves the theorem in all cases. \(\square\)

The result is stronger than saying that the projected classifier usually agrees with the full classifier: every pairwise score difference is unchanged at every \(x\). The absolute scores can differ by a common term that is the same for all classes. A common shift cannot change which score is largest, nor can it change whether two scores tie.

Worked Examples

What the Projection Does—and Does Not—Guarantee

This dimension reduction matters when \(r\ll p\): the classifier can be represented using fewer coordinates without changing its LDA decisions. The bound \(r\leq K-1\) means that with a modest number of classes, the decision-relevant mean-difference space can be much smaller than the original predictor space. This is a statement about the specified shared-covariance LDA model and its score comparisons. It does not say that the discarded coordinates are irrelevant to other scientific questions, nor that arbitrary projections preserve decisions.

A frequent mistake is to project predictors onto an arbitrary subspace, fit a new LDA rule there, and assume that the old decisions must survive. The proof depends on choosing the subspace spanned by \(\Sigma^{-1}\) times the class-mean differences and using the projected covariance \(C=U'\Sigma U\). A projection that discards one of these directions can alter score differences. Another mistake is to treat the case \(r=0\) as though \(C^{-1}\) existed. When all class means coincide, the correct projected scores are simply the log priors; if priors are equal, all classes tie under LDA.

For computation, a practical check is to compare pairwise score differences before and after reduction, rather than comparing raw scores. Raw scores may differ by a shared additive term, as the multiclass example illustrates. Exact preservation of pairwise differences is the algebraic property that guarantees identical decisions.