Affine Invariance Of Gaussian Discriminant Rules

Why the Units and Coordinates Should Not Decide the Class

A Gaussian discriminant rule compares how well each class model fits a predictor vector, accounting for class priors. Changing the units from metres to centimetres, shifting the origin, or replacing coordinates by an invertible linear combination changes the numerical scores. But if the change is applied to every observation and every class model consistently, it should not change which class fits best. This is affine invariance.

The condition that the transformation be common to all classes matters: the transformed predictors must still describe the same observations in a shared coordinate system. We will show that LDA and QDA scores then differ from their original values only by terms that do not depend on the class. Such terms cannot change the maximizing class.

Affine Transformations and Gaussian Class Scores

Let \(X\in\mathbb{R}^p\) be a predictor vector, and suppose there are \(K\) classes. In class \(k\), assume \(X\sim N(\mu_k,\Sigma_k)\), where \(\Sigma_k\) is positive definite. Let \(\pi_k>0\) be the prior probability of class \(k\), with \(\sum_{k=1}^K\pi_k=1\). The Gaussian discriminant score, with the class-independent constant \(-\frac{p}{2}\log(2\pi)\) omitted, is

The rule assigns \(x\) to a class maximizing \(g_k(x)\). If several classes tie for the maximum, the rule returns that same set of maximizers; any fixed tie-breaking convention therefore gives the same assignment.

For LDA, the covariance matrix is shared: \(\Sigma_k=\Sigma\) for every class. For QDA, each class has its own positive definite \(\Sigma_k\). The result below covers both cases.

Proof. Write \(r_k=x-\mu_k\). The transformed residual is \(z-\mu_k^*=Ax+b-(A\mu_k+b)=Ar_k\). Since \(A\) is invertible,

It follows that the transformed quadratic term is exactly the original one:

Also, the determinant multiplication rule gives

Substituting these two identities into the transformed score leaves the quadratic term unchanged and subtracts \(\log|\det A|\) from the score. This subtraction is identical for every class. Thus, for any classes \(j\) and \(k\), \(g_j^*(Ax+b)-g_k^*(Ax+b)=g_j(x)-g_k(x)\). All pairwise score comparisons and ties are preserved, proving the result. \(\square\)

Why LDA Scores Also Retain Their Linear Form

The theorem uses the full Gaussian score, but LDA is often written with its quadratic term in \(x\) removed. When the covariance is shared, expand the squared distance in the score and discard terms common to every class. The resulting linear discriminant score is

Under the affine transformation, this simplified score does not necessarily stay numerically identical. Its change is nevertheless common to every class. To see this, set \(c=A^{-1}b\) and \(M=\Sigma^{-1}\). The transformed covariance inverse is \(A^{-\mathsf{T}}MA^{-1}\), so direct expansion gives

The last two terms depend on \(x\) and the transformation, but not on \(k\). Therefore, comparing the simplified LDA scores gives the same assignment. This is consistent with the theorem: the full density scores and the simplified linear scores omit different class-independent terms, but neither omission changes the winner.

Worked Examples

Fitted Rules and Practical Limits

The invariance also applies to the usual plug-in LDA and QDA rules when their estimates are transformed consistently. Suppose a training observation \(x_{ki}\) in class \(k\) is mapped to \(z_{ki}=Ax_{ki}+b\). The sample mean in that class transforms as

If the within-class sample covariance uses denominator \(n_k-1\), its transformed value is

A pooled covariance formed from these within-class covariance matrices transforms in the same way, because it is a weighted sum of them. Thus, if the transformed fitted rule uses these transformed means and covariances and the same priors, the theorem applies directly. The same conclusion holds for any covariance estimate that is itself equivariant under the chosen transformation; an estimate modified by a non-equivariant procedure need not obey the result.

The restriction to an invertible, common transformation is substantive. If \(A\) is singular, information may be discarded, and the transformed covariance is not positive definite in the original \(p\)-dimensional representation. If different transformations are applied to different classes, there is no single shared transformation of the observed predictor vector to which the theorem applies. And if predictors are transformed but the class means and covariances are left in their old coordinates, the scores no longer describe the same models.

Common Mistake

A frequent error is to compare ordinary Euclidean distances after rescaling and conclude that the classification must change. For example, stretching one coordinate changes Euclidean distances in that coordinate more than distances in other coordinates. But a Gaussian rule uses covariance-adjusted distances, not an unadjusted Euclidean comparison. Under a valid affine transformation, the covariance changes to \(A\Sigma_kA^\mathsf{T}\) at the same time, and the Mahalanobis quadratic term is exactly preserved.

The converse mistake is to claim that any preprocessing leaves predictions unchanged. It does not: the conclusion requires an invertible transformation applied equally to the observations in every class, with all model parameters transformed consistently. Discarding a predictor, transforming classes differently, or changing covariance estimates by a non-equivariant procedure falls outside the result.