Fisher’s Criterion For Separating Class Means

Why Project the Data?

A pair of class means can be far apart in some coordinate yet still be difficult to distinguish if observations vary greatly in that direction. Conversely, a more modest mean difference can be informative when within-class variation along it is small. Fisher’s criterion makes this comparison explicit for a one-dimensional projection: it rewards separation of the projected class means and penalizes variation within the projected classes.

The direction is a vector $a$, and the projected observation is the scalar $a'X$. Multiplying $a$ by any nonzero constant changes the scale or sign of that scalar but not the quality of the separation, so the criterion identifies a direction rather than a unique vector.

Between-Class and Within-Class Variation

Let $G$ be a class label taking values 1 and 2, with $P(G=k)=\pi_k>0$ and $\pi_1+\pi_2=1$. Write the conditional means as $\mu_k=E[X\mid G=k]$ and conditional covariance matrices as $\Sigma_k=\operatorname{Cov}(X\mid G=k)$. Define the mean difference by $\delta=\mu_1-\mu_2$; this orientation will be used consistently below.

Definition: For a nonzero projection vector $a$, the projected class means are $a'\mu_1$ and $a'\mu_2$. Their weighted between-class variance is $\pi_1\pi_2\{a'(\mu_1-\mu_2)\}^2$. The weighted within-class covariance matrix is $\Sigma_W=\pi_1\Sigma_1+\pi_2\Sigma_2$, so the within-class variance of the projection is $a'\Sigma_Wa$. When $\Sigma_W$ is positive definite, Fisher’s two-class criterion is $$ J(a)=\frac{\pi_1\pi_2(a'\delta)^2}{a'\Sigma_Wa}. $$

The factor $\pi_1\pi_2$ follows directly from the variance of a variable that takes the two class-mean values $a'\mu_1$ and $a'\mu_2$ with probabilities $\pi_1$ and $\pi_2$. Its variance is $\pi_1\pi_2\{a'\mu_1-a'\mu_2\}^2$. The denominator averages the conditional variances of the projection across the two classes.

If the classes have a common within-class covariance matrix $\Sigma$, then $\Sigma_W=\Sigma$. The criterion is unchanged in its maximizing direction if the positive factor $\pi_1\pi_2$ is omitted, but retaining it gives the ratio its direct interpretation as weighted between-class variance divided by weighted within-class variance.

Worked Example: Calculating the Criterion and Checking the Direction

Suppose $\mu_1=(0,0)'$, $\mu_2=(2,1)'$, and both classes have covariance matrix $\Sigma_W=\begin{pmatrix}2&1\\1&1\end{pmatrix}$. Take $\pi_1=0.4$ and $\pi_2=0.6$. Find the maximizing direction and compare it with $a=(1,1)'$.

By the definition $\delta=\mu_1-\mu_2$, the mean difference is $\delta=(-2,-1)'$. The determinant of $\Sigma_W$ is $2(1)-1(1)=1$, so $$ \Sigma_W^{-1}=\begin{pmatrix}1&-1\\-1&2\end{pmatrix}, \qquad \Sigma_W^{-1}\delta = \begin{pmatrix}1&-1\\-1&2\end{pmatrix} \begin{pmatrix}-2\\-1\end{pmatrix} = \begin{pmatrix}-1\\0\end{pmatrix}. $$

Thus a maximizing direction is $a=(-1,0)'$; its negative $(1,0)'$ gives the same criterion. For $a=(-1,0)'$, $a'\delta=2$, and $\Sigma_Wa=(-2,-1)'$, giving $a'\Sigma_Wa=2$. Therefore $$ J(a)=\frac{(0.4)(0.6)(2)^2}{2}=0.48. $$

For the alternative $a=(1,1)'$, $a'\delta=-3$, $\Sigma_Wa=(3,2)'$, and $a'\Sigma_Wa=5$. Hence $$ J(a)=\frac{(0.4)(0.6)(-3)^2}{5}=0.432. $$ The candidate direction has a substantial projected mean difference, but its within-class variation is also larger relative to that difference. It does not attain the maximum.

The Maximizing Direction

Theorem: Suppose $\Sigma_W$ is symmetric positive definite and $\delta\ne 0$. Then $J(a)$, over all nonzero $a$, is maximized precisely by nonzero scalar multiples of $\Sigma_W^{-1}\delta$. The maximum is $$ \max_{a\ne 0}J(a)=\pi_1\pi_2\,\delta'\Sigma_W^{-1}\delta. $$

Positive definiteness ensures that the denominator $a'\Sigma_Wa$ is positive for every nonzero $a$, and that the inverse and positive-definite square root used below exist. To prove the result, set $b=\Sigma_W^{1/2}a$ and $c=\Sigma_W^{-1/2}\delta$. Then $a'\delta=b'c$ and $a'\Sigma_Wa=b'b$. The Cauchy–Schwarz inequality gives $$ J(a)=\pi_1\pi_2\frac{(b'c)^2}{b'b} \leq \pi_1\pi_2\frac{(b'b)(c'c)}{b'b} =\pi_1\pi_2\delta'\Sigma_W^{-1}\delta. $$

Equality in Cauchy–Schwarz holds exactly when $b$ is a nonzero scalar multiple of $c$. Since $b=\Sigma_W^{1/2}a$ and $c=\Sigma_W^{-1/2}\delta$, this is equivalent to $a$ being a nonzero scalar multiple of $\Sigma_W^{-1}\delta$. Substituting that direction into the bound attains the displayed maximum. $\square$

If $\delta=0$, then $a'\delta=0$ for every $a$, so $J(a)=0$ for every nonzero direction: the class means are not separated by any linear projection. If $\Sigma_W$ is singular, the theorem as stated does not apply, because the inverse need not exist and some nonzero directions may have zero within-class variance. The positive-definite condition is therefore substantive, not merely a technical convenience.

Worked Example: Unequal Covariances and a Pooled Within-Class Matrix

Let $\pi_1=\pi_2=0.5$, $\delta=(3,1)'$, $\Sigma_1=\begin{pmatrix}1&0\\0&4\end{pmatrix}$, and $\Sigma_2=\begin{pmatrix}4&0\\0&1\end{pmatrix}$. Find the Fisher direction and the maximum criterion.

The weighted within-class covariance is $$ \Sigma_W=0.5\Sigma_1+0.5\Sigma_2 =\begin{pmatrix}2.5&0\\0&2.5\end{pmatrix}, \qquad \Sigma_W^{-1}=\begin{pmatrix}0.4&0\\0&0.4\end{pmatrix}. $$ Thus $\Sigma_W^{-1}\delta=(1.2,0.4)'=0.4(3,1)'$, so $a=(3,1)'$ is a maximizing direction.

For this vector, $a'\delta=10$ and $a'\Sigma_Wa=2.5(3^2+1^2)=25$. The criterion is $$ J(a)=\frac{(0.5)(0.5)(10)^2}{25}=1. $$ The theorem gives the same value: $\delta'\Sigma_W^{-1}\delta=0.4(3^2+1^2)=4$, and $\pi_1\pi_2(4)=0.25(4)=1$. This example uses a weighted pooled within-class covariance; it does not assume that the two class covariance matrices are identical.

Interpretation for Two Gaussian Classes

Suppose $X\mid G=k\sim N(\mu_k,\Sigma)$ for $k=1,2$, where $\Sigma$ is positive definite and common to both classes. For any fixed $a$, the projected observations are normal: $$ a'X\mid G=k\sim N(a'\mu_k,\ a'\Sigma a). $$ The squared difference between their projected means, divided by their common projected variance, is $(a'\delta)^2/(a'\Sigma a)$. Fisher’s criterion is this squared standardized mean gap multiplied by $\pi_1\pi_2$. Its maximizing direction has maximum $$ \max_{a\ne0}J(a)=\pi_1\pi_2\,\delta'\Sigma^{-1}\delta. $$ The factor $\delta'\Sigma^{-1}\delta$ is the squared Mahalanobis distance between the means. Thus Fisher’s direction measures mean separation after accounting for the covariance structure, rather than treating each coordinate as equally variable and independent.

Worked Example: Standardized Separation on the Gaussian Projection

Use the common-covariance Gaussian model from the first example: $\mu_1=(0,0)'$, $\mu_2=(2,1)'$, and $\Sigma=\begin{pmatrix}2&1\\1&1\end{pmatrix}$. With $a=(-1,0)'$, the projected class means are $a'\mu_1=0$ and $a'\mu_2=-2$. The common projected variance is $$ a'\Sigma a=(-1,0)\begin{pmatrix}-2\\-1\end{pmatrix}=2. $$

The absolute mean gap is $2$, so the squared standardized gap is $2^2/2=2$. This agrees with $\delta'\Sigma^{-1}\delta=2$, calculated from $\delta=(-2,-1)'$ and $\Sigma^{-1}\delta=(-1,0)'$. With equal class probabilities, the criterion’s maximum would therefore be $0.25(2)=0.5$; with probabilities $0.4$ and $0.6$, it is $0.24(2)=0.48$.

For instance, with equal class probabilities, the midpoint of the two projected means is $-1$. A rule that assigns a projection below $-1$ to class 2 has error probability $\Phi(-1/\sqrt{2})$ in either class, where $\Phi$ is the standard normal distribution function. This illustrates that the standardized gap describes separation, while a classification threshold and error calculation also depend on the decision rule and class probabilities.

Why the Criterion Matters—and a Common Pitfall

The criterion is useful because it converts a multivariate mean-separation question into a scale-free comparison along each candidate direction. A raw mean difference alone does not account for covariance: variation in the same direction as the difference weakens separation, while covariance in other directions need not have the same effect. The inverse covariance in $\Sigma_W^{-1}\delta$ performs this adjustment.

A common mistake is to say that the best direction must point along $\delta$, because that direction gives the largest unadjusted projected mean difference. That reasoning ignores the denominator. The direction $\delta$ need not maximize $(a'\delta)^2/(a'\Sigma_Wa)$ unless the covariance structure makes it appropriate. The derivation shows that the maximizing direction is $\Sigma_W^{-1}\delta$, which can differ in orientation from $\delta$. Another frequent error is to treat the maximizing vector as unique: replacing it by any nonzero multiple, including its negative, leaves the criterion unchanged.

Key takeaway: Fisher’s two-class criterion compares weighted between-class variation with weighted within-class variation after projection. With positive-definite within-class covariance and distinct class means, its maximizing direction is proportional to $\Sigma_W^{-1}(\mu_1-\mu_2)$, and its maximum is the weighted squared Mahalanobis separation of the class means.

Check Your Understanding

Use the definitions and derivation above to answer these questions.

Why does the weighted between-class variance of the projected class means include the factor $\pi_1\pi_2$?
If $\delta$ is replaced by $-\delta$ because the class labels are swapped, what happens to the criterion and maximizing direction?
For what condition on $\Sigma_W$ does the maximizing-direction theorem apply, and why is that condition useful?
In the common-covariance Gaussian case, what does $\delta'\Sigma^{-1}\delta$ represent in terms of projected means and variance?
Why can the direction $\delta$ fail to maximize Fisher’s criterion even though it directly follows the difference in class means?