Equivalence Of Two-Class LDA And Fisher Discrimination

Why Compare Fisher Discrimination and LDA?

Fisher discrimination and linear discriminant analysis (LDA) arrive at a linear direction from different starting points. Fisher’s criterion seeks a projection that separates class means relative to within-class variation. LDA compares class-conditional probability densities under a common-covariance model. In the two-class case, these procedures select the same direction, up to a nonzero scalar. But a direction is not yet a classification rule: a threshold on that direction is also needed.

The distinction matters in practice. The direction describes which linear combination of the measurements is used to order observations. The threshold determines where that ordering is split into class assignments. Under Gaussian LDA, class means, priors, and the common covariance contribute to the resulting rule; Fisher’s criterion by itself does not specify a prior-dependent threshold.

Definition: Let $X\in\mathbb{R}^p$ be an observation with class label $Y\in\{1,2\}$. Write $\mu_k=E[X\mid Y=k]$ for the class mean and $\Sigma_k=\operatorname{Cov}(X\mid Y=k)$ for the class covariance. Assume the two classes have a common positive-definite covariance matrix $\Sigma_1=\Sigma_2=\Sigma$, and write $\delta=\mu_1-\mu_2$. For a nonzero vector $w$, the two-class Fisher criterion is $$ J(w)=\frac{\{w^\mathsf{T}(\mu_1-\mu_2)\}^2}{w^\mathsf{T}\Sigma w}. $$ It measures squared separation of the projected class means relative to projected within-class variance. A Fisher direction is any nonzero vector maximizing $J(w)$.

The positive-definiteness assumption ensures that $w^\mathsf{T}\Sigma w>0$ for every nonzero $w$, so the criterion is defined. The classes must also have distinct means, $\delta\ne 0$, for the maximizing direction to be informative. Some definitions of Fisher’s criterion use a within-class scatter matrix or multiply the numerator by a positive constant. Such changes in scale do not change the maximizing direction when the within-class matrix is proportional to $\Sigma$.

The Fisher Direction

The numerator of $J(w)$ rewards directions along which the projected means differ. The denominator penalizes directions with large within-class variance. Thus, maximizing the criterion is not generally the same as pointing directly from one mean to the other: the covariance determines how that difference should be weighted.

Theorem: If $\Sigma$ is positive definite and $\delta\ne 0$, the maximizers of the two-class Fisher criterion are precisely the nonzero scalar multiples of $\Sigma^{-1}\delta$. The maximum value is $\delta^\mathsf{T}\Sigma^{-1}\delta$.

Proof. Let $\Sigma^{1/2}$ be the symmetric positive-definite square root of $\Sigma$, and set $z=\Sigma^{1/2}w$ and $a=\Sigma^{-1/2}\delta$. Then $w=\Sigma^{-1/2}z$, so $$ w^\mathsf{T}\delta=z^\mathsf{T}a,\qquad w^\mathsf{T}\Sigma w=z^\mathsf{T}z. $$ Consequently, $$ J(w)=\frac{(z^\mathsf{T}a)^2}{z^\mathsf{T}z} \leq \frac{(z^\mathsf{T}z)(a^\mathsf{T}a)}{z^\mathsf{T}z} =a^\mathsf{T}a =\delta^\mathsf{T}\Sigma^{-1}\delta, $$ where the inequality is Cauchy–Schwarz. Equality holds exactly when $z$ is a nonzero scalar multiple of $a$. In that case, $$ w=\Sigma^{-1/2}z =c\Sigma^{-1/2}a =c\Sigma^{-1}\delta $$ for some $c\ne 0$. Conversely, every such $w$ gives equality. This proves both the maximizing direction and the maximum value. $\square$

Worked Example: Finding the Fisher Direction

Suppose $\mu_1=(2,1)^\mathsf{T}$, $\mu_2=(0,0)^\mathsf{T}$, and $$ \Sigma=\begin{pmatrix}4&1\\1&2\end{pmatrix}. $$ Find the Fisher direction and evaluate its criterion.

The mean difference is $\delta=(2,1)^\mathsf{T}$. The determinant of $\Sigma$ is $4(2)-1(1)=7$, so $$ \Sigma^{-1}=\frac{1}{7}\begin{pmatrix}2&-1\\-1&4\end{pmatrix}, \qquad \Sigma^{-1}\delta=\frac{1}{7}\begin{pmatrix}3\\2\end{pmatrix}. $$ Thus $(3,2)^\mathsf{T}$ is a convenient representative of the Fisher direction. Its projected mean difference is $3(2)+2(1)=8$. Also, $$ \Sigma\begin{pmatrix}3\\2\end{pmatrix} =\begin{pmatrix}14\\7\end{pmatrix}, \qquad \begin{pmatrix}3&2\end{pmatrix}\Sigma\begin{pmatrix}3\\2\end{pmatrix}=56. $$ Therefore $J((3,2)^\mathsf{T})=8^2/56=8/7$. This agrees with the theorem, since $\delta^\mathsf{T}\Sigma^{-1}\delta=(2,1)(3/7,2/7)^\mathsf{T}=8/7$.

The LDA Direction and Its Threshold

For the LDA comparison, add the model assumption that $X\mid Y=k\sim N(\mu_k,\Sigma)$, with prior probabilities $\pi_k=P(Y=k)>0$. The common-covariance Gaussian discriminant score, up to terms shared by both classes, is $g_k(x)=x^\mathsf{T}\Sigma^{-1}\mu_k-\tfrac12\mu_k^\mathsf{T}\Sigma^{-1}\mu_k+\log\pi_k$. The usual Bayes rule assigns an observation to the class with the larger score when misclassification costs are equal.

Theorem: Under the common-covariance Gaussian model with positive class priors, the difference of the two LDA scores is $$ g_1(x)-g_2(x) =x^\mathsf{T}\Sigma^{-1}\delta -\frac12\left(\mu_1^\mathsf{T}\Sigma^{-1}\mu_1-\mu_2^\mathsf{T}\Sigma^{-1}\mu_2\right) +\log\frac{\pi_1}{\pi_2}. $$ Its coefficient vector for $x$ is $\Sigma^{-1}\delta$. Thus LDA and two-class Fisher discrimination have proportional directions. LDA classifies to class 1 when this score difference is positive and to class 2 when it is negative; equality is a decision boundary.

Derivation. The Gaussian log-density for class $k$, including its prior, is $$ \log\pi_k-\frac{p}{2}\log(2\pi)-\frac12\log|\Sigma| -\frac12(x-\mu_k)^\mathsf{T}\Sigma^{-1}(x-\mu_k). $$ Expand the quadratic term: $$ -\frac12 x^\mathsf{T}\Sigma^{-1}x +x^\mathsf{T}\Sigma^{-1}\mu_k -\frac12\mu_k^\mathsf{T}\Sigma^{-1}\mu_k. $$ The terms $-\tfrac{p}{2}\log(2\pi)$, $-\tfrac12\log|\Sigma|$, and $-\tfrac12x^\mathsf{T}\Sigma^{-1}x$ are the same for both classes. Removing them leaves $g_k(x)$. Subtracting $g_2(x)$ from $g_1(x)$ gives the stated expression, whose term linear in $x$ is $x^\mathsf{T}\Sigma^{-1}\delta$. Hence its direction is the Fisher direction. $\square$

The intercept in this score difference is essential. It sets the threshold along the shared direction. In particular, changing the priors changes the intercept but not the coefficient vector $\Sigma^{-1}\delta$. Rescaling a direction also rescales its threshold: if a rule is written $w^\mathsf{T}x>t$, replacing $w$ by $cw$ for $c>0$ requires replacing $t$ by $ct$ to leave the rule unchanged. A negative $c$ reverses the inequality as well as the orientation.

Worked Example: Equal Priors Set a Projected Midpoint Threshold

Use the means and covariance from the first example, with equal priors. Find the LDA boundary using $w=(3,2)^\mathsf{T}$.

The score difference has linear term $x^\mathsf{T}\Sigma^{-1}\delta=(3x_1+2x_2)/7$. Its quadratic-mean term is $\tfrac12(\mu_1^\mathsf{T}\Sigma^{-1}\mu_1-\mu_2^\mathsf{T}\Sigma^{-1}\mu_2)=\tfrac12(8/7)=4/7$; the prior term is zero. Thus class 1 is selected when $$ \frac{3x_1+2x_2}{7}-\frac47>0, \quad\text{or equivalently}\quad 3x_1+2x_2>4. $$ The projected class means are $w^\mathsf{T}\mu_1=8$ and $w^\mathsf{T}\mu_2=0$, so the threshold 4 is their midpoint. For example, $x=(1,0)^\mathsf{T}$ has projection 3 and is assigned to class 2, whereas $x=(2,0)^\mathsf{T}$ has projection 6 and is assigned to class 1. The direction alone gives neither of these assignments; the threshold completes the rule.

Worked Example: Changing Priors Moves the Boundary, Not the Direction

Keep the same means and covariance, but take $\pi_1=0.8$ and $\pi_2=0.2$. Find the new threshold in the coordinate $q=x^\mathsf{T}\Sigma^{-1}\delta$, and classify $x=(0,0)^\mathsf{T}$.

The direction remains $\Sigma^{-1}\delta=(3/7,2/7)^\mathsf{T}$, and the score difference becomes $$ g_1(x)-g_2(x)=q-\frac47+\log\frac{0.8}{0.2} =q-\frac47+\log 4. $$ Class 1 is selected when $q>4/7-\log 4$, approximately $q>-0.815$. For $x=(0,0)^\mathsf{T}$, $q=0$, and the score difference is $-4/7+\log 4$, approximately $0.815>0$, so LDA assigns the observation to class 1. With equal priors the same point has score difference $-4/7<0$ and is assigned to class 2. The prior change has shifted the boundary without changing the Fisher/LDA direction.

Worked Example: Covariance Changes the Direction and Threshold

Let $\mu_1=(2,3)^\mathsf{T}$, $\mu_2=(0,0)^\mathsf{T}$, and $\Sigma=\begin{pmatrix}1&0\\0&9\end{pmatrix}$, with equal priors. Find a Fisher direction, its criterion value, and the LDA rule. Then classify $x=(0,6)^\mathsf{T}$.

Here $\delta=(2,3)^\mathsf{T}$ and $\Sigma^{-1}=\operatorname{diag}(1,1/9)$, giving $\Sigma^{-1}\delta=(2,1/3)^\mathsf{T}$. A proportional direction is $w=(6,1)^\mathsf{T}$. Its projected mean difference is $6(2)+1(3)=15$, while $w^\mathsf{T}\Sigma w=36+9=45$. Hence $J(w)=225/45=5$, also equal to $\delta^\mathsf{T}\Sigma^{-1}\delta=4+1=5$.

The score difference is $$ x^\mathsf{T}\Sigma^{-1}\delta-\frac12\delta^\mathsf{T}\Sigma^{-1}\delta =2x_1+\frac{x_2}{3}-\frac52. $$ Equivalently, class 1 is selected when $6x_1+x_2>7.5$. For $x=(0,6)^\mathsf{T}$, the left side is 6, so the rule assigns class 2. In contrast, ordinary Euclidean distance from this point to $\mu_1$ is $\sqrt{(0-2)^2+(6-3)^2}=\sqrt{13}$, less than its distance 6 to $\mu_2$. That nearest-mean comparison is not the shared-covariance LDA rule: the larger within-class variance in the second coordinate affects the covariance-weighted direction and decision.

What the Equivalence Does—and Does Not—Say

The equivalence is precise but limited. It says that, for two classes and a common positive-definite within-class covariance, the Fisher maximizing direction is proportional to the coefficient vector in the Gaussian LDA score difference. It does not say that Fisher’s criterion alone supplies an intercept, a threshold, class priors, or a complete classification rule. Nor does the direction require Gaussian class distributions: the Fisher optimization uses means and within-class covariance, while the Gaussian assumption is used to obtain the LDA score and its threshold.

The direction is also unchanged by multiplying $\Sigma$ by a positive scalar, since $(c\Sigma)^{-1}\delta=c^{-1}\Sigma^{-1}\delta$ for $c>0$. This changes only the scale of the vector. More generally, any two nonzero scalar multiples encode the same one-dimensional axis, although a classification inequality must be rescaled consistently. These facts explain why direction comparisons should be made up to proportionality rather than by demanding identical coordinates.

Common Mistake

Using the raw difference of means as the direction. The mean difference $\delta$ is the direction only in special covariance settings, such as $\Sigma=cI$ for $c>0$. In general the direction is $\Sigma^{-1}\delta$, which accounts for within-class variation and correlation.
Calling a direction a classifier. A vector $w$ gives the projection $w^\mathsf{T}x$, not the cut point. A classifier also needs a threshold, which in LDA depends on the means and priors.
Assuming priors rotate the direction. In common-covariance two-class Gaussian LDA, priors enter the constant term $\log(\pi_1/\pi_2)$. They shift the boundary but do not change the coefficient $\Sigma^{-1}\delta$.
Comparing vectors without allowing for scale. The vectors $\Sigma^{-1}\delta$ and $7\Sigma^{-1}\delta$ represent the same oriented direction. But if a threshold is expressed using one vector, it must be scaled when the vector is rescaled.

Key takeaway: With two classes and common positive-definite covariance, Fisher’s criterion and Gaussian LDA select the direction $\Sigma^{-1}(\mu_1-\mu_2)$, up to scale. Fisher separation identifies the axis; LDA’s intercept, including its prior contribution, determines where that axis is divided into classifications.

Check Your Understanding

Use the direction–threshold distinction to answer the following questions.

What assumptions ensure that the two-class Fisher direction is proportional to the common-covariance LDA direction?
Why does the Fisher criterion have the same value for every nonzero scalar multiple of a maximizing direction?
If the class priors change but the class means and common covariance do not, which part of the LDA score difference changes?
Under what covariance condition is the Fisher direction proportional to the raw difference $\mu_1-\mu_2$?
Suppose a rule is $w^\mathsf{T}x>t$. How should the threshold change if both $w$ and $t$ are multiplied by the same positive constant?