Canonical Roots As Measures Of Class Separation

From Discriminant Directions to Measures of Separation

A discriminant direction tells us how to project multivariate observations onto a line. But a direction alone does not say how strongly the class means separate relative to the variation within classes. Canonical roots supply that comparison: each is a ratio of between-class to within-class variation along a particular direction.

The largest root identifies the direction with the greatest such ratio. Further roots describe additional directions, subject to being uncorrelated with earlier directions under the within-class scatter geometry. Together, the roots and their associated canonical variates give a compact summary of class-mean separation.

Scatter Matrices and Canonical Roots

Suppose there are \(K\) classes and \(p\)-dimensional observations \(x_{ij}\), where \(i=1,\ldots,K\) indexes class and \(j=1,\ldots,n_i\) indexes observations within class. Write \(n=\sum_{i=1}^K n_i\), let \(\bar{x}_i\) be the mean of class \(i\), and let \(\bar{x}\) be the overall mean. The within-class scatter matrix \(W\) and between-class scatter matrix \(B\) are

Both matrices are positive semidefinite. For a direction \(a\ne 0\), the projected score is \(a'x\). Its within-class sum of squared deviations is \(a'Wa\), and the between-class sum of squared deviations of the projected class means is \(a'Ba\). Thus the directional separation ratio is

The ratio is unchanged if \(a\) is multiplied by a nonzero constant, so it measures a direction rather than the arbitrary scale of its coefficient vector.

If \(a_r\) is a generalized eigenvector, multiplying \(Ba_r=\lambda_r Wa_r\) on the left by \(a_r'\) gives \(a_r'Ba_r=\lambda_r a_r'Wa_r\). Therefore \(R(a_r)=\lambda_r\). The root is a ratio of sums of squared deviations, not a probability or a percentage of correctly classified observations.

Why the Roots Are Successive Maxima

The generalized eigenproblem does more than find stationary directions of the ratio. Its ordered roots solve a sequence of constrained optimization problems. The constraint uses \(W\), not ordinary Euclidean length: two directions are within-class orthogonal when \(a_r'Wa_s=0\).

Proof. By the spectral theorem, the positive definite matrix \(W\) has a symmetric positive definite square root \(W^{1/2}\). Set \(b=W^{1/2}a\), so \(a=W^{-1/2}b\), and define \(C=W^{-1/2}BW^{-1/2}\). The matrix \(C\) is symmetric and positive semidefinite. Substitution gives

The spectral theorem gives orthonormal eigenvectors \(u_1,\ldots,u_p\) of \(C\), with ordered eigenvalues \(\lambda_1\ge\cdots\ge\lambda_p\ge0\). Write \(b=\sum_{j=1}^p c_j u_j\). Then

Equality is attained at \(b=u_1\). More generally, imposing orthogonality to \(u_1,\ldots,u_{r-1}\) sets \(c_1,\ldots,c_{r-1}\) to zero, so the same weighted-average argument bounds the ratio by \(\lambda_r\), with equality at \(b=u_r\). Returning to the original coordinates, \(a_r=W^{-1/2}u_r\). These vectors satisfy \(a_r'Wa_s=u_r'u_s\), and the orthogonality constraints become exactly \(a_s'Wa=0\). Hence the roots have the stated successive-maximization property. \(\square\)

This argument also explains why the generalized eigenvectors are normalized using \(W\). After transforming by \(W^{1/2}\), the problem becomes an ordinary symmetric eigenproblem with Euclidean-orthogonal eigenvectors. If a root is repeated, its individual eigenvectors are not unique: any suitable orthonormal basis for the corresponding eigenspace gives the same root and subspace.

How Many Roots Carry Class Separation?

The between-class scatter depends on the \(K\) class means measured relative to their weighted overall mean. Those centered mean vectors have a weighted sum of zero, so they span at most \(K-1\) independent directions. Consequently, \(\operatorname{rank}(B)\le K-1\), and there can be at most \(\min(p,K-1)\) positive canonical roots. This is the rank reason that a small number of canonical variates can summarize the separation of many-dimensional class means.

A zero root means that, after accounting for the earlier canonical directions, no additional direction has positive between-class variation relative to within-class variation. It does not mean that all observations in the classes coincide. The root concerns class means and their variation, not every feature of the class distributions.

Interpreting Roots Without Overclaiming

The roots are useful when the goal is to describe how class means differ relative to within-class scatter. A large root indicates a direction along which class means are widely spread compared with the scatter of observations around their own class means. A small positive root indicates weaker separation by that criterion. The associated canonical variates provide coordinates for describing those directions.

Roots can also be reported relative to their sum as a descriptive account of how the total generalized eigenvalue mass is distributed across the canonical directions. This normalization is a summary convention, not a probability statement. In particular, it does not establish what fraction of observations will be classified correctly or how much of the overall data variance a direction explains.

The roots depend on the scatter matrices used. The formulas above use unnormalized sums of squares for both \(B\) and \(W\). Multiplying both matrices by the same positive constant leaves every root unchanged, but changing their relative scaling changes the ratios. Comparisons therefore require consistent definitions and weighting.

When \(W\) is singular, the stated generalized-eigenvalue interpretation requires additional care because some nonzero directions can have \(a'Wa=0\). The positive-definite assumption avoids undefined ratios and permits the whitening argument above. In applications with singular within-class scatter, one must specify a restriction or regularization before interpreting roots; silently applying the positive-definite formulas is not justified.