Eigenvalue Decomposition of Cluster Covariance Matrices

From a Covariance Matrix to an Ellipsoid

A Gaussian cluster in $d$ dimensions is described in part by its covariance matrix $\Sigma_g$, where $g$ indexes the component. The entries of this matrix may be difficult to interpret directly, but its eigenstructure gives a geometric description: eigenvectors determine the principal directions of the cluster, while eigenvalues determine how spread out it is in those directions.

This description can be separated into three parts. A single positive scalar records overall scale, a determinant-normalized matrix records relative axis lengths, and an orthogonal matrix records the directions of the axes. This decomposition is useful because two covariance matrices can have the same overall size but quite different shapes or orientations.

Definition: Let $\Sigma_g$ be a real, symmetric, positive-definite $d\times d$ covariance matrix. Write its eigenvalues as $\lambda_{g1},\ldots,\lambda_{gd}$, all positive, and choose corresponding orthonormal eigenvectors as the columns of $D_g$. Then $$ \Sigma_g=D_g\operatorname{diag}(\lambda_{g1},\ldots,\lambda_{gd})D_g^{T}. $$ Define the volume scale $\lambda_g=|\Sigma_g|^{1/d}$, the shape matrix $A_g=\operatorname{diag}(\lambda_{g1}/\lambda_g,\ldots,\lambda_{gd}/\lambda_g)$, and the orientation matrix $D_g$. Thus $\Sigma_g=\lambda_gD_gA_gD_g^T$, with $|A_g|=1$ and $D_g^TD_g=I$.

The symbol $\lambda_g$ here denotes a scalar volume parameter; $\lambda_{gj}$ denotes the $j$th eigenvalue. Since $|\Sigma_g|=\prod_{j=1}^d\lambda_{gj}$, the scalar $\lambda_g$ is the geometric mean of the eigenvalues. It is not, in general, an eigenvalue itself.

Why the Three Parts Have These Names

For a Gaussian component with mean $\mu_g$, a contour at squared Mahalanobis distance $c>0$ is the set of points satisfying $$ (x-\mu_g)^T\Sigma_g^{-1}(x-\mu_g)=c. $$ In the coordinate system of the eigenvectors, this is an ellipsoid. Its principal directions are the columns of $D_g$, and its semiaxis lengths are $\sqrt{c\lambda_{g1}},\ldots,\sqrt{c\lambda_{gd}}$. Thus the eigenvectors give orientation, while the eigenvalues give squared axis lengths, up to the common contour multiplier $c$.

The factor $\lambda_g$ scales all eigenvalues by the same amount when shape and orientation are held fixed. The entries of $A_g$ are eigenvalues relative to their geometric mean, so they describe axis proportions rather than overall scale. Their product is one: the normalized shape does not add or remove determinant. If all eigenvalues are equal, $A_g=I$, and there is no elongation; the contour is a sphere.

Theorem: For $c>0$, the ellipsoid $\{x:(x-\mu_g)^T\Sigma_g^{-1}(x-\mu_g)\leq c\}$ has principal directions given by the eigenvectors of $\Sigma_g$, semiaxis lengths $\sqrt{c\lambda_{gj}}$, and volume $\kappa_d c^{d/2}\sqrt{|\Sigma_g|}$, where $\kappa_d$ is the volume of the unit ball in $d$ dimensions.

Proof. By the spectral decomposition, $\Sigma_g=D_g\Lambda_gD_g^T$, where $\Lambda_g=\operatorname{diag}(\lambda_{g1},\ldots,\lambda_{gd})$. Since $D_g$ is orthogonal, $\Sigma_g^{-1}=D_g\Lambda_g^{-1}D_g^T$. Set $z=D_g^T(x-\mu_g)$. The contour inequality becomes $$ \sum_{j=1}^d\frac{z_j^2}{\lambda_{gj}}\leq c, $$ or, equivalently, $\sum_{j=1}^d z_j^2/(c\lambda_{gj})\leq 1$. This is an ellipsoid in the $z$-coordinates with axes along the coordinate directions and semiaxis lengths $\sqrt{c\lambda_{gj}}$. Transforming back by $D_g$ rotates these axes to the eigenvector directions without changing lengths or volume, because an orthogonal matrix has determinant of absolute value one. The ellipsoid is the image of the unit ball under the linear transformation $D_g\operatorname{diag}(\sqrt{c\lambda_{g1}},\ldots,\sqrt{c\lambda_{gd}})$. The volume multiplier is the absolute determinant of this transformation: $$ \left|D_g\right|\prod_{j=1}^d\sqrt{c\lambda_{gj}} =c^{d/2}\sqrt{\prod_{j=1}^d\lambda_{gj}} =c^{d/2}\sqrt{|\Sigma_g|}. $$ Multiplying by the unit-ball volume $\kappa_d$ proves the result. $\square$

Since $|\Sigma_g|=\lambda_g^d$, the contour volume can also be written $\kappa_dc^{d/2}\lambda_g^{d/2}$. Consequently, $\lambda_g$ is a convenient determinant-based measure of covariance scale, while the actual ellipsoid volume is proportional to its $d/2$ power.

Worked Decompositions

Worked Example: An Axis-Aligned Bivariate Cluster

Consider $\Sigma_g=\begin{pmatrix}4&0\\0&1\end{pmatrix}$. Its eigenvalues are $4$ and $1$, with eigenvectors along the two coordinate axes. Therefore $D_g=I$, and the volume scale is $$ \lambda_g=|\Sigma_g|^{1/2}=(4\cdot1)^{1/2}=2. $$ The normalized shape matrix is $$ A_g=\operatorname{diag}(4/2,1/2)=\operatorname{diag}(2,1/2). $$ The reconstruction is $\lambda_gD_gA_gD_g^T=2\operatorname{diag}(2,1/2)=\operatorname{diag}(4,1)=\Sigma_g$, and $|A_g|=2(1/2)=1$.

At $c=1$, the ellipse has semiaxis lengths $\sqrt{4}=2$ and $\sqrt{1}=1$. Its area is $\pi(2)(1)=2\pi$, which also follows from $\pi\sqrt{|\Sigma_g|}=\pi\sqrt{4}=2\pi$. The longer axis is the first coordinate direction.

Worked Example: Recovering a Rotated Cluster

Suppose a bivariate covariance has eigenvalues $9$ and $1$, with the eigenvector for $9$ pointing at $30^\circ$ to the first coordinate axis. Choose $$ D_g=\begin{pmatrix}\sqrt{3}/2&-1/2\\1/2&\sqrt{3}/2\end{pmatrix}, \qquad \Lambda_g=\operatorname{diag}(9,1). $$ The volume scale is $\lambda_g=(9\cdot1)^{1/2}=3$, and the shape matrix is $A_g=\operatorname{diag}(3,1/3)$. Their determinant check is $|A_g|=1$.

Multiplying $D_g\Lambda_gD_g^T$ gives $$ \Sigma_g= \begin{pmatrix} 9(3/4)+1(1/4)&(9-1)(\sqrt{3}/2)(1/2)\\ (9-1)(\sqrt{3}/2)(1/2)&9(1/4)+1(3/4) \end{pmatrix} = \begin{pmatrix}7&2\sqrt{3}\\2\sqrt{3}&3\end{pmatrix}. $$ Its determinant is $7\cdot3-(2\sqrt{3})^2=21-12=9$, consistent with $\lambda_g^2=9$. The $c=1$ ellipse has semiaxes $3$ and $1$; its longer direction is rotated $30^\circ$. The off-diagonal covariance records that the principal axes are not aligned with the original coordinate axes.

Worked Example: A Three-Dimensional Shape Matrix

Let the eigenvalues of a three-dimensional covariance be $8,2,$ and $0.5$, with orthonormal eigenvectors collected in $D_g$. Their product is $8$, so the volume scale is $$ \lambda_g=8^{1/3}=2. $$ Dividing each eigenvalue by 2 gives $A_g=\operatorname{diag}(4,1,0.25)$. Its determinant is $4\cdot1\cdot0.25=1$, and $$ \Sigma_g=2D_g\operatorname{diag}(4,1,0.25)D_g^T =D_g\operatorname{diag}(8,2,0.5)D_g^T. $$

At $c=1$, the semiaxis lengths are $\sqrt{8}=2\sqrt{2}$, $\sqrt{2}$, and $\sqrt{0.5}=1/\sqrt{2}$, along the three columns of $D_g$, respectively. The ellipsoid volume is $\kappa_3\sqrt{8}$. Its orientation depends on $D_g$; its axis proportions are represented by $4,1,0.25$; and its overall covariance scale is represented by $2$.

What Is and Is Not Identified by the Decomposition

The factorization separates geometric roles, but its orientation matrix should be interpreted with care. An eigenvector and its negative point along the same axis, so changing the sign of a column of $D_g$ does not change $\Sigma_g$. Eigenvectors can also be reordered, provided the corresponding eigenvalues in $A_g$ are reordered in the same way.

If an eigenvalue is repeated, its individual eigenvectors are not uniquely determined: any orthonormal basis of the corresponding eigenspace is valid. For example, when $\Sigma_g$ is a scalar multiple of $I$, every orthonormal coordinate system is an eigenvector basis. The covariance has no preferred orientation in that case. More generally, a repeated eigenvalue means that the covariance determines the associated subspace, but not unique axis directions within it.

The determinant separates volume from relative shape because multiplying every eigenvalue by the same positive constant changes $\lambda_g$ but leaves $A_g$ unchanged. Conversely, covariance matrices can have the same determinant, and therefore the same ellipsoid volume at a fixed $c$, while having different eigenvalue ratios and different orientations. Equal volume does not imply equal geometry.

Common Mistake

A frequent error is to call $|\Sigma_g|$ the ellipsoid volume itself. The determinant is a scale-related quantity, but the ellipsoid volume at squared Mahalanobis radius $c$ is $\kappa_dc^{d/2}\sqrt{|\Sigma_g|}$. In two dimensions, for example, $|\Sigma_g|=4$ gives area $2\pi$ at $c=1$, not area $4$. The square root appears because each covariance eigenvalue is a squared axis length.

Another error is to treat the eigenvectors as uniquely defined in every case. Their signs are arbitrary, their order can be changed along with the eigenvalue order, and repeated eigenvalues allow rotations within the associated eigenspace. The geometric orientation means the principal directions or eigenspaces, not one uniquely specified signed list of vectors.

Key takeaway: For a positive-definite component covariance, the eigenvalues determine axis lengths and their geometric mean gives the determinant-based volume scale. Normalizing the eigenvalues by that mean produces a determinant-one shape matrix, while the eigenvectors give the principal directions. Ellipsoid volume depends on the square root of the covariance determinant.

Check Your Understanding

Use the decomposition and its geometric interpretation to answer the following.

For eigenvalues $16$ and $1$, what are the volume scale and the two entries of the normalized shape matrix?
Why does an orthogonal change of coordinates alter orientation but not ellipsoid volume?
What are the semiaxis lengths of a contour with $c=4$ when the covariance eigenvalues are $9$ and $4$?
Two covariance matrices have the same determinant. What geometric property of their contours at the same $c$ must agree, and what properties may differ?
Why is there no uniquely determined principal orientation for a covariance matrix proportional to the identity?