Eigenstructure of Gaussian Covariance Matrices

Why Covariance Eigenstructure Matters

A covariance matrix records the variances and covariances of a random vector, but its entries do not always make the vector’s main directions of variation obvious. Eigenvectors reveal those directions, while their corresponding eigenvalues quantify the variance along them. This turns a matrix description into a geometric one: it identifies how a Gaussian distribution is oriented, how spread out it is in each direction, and whether it is confined to a lower-dimensional subspace.

Here $X$ is a $p$-dimensional Gaussian random vector with mean vector $\mu$ and covariance matrix $\Sigma$. Since $\Sigma$ is real, symmetric, and positive semidefinite, the spectral theorem gives an orthonormal basis of eigenvectors. The consequences of that fact are especially useful when eigenvalues are repeated or zero.

Spectral Decomposition and Principal Directions

Definition: An eigenvector $q\ne 0$ of a matrix $\Sigma$ is a vector satisfying $\Sigma q=\lambda q$ for some scalar $\lambda$, its eigenvalue. For a covariance matrix $\Sigma$, the eigenvalues are nonnegative. The spectral theorem gives an orthonormal eigenbasis $q_1,\ldots,q_p$, with corresponding eigenvalues $\lambda_1,\ldots,\lambda_p$, such that $$ \Sigma=Q\Lambda Q^{mathsf T},\qquad Q=(q_1,\ldots,q_p),\qquad \Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_p), \qquad \lambda_i\ge 0. $$ A unit eigenvector is a principal direction of the covariance matrix, and its eigenvalue is the variance of the projection in that direction.

To see why the last statement holds, take any unit vector $q_i$. The scalar projection $q_i^{mathsf T}X$ has variance $$ \operatorname{Var}(q_i^{mathsf T}X)=q_i^{mathsf T}\Sigma q_i =q_i^{mathsf T}(\lambda_i q_i)=\lambda_i. $$ Thus the eigenvalue is not merely an algebraic label: it is the variance measured along that direction. The direction of the largest eigenvalue has the greatest variance among all unit directions, and the direction of the smallest eigenvalue has the least.

Worked Example: Finding the Principal Directions in Two Dimensions

Let $X\sim N(\mu,\Sigma)$, where $$ \mu=\begin{pmatrix}1\\-1\end{pmatrix}, \qquad \Sigma=\begin{pmatrix}4&2\\2&4\end{pmatrix}. $$ Find the eigenvalues and principal directions, and interpret their variances.

The characteristic equation is $$ \det(\Sigma-\lambda I) =(4-\lambda)^2-4 =\lambda^2-8\lambda+12 =(\lambda-6)(\lambda-2)=0. $$ So the eigenvalues are $6$ and $2$. For $\lambda=6$, the equation $(\Sigma-6I)q=0$ gives $q_1=q_2$, so a unit eigenvector is $q_1=(1,1)^{mathsf T}/\sqrt{2}$. For $\lambda=2$, it gives $q_1=-q_2$, so a unit eigenvector is $q_2=(1,-1)^{mathsf T}/\sqrt{2}$.

The projection of $X$ along $(1,1)^{mathsf T}/\sqrt{2}$ has variance $6$; the projection along $(1,-1)^{mathsf T}/\sqrt{2}$ has variance $2$. These directions are orthogonal, and the first is the direction of greater spread. For verification, the spectral decomposition reconstructs the original matrix: $$ 6q_1q_1^{mathsf T}+2q_2q_2^{mathsf T} = \begin{pmatrix}3&3\\3&3\end{pmatrix} + \begin{pmatrix}1&-1\\-1&1\end{pmatrix} = \begin{pmatrix}4&2\\2&4\end{pmatrix}. $$

Directional Variance and the Principal Axes

The eigenbasis does more than describe variance in the eigenvector directions. It gives the variance in every direction. Write a unit vector $a$ in the orthonormal eigenbasis as $a=\sum_{i=1}^{p}c_iq_i$, where $\sum_i c_i^2=1$. Then $$ \operatorname{Var}(a^{mathsf T}X) =a^{mathsf T}\Sigma a =\sum_{i=1}^{p}\lambda_i c_i^2. $$ The coefficients $c_i^2$ are nonnegative and sum to one. Therefore the directional variance is a weighted average of the eigenvalues: it cannot exceed the largest eigenvalue or fall below the smallest. Equality at the largest eigenvalue occurs exactly when $a$ lies in the eigenspace for that eigenvalue; equality at the smallest occurs exactly when $a$ lies in its eigenspace.

Theorem: Let $\Sigma$ be a real symmetric positive semidefinite $p\times p$ matrix, with eigenvalues $\lambda_1\ge\cdots\ge\lambda_p$ and an orthonormal eigenbasis $q_1,\ldots,q_p$. For every unit vector $a$, $$ \lambda_p\le a^{mathsf T}\Sigma a\le\lambda_1. $$ The maximum is $\lambda_1$ and the minimum is $\lambda_p$. The vectors attaining either bound are precisely the unit vectors in the corresponding eigenspace.

Proof. Expand $a=\sum_i c_iq_i$, so that $\sum_i c_i^2=1$. Since $\Sigma q_i=\lambda_iq_i$ and the eigenvectors are orthonormal, $$ a^{mathsf T}\Sigma a=\sum_{i=1}^{p}\lambda_i c_i^2. $$ The ordering of the eigenvalues gives $\lambda_p c_i^2\le\lambda_i c_i^2\le\lambda_1c_i^2$ for each $i$. Summing and using $\sum_i c_i^2=1$ proves the two bounds. Equality in the upper bound requires $c_i=0$ whenever $\lambda_i<\lambda_1$; this is exactly membership in the eigenspace for $\lambda_1$. The lower-bound equality follows by the same reasoning. $\square$

For a nonsingular Gaussian vector, the density contours reflect these variances. In eigen-coordinates $y_i=q_i^{mathsf T}(x-\mu)$, the quadratic form in the Gaussian density is $$ (x-\mu)^{mathsf T}\Sigma^{-1}(x-\mu) =\sum_{i=1}^{p}\frac{y_i^2}{\lambda_i}. $$ Thus a fixed contour has larger extent along directions with larger eigenvalues. This gives the axes and relative lengths of the ellipsoidal contours; it does not mean that each coordinate direction in the original coordinate system is a principal direction. For singular covariance matrices, this inverse-based formula does not apply, as discussed below. The geometry of nonsingular Gaussian contours was treated in the earlier material on elliptical contours of Gaussian densities.

Worked Example: Repeated Eigenvalues and Non-Unique Axes

Suppose a three-dimensional Gaussian vector has mean $\mu=(1,0,-1)^{\mathsf T}$ and covariance matrix $$ \Sigma=\begin{pmatrix}5&0&0\\0&2&0\\0&0&2\end{pmatrix}. $$ Describe its principal directions and explain what the repeated eigenvalue means.

The eigenvalue $5$ has eigenvector $e_1=(1,0,0)^{\mathsf T}$. The eigenvalue $2$ has multiplicity two, with eigenspace consisting of all vectors $(0,s,t)^{\mathsf T}$. For example, $e_2=(0,1,0)^{\mathsf T}$ and $e_3=(0,0,1)^{\mathsf T}$ are an orthonormal basis of that eigenspace.

The variance along $e_1$ is $5$, and the variance along either $e_2$ or $e_3$ is $2$. More generally, any unit vector $a=(0,s,t)^{\mathsf T}$ with $s^2+t^2=1$ has variance $$ a^{mathsf T}\Sigma a=2s^2+2t^2=2. $$ So the two-dimensional eigenspace is the lower-variance plane, but there is no unique pair of principal directions within that plane. Any orthonormal basis of it is equally valid. The eigenvalue multiplicity identifies a subspace of directions, not a uniquely determined set of axes inside that subspace.

Gaussian Coordinates in an Eigenbasis

Let $Q$ be an orthogonal matrix whose columns are eigenvectors of $\Sigma$, and define the centered eigen-coordinates $Y=Q^{mathsf T}(X-\mu)$. These coordinates have diagonal covariance $\Lambda$. For a Gaussian vector, the coordinates are in fact independent, including when some have variance zero.

Theorem: If $X\sim N(\mu,\Sigma)$ and $\Sigma=Q\Lambda Q^{mathsf T}$, then $Y=Q^{mathsf T}(X-\mu)$ has independent components $Y_i\sim N(0,\lambda_i)$. When $\lambda_i=0$, $Y_i=0$ with probability one.

Proof. The characteristic function of $X$ is $$ \varphi_X(t)=\exp\left(it^{mathsf T}\mu-\tfrac12t^{mathsf T}\Sigma t\right), \qquad t\in\mathbb R^p. $$ For $s\in\mathbb R^p$, the characteristic function of $Y$ is the characteristic function of $X-\mu$ evaluated at $Qs$. Orthogonality gives $Q^{mathsf T}Q=I$, so $$ \varphi_Y(s) =\exp\left(-\tfrac12s^{mathsf T}Q^{mathsf T}\Sigma Qs\right) =\exp\left(-\tfrac12s^{mathsf T}\Lambda s\right) =\prod_{i=1}^{p}\exp\left(-\tfrac12\lambda_i s_i^2\right). $$ Each factor is the characteristic function of a univariate $N(0,\lambda_i)$ variable when $\lambda_i>0$, and of the constant zero when $\lambda_i=0$. The product form and uniqueness of characteristic functions show that the components are independent with these marginal laws. $\square$

In particular, if an eigenvalue is repeated, the choice of orthonormal basis inside its eigenspace can change the individual coordinates but not their common variance. The covariance structure assigns the same variance to every direction in that eigenspace.

Worked Example: A Zero Eigenvalue and a Singular Gaussian Law

Let $X$ be a three-dimensional Gaussian vector with $$ \mu=\begin{pmatrix}2\\0\\3\end{pmatrix}, \qquad \Sigma=\begin{pmatrix}5/2&3/2&0\\3/2&5/2&0\\0&0&0\end{pmatrix}. $$ Find an eigenbasis, the corresponding variances, and the support of the law.

Two orthonormal eigenvectors in the first two coordinates are $$ q_1=\frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\\0\end{pmatrix}, \qquad q_2=\frac{1}{\sqrt{2}}\begin{pmatrix}1\\-1\\0\end{pmatrix}. $$ Direct multiplication gives $\Sigma q_1=4q_1$ and $\Sigma q_2=q_2$. The third eigenvector is $q_3=(0,0,1)^{\mathsf T}$, and $\Sigma q_3=0$. Thus the eigenvalues are $4,1,0$.

The coordinates $Y_i=q_i^{mathsf T}(X-\mu)$ are independent Gaussian variables with variances $4,1,0$, respectively. In particular, $Y_3=0$ with probability one. Since $q_3^{mathsf T}(X-\mu)=X_3-3$, it follows that $X_3=3$ with probability one. The law is concentrated on the plane $x_3=3$, the affine space $\mu+\operatorname{span}(q_1,q_2)$.

There is no ordinary three-dimensional density for this distribution: all its probability lies on a plane, which has three-dimensional volume zero. The zero eigenvalue expresses this exact constraint, rather than a very small but positive amount of variation. As a check, the spectral reconstruction is $$ 4q_1q_1^{mathsf T}+q_2q_2^{mathsf T} = \begin{pmatrix}2&2&0\\2&2&0\\0&0&0\end{pmatrix} + \begin{pmatrix}1/2&-1/2&0\\-1/2&1/2&0\\0&0&0\end{pmatrix} = \begin{pmatrix}5/2&3/2&0\\3/2&5/2&0\\0&0&0\end{pmatrix}. $$

What Repeated and Zero Eigenvalues Tell Us

Eigenvalue multiplicity and zero eigenvalues have distinct interpretations. If an eigenvalue is repeated, its eigenspace is a subspace on which every unit direction has the same variance. The covariance matrix determines that subspace, but not a preferred orthonormal basis within it. If an eigenvalue is zero, its eigenvector direction has no random variation at all.

More precisely, for any unit eigenvector $q$ with eigenvalue zero, $$ E\left[\{q^{mathsf T}(X-\mu)\}^2\right] =\operatorname{Var}(q^{mathsf T}X) =q^{mathsf T}\Sigma q=0. $$ A nonnegative random variable with expectation zero is zero with probability one, so $q^{mathsf T}(X-\mu)=0$ almost surely. Applying this to an orthonormal basis of the null space shows that the Gaussian vector lies almost surely in $\mu+\operatorname{Range}(\Sigma)$. For a Gaussian law, this affine subspace is its support.

A common mistake is to use the nonsingular density formula when $\Sigma$ has a zero eigenvalue, perhaps by treating zero as merely a small variance. The formula requires $\Sigma^{-1}$ and a positive determinant, neither of which exists in the singular case. The correction is to identify the null space, state the deterministic linear constraints it imposes, and describe the law on its lower-dimensional affine support. Conversely, a repeated positive eigenvalue does not make a covariance matrix singular: singularity occurs precisely when at least one eigenvalue is zero.

Key takeaway: Eigenvectors identify the directions in which a Gaussian vector varies, and eigenvalues give the variances in those directions. Repeated eigenvalues make the choice of axes within an eigenspace non-unique; zero eigenvalues impose exact linear constraints and place the distribution on a lower-dimensional affine subspace.

Check Your Understanding

Use the eigenstructure to answer each question.

For a covariance matrix with eigenvalues $7,3,3$, what is the largest possible variance of a unit projection? What does the repeated eigenvalue tell you?
If $q$ is a unit eigenvector with eigenvalue $0$, what can you conclude about $q^{mathsf T}(X-\mu)$?
Why can two different orthonormal eigenbases describe the same covariance matrix when an eigenvalue is repeated?
A Gaussian covariance matrix has eigenvalues $4,1,0$. Is there an ordinary density with respect to three-dimensional volume? Explain.
For a unit vector $a=\sum_i c_iq_i$, express its directional variance in terms of the eigenvalues and the coefficients $c_i$.