Spectral Geometry of Gaussian Covariance

Why the Covariance Spectrum Describes Geometry

A covariance matrix records how a random vector varies in each coordinate and how its coordinates vary together. Its eigenvectors and eigenvalues give a more geometric description: the eigenvectors identify directions of variation, while the eigenvalues quantify the variance along those directions. For a Gaussian vector, this spectral information determines the orientation and relative widths of its density contours.

This picture changes when some eigenvalues are zero. A zero eigenvalue means there is no variation in its eigenvector direction. The Gaussian distribution is then confined to a lower-dimensional affine subspace, rather than having a density spread throughout the ambient space. The distinction matters both for interpreting a covariance matrix and for deciding which density formula is valid.

Definition: Let $X$ be a $p$-dimensional Gaussian vector with mean $\mu$ and covariance matrix $\Sigma$. A spectral decomposition of $\Sigma$ is $$ \Sigma=Q\Lambda Q^T, $$ where $Q$ is orthogonal, its columns $u_1,\ldots,u_p$ are unit eigenvectors of $\Sigma$, and $\Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_p)$ contains the corresponding eigenvalues. Since a covariance matrix is symmetric and positive semidefinite, the eigenvectors can be chosen orthonormal and every $\lambda_j\geq 0$. The directions $u_j$ are the principal directions of the covariance; the variance in direction $u_j$ is $\lambda_j$.

The variance statement follows directly from the definition of covariance. For any unit vector $u$, the scalar projection $u^T(X-\mu)$ has variance $u^T\Sigma u$. If $u=u_j$, then $$ \operatorname{Var}\bigl(u_j^T(X-\mu)\bigr)=u_j^T\Sigma u_j =u_j^T(\lambda_j u_j)=\lambda_j. $$ Thus a large eigenvalue corresponds to substantial spread along its eigenvector, and a small eigenvalue corresponds to comparatively little spread.

Density Contours and Principal Axes

First suppose $\Sigma$ is positive definite, so every eigenvalue is positive. The Gaussian vector has a density with respect to $p$-dimensional Lebesgue measure. Its density at $x\in\mathbb{R}^p$ is $$ f(x)=\frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}} \exp\left\{-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)\right\}. $$ The exponent depends on $x$ through the quadratic form $$ d_\Sigma^2(x,\mu)=(x-\mu)^T\Sigma^{-1}(x-\mu). $$ Consequently, the sets on which the density is constant are the level sets $d_\Sigma^2(x,\mu)=c$, for $c\geq 0$. They are ellipsoids centered at $\mu$.

The eigenbasis makes their shape explicit. Write $x-\mu=\sum_{j=1}^p a_j u_j$, where $a_j=u_j^T(x-\mu)$ is the coordinate in direction $u_j$. Since $\Sigma^{-1}u_j=\lambda_j^{-1}u_j$, $$ d_\Sigma^2(x,\mu)=\sum_{j=1}^p\frac{a_j^2}{\lambda_j}. $$ Therefore the contour at level $c>0$ satisfies $\sum_j a_j^2/\lambda_j=c$. Its semi-axis in direction $u_j$ has length $\sqrt{c\lambda_j}$. The eigenvectors set the orientation, and the square roots of the eigenvalues set the contour widths, up to the common factor $\sqrt c$.

A contour level describes density, not by itself a probability. A particular contour encloses a particular probability only after the distribution of the corresponding quadratic form and the chosen level have been taken into account. Also, when an eigenvalue is repeated, its eigenvectors within that eigenspace are not uniquely determined. The geometric eigenspace is well defined, but no unique set of individual axes is selected inside it.

Worked Example: Axis-Aligned Gaussian Contours

Let $X$ be a bivariate Gaussian vector with mean $\mu=(1,-2)^T$ and covariance $$ \Sigma= \begin{pmatrix} 4&0\\ 0&1 \end{pmatrix}. $$ Find its principal directions and the semi-axis lengths of the contour $d_\Sigma^2(x,\mu)=4$.

The covariance matrix is already diagonal. Its eigenvalues are $4$ and $1$, with eigenvectors $u_1=(1,0)^T$ and $u_2=(0,1)^T$. The contour equation, writing $x-\mu=(a_1,a_2)^T$, is $$ \frac{a_1^2}{4}+\frac{a_2^2}{1}=4. $$ In the first coordinate direction the semi-axis length is $\sqrt{4\cdot4}=4$. In the second it is $\sqrt{4\cdot1}=2$. Thus this contour is centered at $(1,-2)^T$, extends 4 units in either direction along the first coordinate, and extends 2 units along the second.

The longer axis corresponds to the larger variance, $4$. The axes are coordinate-aligned here because the covariance has no off-diagonal entries.

Worked Example: Correlation Rotates the Contours

Consider a bivariate Gaussian vector with mean $\mu=(0,0)^T$ and covariance $$ \Sigma= \begin{pmatrix} 4&3\\ 3&4 \end{pmatrix}. $$ Find its eigenvalues and principal directions, then give the semi-axis lengths of the contour at level $c=4$.

The vectors $u_1=(1,1)^T/\sqrt{2}$ and $u_2=(1,-1)^T/\sqrt{2}$ are orthonormal. Direct multiplication gives $$ \Sigma u_1=7u_1,\qquad \Sigma u_2=u_2. $$ The eigenvalues are therefore $7$ and $1$. At level $c=4$, the semi-axis lengths are $\sqrt{4\cdot7}=2\sqrt{7}$ along $u_1$, and $\sqrt{4\cdot1}=2$ along $u_2$.

The contour is elongated along the line $x_2=x_1$, not along either original coordinate axis. The off-diagonal covariance rotates the principal directions away from the coordinate directions. The eigenvalues quantify the spread along the rotated axes.

Rank Deficiency and the Support of the Distribution

A covariance matrix may be positive semidefinite without being positive definite. If its rank is $r<p$, then exactly $r$ eigenvalues are positive and the remaining $p-r$ are zero. A zero eigenvalue has a stronger implication than merely indicating a narrow contour: the corresponding projection has variance zero and is therefore constant with probability one. The Gaussian vector cannot vary in that direction.

Definition: The support subspace of a Gaussian vector $X$ with covariance $\Sigma$ is $\operatorname{Range}(\Sigma)$, the span of the eigenvectors with positive eigenvalues. Its affine support is $\mu+\operatorname{Range}(\Sigma)$. If $\Sigma$ has rank $r<p$, the vector lies in this $r$-dimensional affine subspace with probability one.

For a singular covariance matrix, the inverse and determinant in the ordinary $p$-dimensional Gaussian density formula are not available. In particular, the distribution has no density with respect to $p$-dimensional Lebesgue measure: all its probability lies in a lower-dimensional subspace, which has zero $p$-dimensional volume. One can instead describe its density using $r$-dimensional volume within its affine support.

To state that intrinsic density precisely, order the positive eigenvalues first and let $U_r$ be the $p$-by-$r$ matrix whose columns are their orthonormal eigenvectors. Write $D_r=\operatorname{diag}(\lambda_1,\ldots,\lambda_r)$. For $x$ on the affine support, define its coordinates $y=U_r^T(x-\mu)$. The density with respect to ordinary $r$-dimensional Lebesgue measure in these orthonormal coordinates is $$ f_{\mathrm{supp}}(y)=\frac{1}{(2\pi)^{r/2}|D_r|^{1/2}} \exp\left\{-\frac{1}{2}y^TD_r^{-1}y\right\}. $$ Its contours within the support are ellipsoids with semi-axis lengths $\sqrt{c\lambda_j}$ at level $c$, for $j=1,\ldots,r$. There are no axes of positive length in the zero-eigenvalue directions; those directions are excluded from the support.

Theorem: Let $X$ be a Gaussian vector with mean $\mu$ and covariance $\Sigma$, and suppose $\Sigma=Q\Lambda Q^T$ has rank $r$. In an orthonormal eigenbasis, $X$ has $r$ varying Gaussian coordinates with variances equal to the positive eigenvalues, and its remaining $p-r$ coordinates are zero with probability one after centering at $\mu$. Hence its affine support is $\mu+\operatorname{Range}(\Sigma)$.

Proof. Use the affine-image representation of a Gaussian vector: $X$ can be represented in distribution as $\mu+AZ$, where $Z$ is a standard normal vector and $AA^T=\Sigma$. Take a singular value decomposition of $A$. The positive singular values are $\sqrt{\lambda_1},\ldots,\sqrt{\lambda_r}$, and the corresponding left singular vectors are the eigenvectors $u_1,\ldots,u_r$ of $\Sigma$. Orthogonal transformations preserve the standard normal distribution, so in these eigenvector coordinates the centered vector has the same distribution as $$ Q^T(X-\mu)= \bigl(\sqrt{\lambda_1}G_1,\ldots,\sqrt{\lambda_r}G_r,0,\ldots,0\bigr)^T, $$ where $G_1,\ldots,G_r$ are independent standard normal variables. Thus the last $p-r$ coordinates vanish with probability one, and the first $r$ coordinates have the stated variances. The set of possible values is contained in the span of $u_1,\ldots,u_r$; since each positive-variance normal coordinate has support all of $\mathbb{R}$, the support is exactly that span. Translating by $\mu$ gives the affine support. $\square$

Worked Example: A Gaussian Confined to a Line

Let $Z\sim N(0,1)$ and define $X=\mu+(Z,2Z)^T$, where $\mu=(1,-1)^T$. Find the covariance, its nonzero eigenvalue and eigenvector, and the support.

The centered vector is $X-\mu=(Z,2Z)^T$. Its covariance is $$ \Sigma=E\bigl[(X-\mu)(X-\mu)^T\bigr] = \begin{pmatrix} E[Z^2]&2E[Z^2]\\ 2E[Z^2]&4E[Z^2] \end{pmatrix} = \begin{pmatrix} 1&2\\ 2&4 \end{pmatrix}, $$ because $E[Z^2]=1$. The vector $(1,2)^T$ is an eigenvector with eigenvalue $5$, since multiplication by $\Sigma$ gives $(5,10)^T=5(1,2)^T$. The corresponding unit eigenvector is $u=(1,2)^T/\sqrt{5}$. The other eigenvalue is zero: the matrix has determinant $4-4=0$, and its rank is one.

The centered vector always lies on the span of $(1,2)^T$. Therefore the affine support is the line $$ \mu+\operatorname{span}\{(1,2)^T\} =\{(1,-1)^T+t(1,2)^T:t\in\mathbb{R}\}. $$ There is one varying direction, with variance $5$, and no variation perpendicular to this line. The distribution has a one-dimensional Gaussian density along the line, but no ordinary two-dimensional density.

How to Read the Geometry Reliably

A useful geometric summary is to separate three questions. First, where is the distribution centered? The answer is $\mu$. Second, which directions carry variation, and how much? The positive-eigenvalue eigenvectors give the directions, and their eigenvalues give the variances. Third, is the distribution full dimensional? It is if and only if all eigenvalues are positive. If any eigenvalue is zero, the distribution is supported on a proper affine subspace.

A common mistake is to treat a singular covariance as if it merely produced an extremely thin full-dimensional ellipsoid, then substitute it into the usual density formula. This fails: a zero eigenvalue makes $\Sigma^{-1}$ and $|\Sigma|^{-1/2}$ undefined. A very small positive eigenvalue does produce a narrow but full-dimensional distribution; an exactly zero eigenvalue instead removes a direction from the support. The distinction is mathematical, not just visual.

Another useful caution is that covariance ellipsoids describe relative density levels, not necessarily the outline of a bounded region containing all possible observations. For a nonsingular Gaussian, the support is all of $\mathbb{R}^p$, even though its density contours are bounded ellipsoids. In a singular Gaussian, the support itself is a lower-dimensional affine subspace, while its density contours within that subspace are still bounded ellipsoids.

Key takeaway: Positive covariance eigenvalues determine the squared scale of Gaussian variation along their eigenvectors. The resulting density contours are ellipsoids in the varying subspace; zero eigenvalues remove directions from the support and rule out a density with respect to full-dimensional volume.

Check Your Understanding

Use the eigenvalues and eigenvectors to reason about shape, scale, and support.

If a covariance matrix has eigenvalues $9$, $4$, and $1$, what are the semi-axis lengths of the contour at level $c=1$, and which information is needed to determine its orientation?
Why does a zero covariance eigenvalue imply that the corresponding centered coordinate is constant with probability one?
For a rank-two Gaussian vector in three dimensions, what is the dimension of its affine support, and does it have a density with respect to three-dimensional Lebesgue measure?
A covariance matrix has two equal positive eigenvalues. What is determined about the contour within their eigenspace, and what is not uniquely determined?
Explain why a small positive eigenvalue and an exactly zero eigenvalue lead to different support properties.

Spectral Geometry of Gaussian Covariance

What You'll Learn