Singular Gaussian Distributions and Their Support

When a Gaussian Vector Has No Full-Dimensional Density

A covariance matrix need not be invertible. For example, if one coordinate of a random vector is always twice another, their covariance matrix has a zero eigenvalue. The vector is still Gaussian, but it never leaves a line in the plane. In higher dimensions, the same phenomenon places the entire distribution on an affine subspace. Recognizing that subspace is essential: an ordinary density formula requiring an inverse covariance matrix cannot describe a singular Gaussian.

Throughout, let \((\Omega,\mathcal{F},\mathbb{P})\) be a probability space, and let \(X\) be a random vector with state space \(\mathbb{R}^d\). For a vector \(m\in\mathbb{R}^d\) and a \(d\times d\) positive semidefinite matrix \(\Sigma\), define a Gaussian vector with mean \(m\) and covariance \(\Sigma\) by \(X=m+\Sigma^{1/2}Z\), where \(Z\) has \(d\) independent standard normal coordinates and \(\Sigma^{1/2}\) is the positive semidefinite square root of \(\Sigma\). This definition includes singular \(\Sigma\). It is equivalent to defining a Gaussian vector as an affine transformation of a finite-dimensional standard normal vector: an affine transformation with matrix \(A\) has covariance \(AA^{\mathsf T}\), and a singular value decomposition shows that its law is also represented by the square root of that covariance.

Two elementary facts explain why positive semidefiniteness is the right condition. First, for any \(a\in\mathbb{R}^d\), \(a^{\mathsf T}\Sigma a=\operatorname{Var}(a^{\mathsf T}X)\ge0\), so a covariance matrix is positive semidefinite. Second, the spectral theorem gives an orthogonal matrix \(U\) and nonnegative eigenvalues \(\lambda_1,\ldots,\lambda_d\) for every real symmetric positive semidefinite \(\Sigma\). Thus \(\Sigma^{1/2}=U\operatorname{diag}(\sqrt{\lambda_1},\ldots,\sqrt{\lambda_d})U^{\mathsf T}\) exists, and using it in the definition constructs a Gaussian vector with covariance \(\Sigma\). Indeed, \(\mathbb{E}X=m\), and \(\mathbb{E}[(X-m)(X-m)^{\mathsf T}]=\Sigma^{1/2}I\Sigma^{1/2}=\Sigma\).

The Support Subspace

Proof. Write the spectral decomposition as \(\Sigma=U\operatorname{diag}(\lambda_1,\ldots,\lambda_d)U^{\mathsf T}\), ordering the eigenvalues so that \(\lambda_1,\ldots,\lambda_r\) are strictly positive and the remaining eigenvalues are zero. Let \(u_i\) be the \(i\)th column of \(U\), and let \(Z_1,\ldots,Z_d\) be the independent standard normal coordinates in the definition of \(X\). Then \(X=m+\sum_{i=1}^{r}\sqrt{\lambda_i}Z_i u_i\), because every term with eigenvalue zero vanishes. The vectors \(u_1,\ldots,u_r\) span \(\operatorname{Range}(\Sigma)\), so this expression places \(X\) in \(m+\operatorname{Range}(\Sigma)\) with probability one. If \(r=0\), the sum is empty and \(X=m\) with probability one, giving the claimed singleton support.

Suppose now \(r>0\), and fix any \(x=m+\sum_{i=1}^r c_i u_i\) in the affine subspace. The map from \(z\in\mathbb{R}^r\) to \(m+\sum_{i=1}^r\sqrt{\lambda_i}z_i u_i\) is a continuous bijection from \(\mathbb{R}^r\) onto the subspace, with continuous inverse, since each \(\lambda_i>0\). The standard normal vector \((Z_1,\ldots,Z_r)\) has a strictly positive density everywhere in \(\mathbb{R}^r\). Therefore every neighborhood of the point \(z_i=c_i/\sqrt{\lambda_i}\) has positive probability, and continuity of the map implies that every open ball around \(x\) has positive probability. Thus every point of the affine subspace is in the support. Finally, the subspace is closed. If \(y\) lies outside it, its distance from the subspace is strictly positive, so an open ball around \(y\) is disjoint from the subspace and has probability zero. Such a \(y\) is not in the support. This proves both inclusions. \(\square\)

The same decomposition identifies the deterministic linear constraints. A vector \(a\) is in the null space of \(\Sigma\) exactly when \(a\) is orthogonal to its range. For each such \(a\), \(a^{\mathsf T}(X-m)=0\) with probability one. In particular, \(a^{\mathsf T}\Sigma a=0\) means the scalar random variable \(a^{\mathsf T}X\) has variance zero, so it equals its mean \(a^{\mathsf T}m\) almost surely. These constraints describe the support, provided one takes enough independent vectors from the null space.

Dimension, Density, and a Common Mistake

If \(r=\operatorname{rank}(\Sigma)<d\), the Gaussian law is concentrated on an \(r\)-dimensional affine subspace. Consequently, that subspace has zero \(d\)-dimensional Lebesgue measure, while the Gaussian law assigns it probability one. There can be no ordinary density with respect to \(d\)-dimensional Lebesgue measure. In coordinates along the \(r\) positive-eigenvalue directions, however, the law has the usual \(r\)-dimensional Gaussian density. When \(r=0\), the distribution is a point mass rather than a positive-dimensional density.

A common mistake is to use the nonsingular Gaussian density formula \((2\pi)^{-d/2}(\det\Sigma)^{-1/2}\exp[-\tfrac12(x-m)^{\mathsf T}\Sigma^{-1}(x-m)]\) when \(\det\Sigma=0\). For the covariance \(\begin{bmatrix}1&1\\1&1\end{bmatrix}\), the determinant is \(1\cdot1-1\cdot1=0\), so the inverse and the displayed density do not exist. The correction is not to force a limiting value into that formula: the vector \((Z,Z)\) has probability one on the line \(x_2=x_1\), and its support is that line. A useful diagnostic is to find the null space of \(\Sigma\): its vectors give the exact linear constraints that hold almost surely.

To identify a singular Gaussian support in practice, diagonalize the covariance or find its range and null space. Positive eigenvalues correspond to random directions; zero eigenvalues correspond to deterministic linear combinations. Translate the resulting range by the mean, and verify any proposed constraints by checking both their mean values and their variances. This approach describes where the law lives without invoking an inverse covariance matrix or conditional Gaussian formulas.