Singular Multivariate Normal Laws

Why a Gaussian Vector Can Be Singular

A multivariate normal vector need not vary freely in every coordinate direction. Some of its coordinates may be exact linear combinations of others, or a linear combination may have zero variance and therefore be fixed. In these cases the covariance matrix is singular: it has rank less than the number of coordinates.

The key representation is simple but important. A Gaussian vector with \(p\) coordinates and covariance rank \(r\) can be generated from only \(r\) independent standard normal coordinates. When \(r<p\), the resulting vector is concentrated on a lower-dimensional affine set. That concentration is why the law has no density with respect to \(p\)-dimensional Lebesgue measure.

Definition: A random vector \(X\in\mathbb{R}^p\) is multivariate normal if every linear combination \(a^{\mathsf T}X\), for \(a\in\mathbb{R}^p\), has a univariate normal distribution, allowing a degenerate normal distribution with variance zero. Write its mean as \(\mu=E[X]\) and covariance matrix as \(\Sigma=\operatorname{Cov}(X)\). The vector is singular if \(\Sigma\) is not positive definite, equivalently if \(\operatorname{rank}(\Sigma)=r<p\).

A Gaussian vector with mean \(\mu\) and covariance \(\Sigma\) has characteristic function \(\varphi_X(t)=\exp\{i t^{\mathsf T}\mu-\tfrac12t^{\mathsf T}\Sigma t\}\), for \(t\in\mathbb{R}^p\). This formula includes singular covariance matrices. Positive semidefiniteness allows zero eigenvalues; positive definiteness, which is needed for the usual full-dimensional Gaussian density, does not.

Constructing the Lower-Rank Representation

Because \(\Sigma\) is symmetric and positive semidefinite, the spectral theorem gives an orthonormal eigenbasis with nonnegative eigenvalues. If its rank is \(r\), exactly \(r\) eigenvalues are positive. Retain those eigenvalues and their corresponding unit eigenvectors. If \(Q_r\) is the \(p\times r\) matrix of retained eigenvectors and \(\Lambda_r\) is the diagonal matrix of positive eigenvalues, define \(A=Q_r\Lambda_r^{1/2}\). Then \(\Sigma=AA^{\mathsf T}\), and \(A\) has \(r\) linearly independent columns.

Theorem: Let \(X\) be a multivariate normal vector in \(\mathbb{R}^p\) with mean \(\mu\) and covariance \(\Sigma\) of rank \(r\). There is a \(p\times r\) matrix \(A\) of rank \(r\) and a standard normal vector \(Z\sim N_r(0,I_r)\) such that \(X\) and \(\mu+AZ\) have the same distribution. If \(r=0\), the representation means \(X=\mu\) with probability one.

Proof. First suppose \(r>0\). Use the factorization \(\Sigma=AA^{\mathsf T}\) above, and let \(Z\sim N_r(0,I_r)\). For any \(t\in\mathbb{R}^p\), the characteristic function of \(\mu+AZ\) is

\begin{aligned} \varphi_{\mu+AZ}(t) &= \exp(i t^{\mathsf T}\mu)\,E[\exp(i t^{\mathsf T}AZ)]\\ &= \exp(i t^{\mathsf T}\mu)\, \exp\!\left(-\frac12 (A^{\mathsf T}t)^{\mathsf T}(A^{\mathsf T}t)\right)\\ &= \exp\!\left(i t^{\mathsf T}\mu-\frac12t^{\mathsf T}AA^{\mathsf T}t\right)\\ &= \exp\!\left(i t^{\mathsf T}\mu-\frac12t^{\mathsf T}\Sigma t\right) = \varphi_X(t). \end{aligned}

The characteristic function of a standard normal vector is used in the second equality. Since characteristic functions determine probability distributions, equality of these functions for every \(t\) proves \(X\overset{d}{=}\mu+AZ\). If \(r=0\), positive semidefiniteness and rank zero imply \(\Sigma=0\). Each coordinate then has variance zero and equals its mean with probability one, so \(X=\mu\) almost surely. This proves the theorem. \(\square\)

This representation also explains the covariance directly. For \(Y=\mu+AZ\), \(E[Y]=\mu\) because \(E[Z]=0\), and

\operatorname{Cov}(Y) =A\operatorname{Cov}(Z)A^{\mathsf T} =A I_r A^{\mathsf T} =AA^{\mathsf T} =\Sigma.

Thus the number of independent standard normal coordinates needed is the covariance rank. The \(p\) coordinates of \(X\) are not necessarily \(p\) independent sources of randomness.

Worked Example: A Two-Coordinate Vector on a Line

Let \(Z\sim N(0,1)\) and define \(X=(Z,2Z)^{\mathsf T}\). Find its mean and covariance, and explain why it has no density on \(\mathbb{R}^2\).

Write \(X=AZ\), where \(A=(1,2)^{\mathsf T}\). Since \(E[Z]=0\), \(E[X]=(0,0)^{\mathsf T}\). Its covariance is

\Sigma=AA^{\mathsf T} = \begin{pmatrix}1\\2\end{pmatrix} \begin{pmatrix}1&2\end{pmatrix} = \begin{pmatrix}1&2\\2&4\end{pmatrix}.

The second row is twice the first, so \(\operatorname{rank}(\Sigma)=1<2\). Every possible value of \(X\) satisfies \(x_2=2x_1\); in fact, this holds with probability one. The vector is therefore generated by one standard normal coordinate, although it has two displayed coordinates. The line \(x_2=2x_1\) has two-dimensional Lebesgue measure zero, so this law cannot have a density on \(\mathbb{R}^2\).

Why There Is No Full Ambient Density

For \(r<p\), the matrix \(A\) in the theorem has only \(r\) columns. Every value of \(\mu+AZ\) lies in the affine set \(\mu+\operatorname{col}(A)\), whose dimension is \(r\). Consequently,

P\bigl(X\in\mu+\operatorname{col}(A)\bigr)=1.

A proper affine subspace of \(\mathbb{R}^p\) has \(p\)-dimensional Lebesgue measure zero. To see this, translate it to a linear subspace and choose an orthogonal coordinate system in which that subspace lies in the set \(y_p=0\). This hyperplane has Lebesgue measure zero: within any bounded box, its measure is zero by integration in the \(y_p\) coordinate, and all of \(\mathbb{R}^p\) is a countable union of bounded boxes. Translation and orthogonal changes of coordinates preserve Lebesgue null sets, so the original affine subspace is null as well.

Result: If \(X\) is a singular Gaussian vector in \(\mathbb{R}^p\), its distribution has no density with respect to \(p\)-dimensional Lebesgue measure. It assigns probability one to an affine set of Lebesgue measure zero.

Indeed, if a density \(g\) with respect to \(p\)-dimensional Lebesgue measure existed, the probability of every Lebesgue-null set \(S\) would be \(\int_S g(x)\,dx=0\). But for \(S=\mu+\operatorname{col}(A)\), the probability is one. This contradiction proves the claim.

Worked Example: A Three-Coordinate Vector from Two Normal Coordinates

Let \(Z_1,Z_2\) be independent \(N(0,1)\) random variables and set \(X=(Z_1,Z_2,Z_1+Z_2)^{\mathsf T}\). Find a lower-rank representation and identify the exact constraint.

Here \(X=AZ\), with \(Z=(Z_1,Z_2)^{\mathsf T}\) and \(A=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}\). Therefore

\operatorname{Cov}(X)=AA^{\mathsf T} = \begin{pmatrix} 1&0&1\\ 0&1&1\\ 1&1&2 \end{pmatrix}.

The first two columns of \(A\) are linearly independent, so \(\operatorname{rank}(A)=2\) and \(\operatorname{rank}(AA^{\mathsf T})=2\). Thus the vector has three coordinates but only two standard normal sources of variation. Its coordinates obey \(X_3=X_1+X_2\) with probability one. The plane described by this equation has three-dimensional Lebesgue measure zero; hence \(X\) has no density on \(\mathbb{R}^3\).

The same constraint is visible in the covariance. For \(c=(-1,-1,1)^{\mathsf T}\), \(c^{\mathsf T}X=0\) almost surely, and

\operatorname{Var}(c^{\mathsf T}X) =c^{\mathsf T}\operatorname{Cov}(X)c =0.

A zero-variance linear combination is a deterministic constraint, not a normally varying direction.

Degenerate Coordinates and a Rank-Zero Case

Worked Example: One Coordinate Is Fixed

Let \(Z_1,Z_2\) be independent \(N(0,1)\) variables and define \(X=(4+3Z_1,\,7,\, -1+2Z_2)^{\mathsf T}\). Identify its mean, covariance rank, and a constraint.

The mean is \(\mu=(4,7,-1)^{\mathsf T}\). The representation uses \(A=\begin{pmatrix}3&0\\0&0\\0&2\end{pmatrix}\), so

\Sigma=AA^{\mathsf T} = \begin{pmatrix} 9&0&0\\ 0&0&0\\ 0&0&4 \end{pmatrix}.

This matrix has rank \(2\), less than the ambient dimension \(3\). The second coordinate is exactly 7, so the vector lies in the plane \(x_2=7\) with probability one. That plane has three-dimensional Lebesgue measure zero, and the distribution has no density on \(\mathbb{R}^3\).

At the extreme, if all eigenvalues of \(\Sigma\) are zero, then \(\Sigma=0\) and the Gaussian vector is the constant \(\mu\). Its probability is concentrated at one point. A point also has zero \(p\)-dimensional Lebesgue measure when \(p\geq1\), so a point mass is not a full-dimensional density either.

Common Mistake: Treating Singularity as a Failed Formula

A frequent error is to take the nonsingular Gaussian density formula and substitute a singular covariance matrix into it. That formula contains \((\det\Sigma)^{-1/2}\) and \(\Sigma^{-1}\). When \(\Sigma\) is singular, its determinant is zero and its inverse does not exist. This is not a matter of simplifying the same density formula: there is no density with respect to ambient Lebesgue measure to represent the law.

The correct reasoning is to describe the distribution through its Gaussian representation or characteristic function, then note that it puts probability one on a lower-dimensional set. The law remains a valid probability distribution even though it has no full-dimensional density. In particular, a zero-variance contrast signals exact dependence, not a small but nonzero amount of variation.

Find the covariance rank.
For covariance \(\Sigma\), determine \(r=\operatorname{rank}(\Sigma)\). Singularity means \(r<p\).

Factor the covariance.
Write \(\Sigma=AA^{\mathsf T}\) with \(A\) having \(r\) independent columns.

Use standard normal coordinates.
Represent the vector in distribution as \(\mu+AZ\), where \(Z\sim N_r(0,I_r)\).

Identify the concentration set.
The representation lies in \(\mu+\operatorname{col}(A)\), a lower-dimensional affine set, so it cannot have a density on the full ambient space.

Check Your Understanding

Use the representation and rank arguments from this tutorial to answer the following.

What is the covariance rank of \(X=(Z,3Z)^{\mathsf T}\) when \(Z\sim N(0,1)\), and what deterministic relation do its coordinates satisfy?
For a Gaussian vector in \(\mathbb{R}^5\) with covariance rank \(3\), how many independent standard normal coordinates are needed in the lower-rank representation?
Why does probability one on a Lebesgue-null affine subspace rule out a density with respect to ambient Lebesgue measure?
If \(\operatorname{Cov}(X)=0\), what does this imply about \(X\) and its distribution?
Why is substituting a singular covariance matrix into the nonsingular Gaussian density formula not a valid way to describe its law?