Singular Covariance Matrices in Multivariate Limits

Why a Multivariate Normal Limit Can Be Singular

A multivariate central limit theorem describes the joint behavior of a vector, but its limiting coordinates need not vary independently in every direction. If one coordinate is a linear combination of others, or if some variation disappears asymptotically, the covariance matrix of the limit is singular. The limit is still a valid multivariate normal distribution; it is concentrated on a lower-dimensional subset of the full coordinate space.

The practical task is to interpret that concentration correctly. A zero eigenvalue identifies a direction with no limiting random variation, and each such direction gives a deterministic linear constraint on the limiting vector. This is useful both for understanding the limit and for avoiding inappropriate calculations, such as trying to invert its covariance matrix.

Definition and Geometry

Definition: A \(d\)-dimensional random vector \(Z\) has a possibly degenerate multivariate normal distribution \(N_d(\mu,\Sigma)\) if every linear combination \(a^T Z\), for \(a\in\mathbb{R}^d\), is univariate normal, allowing a constant as a normal variable of variance zero. Here \(\mu=E[Z]\), and \(\Sigma=E[(Z-\mu)(Z-\mu)^T]\) is a symmetric positive semidefinite matrix. The distribution is singular if \(\Sigma\) is not invertible.

Positive semidefiniteness means \(a^T\Sigma a\geq 0\) for every vector \(a\). It is necessary because \(a^T\Sigma a=\operatorname{Var}(a^TZ)\). A covariance matrix may have a zero eigenvalue without violating this condition. In that case, some nonzero direction has zero variance.

The distribution \(N_d(\mu,\Sigma)\) can also be specified by its characteristic function:

\varphi_Z(t)=\exp\left(i t^T\mu-\frac{1}{2}t^T\Sigma t\right),\qquad t\in\mathbb{R}^d.

This formula remains valid when \(\Sigma\) is singular. In particular, the definition does not require a density with respect to \(d\)-dimensional Lebesgue measure.

Theorem: Let \(Z\sim N_d(\mu,\Sigma)\), where \(\Sigma\) is positive semidefinite. For any \(a\in\mathbb{R}^d\), \(a^TZ\) is constant almost surely if and only if \(a\) belongs to the null space of \(\Sigma\). In that case, \(a^TZ=a^T\mu\) almost surely. If \(r=\operatorname{rank}(\Sigma)\), the distribution is supported on the affine space \(\mu+\operatorname{Range}(\Sigma)\), of dimension \(r\).

Proof. The variance of the linear combination is

\operatorname{Var}(a^TZ)=a^T\Sigma a.

If \(\Sigma a=0\), then \(a^T\Sigma a=0\), so \(a^TZ\) has variance zero and equals its mean \(a^T\mu\) almost surely. Conversely, suppose \(a^TZ\) is constant almost surely. Its variance is zero, so \(a^T\Sigma a=0\). By the spectral decomposition of the symmetric positive semidefinite matrix \(\Sigma\), write \(\Sigma=Q\Lambda Q^T\), with nonnegative diagonal entries in \(\Lambda\). Then \(a^T\Sigma a=\sum_j\lambda_j (Q^Ta)_j^2=0\). Each term is nonnegative, so every coordinate corresponding to a positive \(\lambda_j\) is zero. Therefore \(\Sigma a=0\). This proves the equivalence.

For the support statement, the spectral decomposition also gives a matrix \(A\) with \(r\) linearly independent columns such that \(\Sigma=AA^T\). If \(G\sim N_r(0,I_r)\), then \(\mu+AG\) has characteristic function \(\exp(i t^T\mu-\tfrac12 t^TAA^Tt)\), so it has distribution \(N_d(\mu,\Sigma)\). Every value of \(\mu+AG\) lies in \(\mu+\operatorname{Range}(\Sigma)\), since \(\operatorname{Range}(A)=\operatorname{Range}(AA^T)\). The standard normal \(G\) has positive density throughout \(\mathbb{R}^r\), and the map \(g\mapsto\mu+Ag\) is one-to-one. Thus the distribution fills this affine space, whose dimension is \(r\). \(\square\)

Reading a Singular Limit in a Central Limit Theorem

In a multivariate CLT, the limiting covariance matrix is allowed to be positive semidefinite rather than positive definite. The Cramér–Wold device tests convergence of every scalar projection. If the projection in direction \(a\) has limiting variance \(a^T\Sigma a=0\), its limiting normal distribution is the constant \(a^T\mu\). Such a projection is not an exception to the theorem; it records a deterministic direction in the joint limit.

Key implication: If \(Z_n\xrightarrow{d}N_d(\mu,\Sigma)\) and \(a\) is in the null space of \(\Sigma\), then \(a^TZ_n\xrightarrow{d}a^T\mu\). Thus every null-space vector specifies a linear constraint on the limiting distribution. The constraint concerns the limit; it need not hold exactly for every finite \(n\).

The theorem above describes all such constraints at once. If the rank is \(r<d\), there are \(d-r\) linearly independent null-space directions and hence \(d-r\) independent linear constraints. Equivalently, the limiting vector varies in only \(r\) directions. Its singularity does not mean that the limit fails to exist or that the joint normal approximation is unusable.

Worked Example: Two Coordinates with Exact Linear Dependence

Let \(B_1,B_2,\ldots\) be independent Bernoulli random variables with \(P(B_i=1)=P(B_i=0)=1/2\). Define \(X_i=(B_i-\tfrac12,\ 3(B_i-\tfrac12))^T\), and consider \(Z_n=n^{-1/2}\sum_{i=1}^n X_i\). Find its limiting distribution and constraint.

Each \(X_i\) has mean zero. Since \(B_i-\tfrac12\) takes the values \(-1/2\) and \(1/2\) with equal probability, its variance is \(1/4\). Therefore

\operatorname{Cov}(X_i)=\frac14 \begin{pmatrix} 1&3\\ 3&9 \end{pmatrix}.

The determinant is \(\tfrac{1}{16}(9-9)=0\), and the matrix has rank one. Let \(S_n=n^{-1/2}\sum_{i=1}^n(B_i-\tfrac12)\). The univariate CLT gives \(S_n\xrightarrow{d}G\), where \(G\sim N(0,1/4)\). For \(t=(t_1,t_2)^T\), the projection \(t^TZ_n\) equals \((t_1+3t_2)S_n\), so its limiting distribution is normal with mean zero and variance \((t_1+3t_2)^2/4\). The Cramér–Wold criterion therefore gives

Z_n\xrightarrow{d}(G,3G)^T,\qquad G\sim N(0,1/4).

The vector \((-3,1)^T\) is in the covariance matrix's null space. Accordingly, the limit satisfies \(-3Z_1+Z_2=0\) almost surely. Here the constraint holds for every \(n\) as well, because the second coordinate is exactly three times the first.

Worked Example: Three Coordinates with One Constraint

For each \(i\), let \(A_i\) and \(C_i\) be independent Rademacher variables, each taking values \(-1\) and \(1\) with equal probability. Assume the pairs are independent across \(i\), and form \(X_i=(A_i,C_i,-A_i-C_i)^T\). Find the limit of \(Z_n=n^{-1/2}\sum_{i=1}^n X_i\).

The means are zero, and \(\operatorname{Var}(A_i)=\operatorname{Var}(C_i)=1\), while \(\operatorname{Cov}(A_i,C_i)=0\). Calculating the entries from the three components gives

\Sigma=\operatorname{Cov}(X_i)= \begin{pmatrix} 1&0&-1\\ 0&1&-1\\ -1&-1&2 \end{pmatrix}.

The third component is the negative sum of the first two, so the columns of \(\Sigma\) sum to zero. Its upper-left \(2\) by \(2\) submatrix is the identity, so its rank is at least two; the zero column sum makes its rank at most two. Thus its rank is exactly two, and \((1,1,1)^T\) spans its null space.

For any \(t=(t_1,t_2,t_3)^T\), the one-row projection is

t^TX_i=(t_1-t_3)A_i+(t_2-t_3)C_i, \qquad \operatorname{Var}(t^TX_i)=(t_1-t_3)^2+(t_2-t_3)^2.

The scalar CLT applied to each projection, together with the Cramér–Wold criterion, yields the limit \(Z_n\xrightarrow{d}(G_1,G_2,-G_1-G_2)^T\), where \(G_1,G_2\) are independent \(N(0,1)\) variables. Its covariance is the displayed \(\Sigma\), and the null-space constraint is \(Z_1+Z_2+Z_3=0\) almost surely. The limit has three coordinates but only two random degrees of freedom.

Worked Example: A Singular Covariance Appearing Only in the Limit

Let \(X_i,Y_i\) be mutually independent Rademacher variables, independent across \(i\). For each \(n\), set

Z_n=\frac{1}{\sqrt n}\sum_{i=1}^n \begin{pmatrix} X_i\\ X_i+n^{-1/2}Y_i \end{pmatrix}.

Each summand is centered. Independence and unit variances give

\operatorname{Cov}(Z_n)= \begin{pmatrix} 1&1\\ 1&1+n^{-1} \end{pmatrix}.

Its determinant is \(1/n>0\), so this covariance is nonsingular for every finite \(n\). However, it converges to \(\Sigma=\begin{pmatrix}1&1\\1&1\end{pmatrix}\), which has rank one. To calculate the joint limit, fix \(t=(t_1,t_2)^T\). The scalar projection \(t^TZ_n\) is a normalized sum of independent centered variables, with variance

(t_1+t_2)^2+\frac{t_2^2}{n}.

For fixed \(t\), the absolute value of each unnormalized projected summand is at most \(|t_1+t_2|+|t_2|/\sqrt n\), a bound independent of \(n\). After division by \(\sqrt n\), the summands are eventually smaller than any fixed positive threshold. Thus the Lindeberg condition holds for these projections. The scalar CLT and Cramér–Wold give the limit \(Z_n\xrightarrow{d}(G,G)^T\), where \(G\sim N(0,1)\). The null-space direction \((-1,1)^T\) gives the limiting constraint \(Z_2-Z_1=0\), even though that constraint is not exact at finite \(n\).

Common Mistake: Requiring an Inverse

A common error is to conclude that a multivariate normal limit is invalid because its covariance matrix has determinant zero, or to try to use a formula involving \(\Sigma^{-1}\) as though the limit had a full-dimensional density. For a singular limit, there is no density with respect to \(d\)-dimensional Lebesgue measure: the distribution is concentrated on a lower-dimensional affine space. The correction is to work with its characteristic function, its linear projections, or a representation \(Z=\mu+AG\), and to use the null space to state its deterministic constraints. Do not invert \(\Sigma\) unless it is positive definite and the calculation actually requires an inverse.

Singularity can also be mistaken for independence. Zero variance in a direction means a particular linear combination is constant; it does not say that the original coordinates are independent. In the first example, the coordinates are perfectly linearly related. In the second, all three coordinates obey a sum constraint, although the first two limiting coordinates are independent.

Key takeaway: A singular covariance matrix describes a valid Gaussian limit with fewer random directions than coordinates. Its null space gives the deterministic linear constraints, while its rank gives the dimension of the affine support.

Check Your Understanding

Use the covariance matrix and the limiting distribution to answer each question.

If \(Z\sim N_3(\mu,\Sigma)\) and \(a\) is in the null space of \(\Sigma\), what value does \(a^TZ\) take almost surely?
A singular \(4\) by \(4\) covariance matrix has rank two. What is the dimension of the affine support, and how many independent null-space constraints are there?
For a limiting vector with covariance \(\begin{pmatrix}1&2\\2&4\end{pmatrix}\), identify a nonzero null-space vector and state the corresponding constraint when the mean is zero.
Can the covariance matrices of a sequence of random vectors all be nonsingular while their limiting covariance matrix is singular? Explain using the near-dependence example.
Why does a singular multivariate normal distribution not have a density with respect to full-dimensional Lebesgue measure?

Singular Covariance Matrices in Multivariate Limits

What You'll Learn