Support of a Singular Gaussian Law

Why Singular Gaussian Laws Need a Support Description

A nonsingular Gaussian vector spreads probability throughout its ambient Euclidean space. A singular Gaussian vector does not: some nonzero linear combinations of its coordinates have zero variance, so the vector is confined to a lower-dimensional set. To describe where its probability is concentrated, we need more than the mean and covariance entries viewed separately. The relevant geometric object is the range of the covariance matrix, translated by the mean.

This description is exact. The vector lies in the resulting affine subspace with probability one, and every point of that subspace has positive probability in each of its neighborhoods. The latter condition matters: a set can contain all the probability without being the smallest closed set that does so.

Definition: Let $X$ be a random vector in $\mathbb{R}^d$. Its support is the set $$ \operatorname{supp}(X)=\{x\in\mathbb{R}^d: P(\|X-x\|<\varepsilon)>0\text{ for every }\varepsilon>0\}. $$ Here $\|\cdot\|$ is the Euclidean norm. Thus $x$ is in the support exactly when every open ball centered at $x$ has positive probability under the law of $X$.

For a $d\times d$ matrix $\Sigma$, its range is $\operatorname{Range}(\Sigma)=\{\Sigma v:v\in\mathbb{R}^d\}$. If $\mu\in\mathbb{R}^d$, the translated set $\mu+\operatorname{Range}(\Sigma)$ consists of all points $\mu+v$ with $v$ in that range. It is a linear subspace translated so that it passes through $\mu$, and is therefore an affine subspace.

The Support Characterization

Theorem: Suppose $X\sim N_d(\mu,\Sigma)$, where $\Sigma$ is a symmetric positive semidefinite $d\times d$ covariance matrix, possibly singular. Then $$ \operatorname{supp}(X)=\mu+\operatorname{Range}(\Sigma). $$ In particular, $P\{X\in\mu+\operatorname{Range}(\Sigma)\}=1$, and every point of this affine subspace belongs to the support.

We use the spectral decomposition of the covariance matrix. Let $r=\operatorname{rank}(\Sigma)$. Choose an orthogonal matrix $Q$ whose columns are orthonormal eigenvectors of $\Sigma$, ordered so that the first $r$ eigenvalues are positive: $\lambda_1,\ldots,\lambda_r>0$, with the remaining $d-r$ eigenvalues equal to zero. In the rotated coordinates $Q^T(X-\mu)$, the characteristic function factors into the characteristic functions of independent normal coordinates. The first $r$ coordinates have distributions $N(0,\lambda_j)$, while each remaining coordinate is zero with probability one. Equivalently, the law can be represented as $$ X\ \overset{d}{=}\ \mu+Q_r\Lambda_r^{1/2}Z, $$ where $Q_r$ contains the first $r$ columns of $Q$, $\Lambda_r=\operatorname{diag}(\lambda_1,\ldots,\lambda_r)$, and $Z\sim N_r(0,I_r)$. When $r=0$, the random part is identically zero. This representation is the useful consequence of singular multivariate normality for the support question.

Locate the possible values.
The random part in the representation is a linear combination of the first $r$ eigenvectors. Those eigenvectors span $\operatorname{Range}(\Sigma)$, so $X$ lies in the translated range with probability one.

Check every point in that set.
The positive-eigenvalue coordinates have a strictly positive normal density throughout $\mathbb{R}^r$. Consequently, a neighborhood of any point in the range has positive probability after mapping these coordinates into the affine subspace.

Exclude all other points.
The affine subspace is closed. A point outside it has an open ball disjoint from it, and that ball has probability zero because the random vector lies in the subspace almost surely.

Proof of the Theorem

Write $A=Q_r\Lambda_r^{1/2}$, so that $X\overset{d}{=}\mu+AZ$. The columns of $A$ are the positive-eigenvalue eigenvectors multiplied by positive constants. They are linearly independent, and their span is $\operatorname{Range}(\Sigma)$. Therefore every possible value $\mu+Az$ belongs to $\mu+\operatorname{Range}(\Sigma)$. The representation gives $P\{X\in\mu+\operatorname{Range}(\Sigma)\}=1$.

Now fix $x\in\mu+\operatorname{Range}(\Sigma)$. Since the columns of $A$ span the range, there is a $z_0\in\mathbb{R}^r$ such that $x=\mu+Az_0$. Let $\varepsilon>0$. The linear map $z\mapsto Az$ is continuous, so there is a $\delta>0$ such that $\|z-z_0\|<\delta$ implies $\|A(z-z_0)\|<\varepsilon$. Hence \[ \{\,\|Z-z_0\|<\delta\,\}\ \subseteq\ \{\,\|X-x\|<\varepsilon\,\} \] under the stated representation. If $r>0$, the standard normal density of $Z$ is strictly positive everywhere, so its integral over the nonempty open ball $\{z:\|z-z_0\|<\delta\}$ is positive. Thus $P(\|X-x\|<\varepsilon)>0$ for every $\varepsilon>0$, which proves that $x$ is in the support. If $r=0$, the affine subspace is the single point $\{\mu\}$, and $X=\mu$ almost surely; every neighborhood of $\mu$ has probability one.

Finally, suppose $x\notin\mu+\operatorname{Range}(\Sigma)$. The affine subspace is closed, so its complement is open. There is therefore an $\varepsilon>0$ such that the open ball of radius $\varepsilon$ around $x$ is disjoint from the affine subspace. Since $X$ belongs to that subspace almost surely, this ball has probability zero. Hence $x$ is not in the support. These two inclusions prove the claimed equality. $\square$

Worked Examples

Worked Example: A Gaussian Vector Supported on a Line

Let $Z\sim N(0,1)$ and set $X=(2+Z,\,-1+2Z,\,4-Z)^T$. Find its mean, covariance, and support.

The random part is $Za$, where $a=(1,2,-1)^T$. Since $E[Z]=0$ and $\operatorname{Var}(Z)=1$, \[ \mu=(2,-1,4)^T,\qquad \Sigma=aa^T= \begin{pmatrix} 1&2&-1\\ 2&4&-2\\ -1&-2&1 \end{pmatrix}. \] Every column of $\Sigma$ is a multiple of $a$, and its first column is $a$, so $\operatorname{Range}(\Sigma)=\operatorname{span}\{a\}$. The support is consequently \[ \{(2,-1,4)^T+t(1,2,-1)^T:t\in\mathbb{R}\}. \] In coordinates, points on the line satisfy $x_2=2x_1-5$ and $x_1+x_3=6$. Indeed, substituting $x_1=2+t$, $x_2=-1+2t$, and $x_3=4-t$ verifies both equations. The vector has probability one on this line, and every open ball centered at a point of the line has positive probability.

Worked Example: A Gaussian Vector Supported on a Plane

Suppose $X\sim N_3(\mu,\Sigma)$, with \[ \mu=(1,-2,5)^T,\qquad \Sigma= \begin{pmatrix} 4&0&0\\ 0&0&0\\ 0&0&9 \end{pmatrix}. \] Determine the support and its dimension.

The first and third coordinate directions are eigenvectors with positive eigenvalues $4$ and $9$; the second direction has eigenvalue zero. Thus $\operatorname{Range}(\Sigma)=\{(u,0,v)^T:u,v\in\mathbb{R}\}$, a two-dimensional linear subspace. Translating by $\mu$ gives \[ \operatorname{supp}(X)=\{(1+u,-2,5+v)^T:u,v\in\mathbb{R}\} =\{x\in\mathbb{R}^3:x_2=-2\}. \] The middle coordinate is $-2$ almost surely, while the first and third coordinates vary with positive normal variances. For any point $(x_1,-2,x_3)$ on the plane and any positive-radius ball around it, there is a small open rectangle around $(x_1,x_3)$ in the first-and-third coordinate plane whose points lie inside the ball. The independent normal coordinates have positive density on that rectangle, so the ball has positive probability. The support is a plane, not all of $\mathbb{R}^3$.

Worked Example: A Rank-One Covariance With a Nonzero Correlation

Let $Z\sim N(0,1)$ and let $X=(3+Z,\,1-2Z)^T$. Compute the covariance matrix and express the support as a line equation.

Here $\mu=(3,1)^T$, and the random part is $Z(1,-2)^T$. Therefore \[ \Sigma= \begin{pmatrix}1\\-2\end{pmatrix} \begin{pmatrix}1&-2\end{pmatrix} = \begin{pmatrix} 1&-2\\ -2&4 \end{pmatrix}. \] Its columns are multiples of $(1,-2)^T$, so its range is the span of that vector. The support is \[ \{(3,1)^T+t(1,-2)^T:t\in\mathbb{R}\}. \] Along this line, $x_1=3+t$ and $x_2=1-2t$, giving $2x_1+x_2=7$. Conversely, any point satisfying this equation has $t=x_1-3$ and $x_2=1-2t$, so it lies on the displayed line. This verifies the support equation in both directions.

Worked Example: The Rank-Zero Case

Let $X\sim N_2((4,-3)^T,0)$, where the covariance matrix is the $2\times2$ zero matrix. What is its support?

The range of the zero matrix is $\{(0,0)^T\}$, so the translated range is the singleton $\{(4,-3)^T\}$. Each coordinate has variance zero and equals its mean almost surely; therefore $P\{X=(4,-3)^T\}=1$. Every ball centered at $(4,-3)^T$ has probability one, while any other point has a sufficiently small ball that excludes $(4,-3)^T$ and has probability zero. Thus the support is exactly $\{(4,-3)^T\}$.

Support as the Smallest Closed Set of Full Probability

The support is not merely one closed set containing all the probability. It is the smallest such set under inclusion. For the Gaussian law above, the affine subspace $\mu+\operatorname{Range}(\Sigma)$ is closed and has probability one, so the theorem already shows that the support is a closed set of full probability.

To see why no proper closed subset of this affine subspace can also have probability one, let $C$ be a proper closed subset of it. Choose a support point $x\in\mu+\operatorname{Range}(\Sigma)$ with $x\notin C$. Because $C$ is closed, its complement is open, so there is an open ball centered at $x$ that is disjoint from $C$. Since $x$ is in the support, that ball has positive probability. Hence $P(X\in C)<1$. More generally, any closed set $C$ with $P(X\in C)=1$ must contain every support point: if a support point were outside $C$, closedness would again give a ball around it disjoint from $C$, and that ball would have positive probability despite being disjoint from a full-probability set. This argument uses closedness to obtain a disjoint neighborhood; merely noting that a neighborhood intersects a set would not suffice.

Common Pitfall: Confusing Support With an Ambient Density

If $\Sigma$ is singular, the Gaussian law does not have a density with respect to $d$-dimensional Lebesgue measure on all of $\mathbb{R}^d$. Its probability is concentrated on a lower-dimensional affine subspace, which has $d$-dimensional Lebesgue measure zero. It is therefore incorrect to look for an ordinary density that is positive throughout $\mathbb{R}^d$ and use that to identify the support.

A related mistake is to argue that points on the support must each have positive probability. For a Gaussian distribution of positive rank, the probability of an individual point is zero. Support instead concerns every open neighborhood of the point. In the rank-one line example, each particular point on the line has probability zero, but every ball around it has positive probability because it contains values arising from an interval of positive probability for $Z$.

Key takeaway: For $X\sim N_d(\mu,\Sigma)$, including the singular case, the exact support is $\mu+\operatorname{Range}(\Sigma)$. The covariance range gives the directions in which the vector can vary; translation by the mean places that set correctly in $\mathbb{R}^d$.

Check Your Understanding

Use the support definition and the mean-and-covariance characterization to answer each question.

If $\Sigma$ has rank $r$, what is the dimension of the support of $N_d(\mu,\Sigma)$? Explain using the range of $\Sigma$.
A Gaussian vector has mean $(0,0)^T$ and covariance matrix $\begin{pmatrix}1&1\\1&1\end{pmatrix}$. Find the range and give a line equation for its support.
Why does every ball centered at a point in the affine support have positive probability, even though a single point may have probability zero?
What is the support of a Gaussian vector whose covariance matrix is zero? What changes if its mean is changed?
Why does a closed set of full probability have to contain every support point?