The information contained in a Gaussian covariance matrix
For a general random vector, the mean and covariance summarize its location and linear variation, but they do not record its whole distribution. For a Gaussian vector, they do: once the mean vector and covariance matrix are fixed, every linear combination has a specified univariate normal distribution. The resulting characteristic function therefore has no remaining parameters to choose.
We will establish this fact for covariance matrices that may be singular. Singularity matters because a Gaussian vector can be concentrated on a line, a plane, or a single point. In those cases it has no density with respect to full-dimensional Lebesgue measure, so a proof based on the usual nonsingular Gaussian density would not cover all cases.
Let \(X=(X_1,\ldots,X_d)^{\mathsf T}\) be a random vector on a probability space \((\Omega,\mathcal{F},\mathbb{P})\), taking values in \(\mathbb{R}^d\), with finite second moments. Its mean is \(\mu=\mathbb{E}X\), and its covariance matrix is \(\Sigma=\mathbb{E}[(X-\mu)(X-\mu)^{\mathsf T}]\). A covariance matrix is symmetric and positive semidefinite: for every \(t\in\mathbb{R}^d\), \(t^{\mathsf T}\Sigma t=\operatorname{Var}(t^{\mathsf T}X)\ge 0\). A matrix can be positive semidefinite without being invertible.
For a Gaussian vector with mean \(\mu\) and covariance \(\Sigma\), the scalar \(t^{\mathsf T}X\) has mean \(t^{\mathsf T}\mu\) and variance \(t^{\mathsf T}\Sigma t\). The univariate normal characteristic-function formula, including the zero-variance case, gives
Indeed, when \(t^{\mathsf T}\Sigma t=0\), the scalar \(t^{\mathsf T}X\) equals \(t^{\mathsf T}\mu\) almost surely, and the same expression reduces to \(\exp(i t^{\mathsf T}\mu)\). Thus the formula does not require an inverse of \(\Sigma\).
The characterization theorem
Proof. First assume the means and covariance matrices agree. For each \(t\in\mathbb{R}^d\), the Gaussian characteristic-function formula gives \(\varphi_X(t)=\exp(i t^{\mathsf T}\mu-\tfrac12t^{\mathsf T}\Sigma t)=\varphi_Y(t)\). The standard uniqueness theorem for characteristic functions states that two probability distributions on \(\mathbb{R}^d\) with equal characteristic functions at every \(t\) are equal. This is the standard Fourier-uniqueness result; its proof is part of measure-theoretic Fourier analysis and is not needed here. Applying it shows that \(X\) and \(Y\) have the same distribution. No step used \(\Sigma^{-1}\), so the argument includes singular \(\Sigma\). For the converse, equality in distribution implies equality of expectations of every integrable function of the vector. Applying this to each coordinate and to each product of two coordinates gives equality of means and second moments. Since \(\operatorname{Cov}(X)=\mathbb{E}[XX^{\mathsf T}]-\mu\mu^{\mathsf T}\), the covariance matrices also agree. \(\square\)
The theorem is an if-and-only-if characterization when restricted to Gaussian vectors: their distributions agree exactly when their means and covariance matrices agree. The forward direction of the proof explains the main mechanism: covariance fixes the variance of every projection \(t^{\mathsf T}X\), and Gaussianity turns those projection variances into the entire characteristic function.
Worked Example: Equal Covariance Under Different Coordinate Constructions
Let \(Z_1,Z_2\) be independent standard normal variables, and let \(W_1,W_2\) be independent standard normal variables on another probability space. Define \(X=(1+Z_1,\,2-Z_2)^{\mathsf T}\) and \(Y=(1+W_2,\,2+W_1)^{\mathsf T}\). Both vectors are Gaussian because every linear combination of their coordinates is a linear combination of independent normal variables.
The means are \(\mathbb{E}X=(1,2)^{\mathsf T}\) and \(\mathbb{E}Y=(1,2)^{\mathsf T}\). For \(X\), the centered coordinates are \(Z_1\) and \(-Z_2\); their variances are both \(1\), and their covariance is \(\mathbb{E}[-Z_1Z_2]=0\) by independence and zero means. Thus \(\operatorname{Cov}(X)=\begin{bmatrix}1&0\\0&1\end{bmatrix}\). For \(Y\), the centered coordinates are \(W_2\) and \(W_1\); their variances are both \(1\), and their covariance is \(\mathbb{E}[W_2W_1]=0\). Thus \(\operatorname{Cov}(Y)=\begin{bmatrix}1&0\\0&1\end{bmatrix}\) as well.
For \(t=(t_1,t_2)^{\mathsf T}\), the characteristic function of \(X\) is \(\exp(i(t_1+2t_2))\exp(-\tfrac12t_1^2)\exp(-\tfrac12t_2^2)\). The characteristic function of \(Y\) is \(\exp(i(t_1+2t_2))\exp(-\tfrac12t_1^2)\exp(-\tfrac12t_2^2)\) too: the centered random part is \(t_1W_2+t_2W_1\), with independent standard normal terms. Therefore the characteristic functions agree for every \(t\), verifying the theorem directly in this example. In particular, swapping which independent normal supplies each coordinate does not change the joint law.
Singular covariance: the law can live on a lower-dimensional set
If \(\Sigma\) is singular, some nonzero direction \(a\) satisfies \(a^{\mathsf T}\Sigma a=0\). For a Gaussian vector this means \(\operatorname{Var}(a^{\mathsf T}X)=0\), so \(a^{\mathsf T}X=a^{\mathsf T}\mu\) almost surely. The vector is therefore confined to a lower-dimensional affine set. This is not a failure of the theorem; the characteristic-function formula still identifies the distribution in every direction.
Worked Example: A Gaussian Vector Supported on a Line
Let \(U,V\) be independent standard normal variables and let \(W\) be a standard normal variable. Define \(X=(U+V,\,2U+2V)^{\mathsf T}\) and \(Y=(\sqrt{2}W,\,2\sqrt{2}W)^{\mathsf T}\), with the variables for the two vectors allowed to be on different probability spaces. Each vector is Gaussian, and both have mean \((0,0)^{\mathsf T}\).
For \(X\), the first coordinate has variance \(1+1=2\), the second has variance \(4+4=8\), and their covariance is \(\mathbb{E}[(U+V)(2U+2V)]=2\mathbb{E}[U^2]+2\mathbb{E}[V^2]=4\). Thus \(\operatorname{Cov}(X)=\begin{bmatrix}2&4\\4&8\end{bmatrix}\). For \(Y\), the coordinate variances are \((\sqrt{2})^2=2\) and \((2\sqrt{2})^2=8\); their covariance is \(\sqrt{2}(2\sqrt{2})\mathbb{E}[W^2]=4\). Hence \(\operatorname{Cov}(Y)=\begin{bmatrix}2&4\\4&8\end{bmatrix}\).
For \(t=(t_1,t_2)^{\mathsf T}\), write \(a=t_1+2t_2\). Then \(t^{\mathsf T}X=a(U+V)\), whose variance is \(2a^2\), so \(\varphi_X(t)=\exp(-a^2)\). Also \(t^{\mathsf T}Y=\sqrt{2}aW\), whose variance is \(2a^2\), so \(\varphi_Y(t)=\exp(-a^2)\). This checks equality of their characteristic functions. The covariance determinant is \(2\cdot8-4\cdot4=0\), and both vectors satisfy \(X_2=2X_1\) or \(Y_2=2Y_1\) almost surely. Their common law is concentrated on that line, yet it is fully determined by the displayed mean and covariance.
A singular covariance matrix does not mean every coordinate is constant. It means that at least one nontrivial linear combination has zero variance. In this example, the combination \(X_2-2X_1\) is constant at zero, while \(X_1\) itself has variance \(2\). The characteristic-function argument accommodates both the deterministic direction and the variable direction at once.
Worked Example: Two Representations of a Singular Three-Dimensional Law
Let \(U,V\) be independent standard normal variables, and let \(A,B\) be another independent standard normal pair. Set \(X=(U,V,U+V)^{\mathsf T}\) and \(Y=((A+B)/\sqrt{2},\,(A-B)/\sqrt{2},\,\sqrt{2}A)^{\mathsf T}\). Both vectors have mean zero. We compute their covariance matrices entry by entry.
For \(X\), the variances are \(1,1,2\). Its coordinate covariances are \(\operatorname{Cov}(U,V)=0\), \(\operatorname{Cov}(U,U+V)=1\), and \(\operatorname{Cov}(V,U+V)=1\). Therefore \(\operatorname{Cov}(X)=\begin{bmatrix}1&0&1\\0&1&1\\1&1&2\end{bmatrix}\).
For \(Y\), the first variance is \((1+1)/2=1\), the second is \((1+1)/2=1\), and the third is \(2\). The covariance of the first two coordinates is \(\mathbb{E}[(A+B)(A-B)]/2=(\mathbb{E}[A^2]-\mathbb{E}[B^2])/2=0\). The first-third covariance is \(\mathbb{E}[(A+B)A]=1\), and the second-third covariance is \(\mathbb{E}[(A-B)A]=1\). So \(\operatorname{Cov}(Y)=\begin{bmatrix}1&0&1\\0&1&1\\1&1&2\end{bmatrix}\), as required.
To check the law directly, the centered linear combination for \(X\) is \((t_1+t_3)U+(t_2+t_3)V\), with variance \((t_1+t_3)^2+(t_2+t_3)^2\). For \(Y\), the coefficients of \(A\) and \(B\) in \(t^{\mathsf T}Y\) are \((t_1+t_2+2t_3)/\sqrt{2}\) and \((t_1-t_2)/\sqrt{2}\). Its variance is their squared sum: \[ \frac{(t_1+t_2+2t_3)^2+(t_1-t_2)^2}{2} =(t_1+t_3)^2+(t_2+t_3)^2. \] Thus both characteristic functions equal the exponential of minus one-half this common variance. The covariance matrix is singular because the third coordinate equals the sum of the first two almost surely in both constructions. This example verifies the theorem even when different linear representations of the same lower-dimensional Gaussian law are used.
Why the Gaussian assumption cannot be dropped
The proof depends on Gaussianity at the point where the covariance determines the characteristic function. For an arbitrary random vector, equal mean and covariance do not determine its law. A common error is to use the Gaussian formula \(\exp(i t^{\mathsf T}\mu-\tfrac12t^{\mathsf T}\Sigma t)\) for a vector that has not been shown to be Gaussian. The formula follows from normality of every projection; it is not a general consequence of knowing only two moments.
Worked Example: Same Mean and Variance, Different Laws
Let \(R\) take values \(1\) and \(-1\), each with probability \(\tfrac12\), and let \(Z\) be standard normal. Both have mean zero. For \(R\), \(\mathbb{E}[R^2]=\tfrac12(1)+\tfrac12(1)=1\), so \(\operatorname{Var}(R)=1\). For \(Z\), \(\operatorname{Var}(Z)=1\). Their means and variances match, but they do not have the same distribution: \(\mathbb{P}(R=0)=0\), whereas \(\mathbb{P}(Z=0)=0\) also, so that event does not distinguish them. Instead, \(\mathbb{P}(|R|\le\tfrac12)=0\), while \(\mathbb{P}(|Z|\le\tfrac12)>0\), because the standard normal density is strictly positive on the interval \([-\tfrac12,\tfrac12]\). Thus the laws differ despite equal first and second moments.
The correction is to check the Gaussian assumption before using covariance to identify a law. For Gaussian vectors, the characteristic-function formula and uniqueness theorem prove the conclusion, including singular cases. For non-Gaussian vectors, additional distributional information is generally needed.
The distinction is useful whenever a model is specified through covariance. In Gaussian modeling, covariance does more than quantify pairwise linear variation: together with the mean, it specifies all finite-dimensional probabilities. In non-Gaussian modeling, covariance is only a partial summary. The singular examples also show why densities and matrix inverses should not be built into the characterization: the law may be entirely valid while supported on a lower-dimensional set.
Check Your Understanding
For each question, distinguish what follows from Gaussianity from what follows from covariance alone.
- Why does \(t^{\mathsf T}\Sigma t=0\) imply that \(t^{\mathsf T}X\) is constant almost surely, and what is that constant?
- For a Gaussian vector with mean \(\mu\) and covariance \(\Sigma\), write its characteristic function and identify where singularity of \(\Sigma\) could have caused a problem in a density-based approach.
- For the line-supported vector in the second worked example, verify that the nonzero vector \(a=(-2,1)^{\mathsf T}\) satisfies \(a^{\mathsf T}\Sigma a=0\).
- Which named standard result turns equality of characteristic functions at every argument into equality of distributions?
- Give a reason the equal-mean, equal-covariance conclusion cannot be applied to the Rademacher and standard normal variables in the final example.