Gaussian Vectors and Covariance Geometry

What Makes a Vector Gaussian?

A collection of random variables can each have a normal distribution without having a jointly Gaussian distribution. The distinction matters: joint Gaussianity gives a precise description of how the coordinates vary together, and it makes the covariance matrix informative about the whole joint law. The key definition tests every linear combination of the coordinates, not just the coordinates one at a time.

Let $(\Omega,\mathcal{F},\mathbb{P})$ be a probability space, and let $X=(X_1,\ldots,X_d)^\mathsf{T}$ be a random vector taking values in $\mathbb{R}^d$. We use column vectors for $X$, its mean $\mu$, and coefficient vectors $a$. A univariate normal distribution is allowed to have variance zero; in that case it is a constant random variable.

Definition: The vector $X$ is multivariate normal (or Gaussian) if, for every $a\in\mathbb{R}^d$, the scalar $a^\mathsf{T}X$ has a univariate normal distribution, possibly degenerate. Its mean vector is $\mu=\mathbb{E}[X]$, and its covariance matrix is $\Sigma=\operatorname{Cov}(X)=\mathbb{E}[(X-\mu)(X-\mu)^\mathsf{T}]$. The entry $\Sigma_{ij}$ is $\operatorname{Cov}(X_i,X_j)$.

A Gaussian vector has finite second moments, so these quantities are defined. The matrix $\Sigma$ is symmetric, and its diagonal entries are the coordinate variances. More generally, for every coefficient vector $a$, the variance in direction $a$ is

\operatorname{Var}(a^\mathsf{T}X)=a^\mathsf{T}\Sigma a.

This identity follows by expanding the scalar variance: $a^\mathsf{T}(X-\mu)$ multiplied by itself has expectation $a^\mathsf{T}\mathbb{E}[(X-\mu)(X-\mu)^\mathsf{T}]a$. Since a variance cannot be negative, $a^\mathsf{T}\Sigma a\ge0$ for every $a$. Thus a covariance matrix is positive semidefinite as well as symmetric. Geometrically, $a^\mathsf{T}\Sigma a$ measures the variation seen when the vector is projected onto direction $a$.

Theorem: For a Gaussian vector with mean $\mu$ and covariance matrix $\Sigma$, the characteristic function is $$ \varphi_X(t)=\mathbb{E}\big[e^{\,i t^\mathsf{T}X}\big] =\exp\!\left(i t^\mathsf{T}\mu-\tfrac12 t^\mathsf{T}\Sigma t\right),\qquad t\in\mathbb{R}^d. $$ The standard uniqueness theorem for characteristic functions says that a probability distribution is determined by its characteristic function. It is a Fourier-analysis result; its proof is beyond this tutorial. Consequently, the mean vector and covariance matrix specify the joint distribution once Gaussianity is known.

Proof. Fix any $t\in\mathbb{R}^d$. By the definition of a Gaussian vector, $t^\mathsf{T}X$ is normal. Its mean is $t^\mathsf{T}\mu$, and its variance is $t^\mathsf{T}\Sigma t$, by the variance identity above. A normal random variable with mean $m$ and variance $v\ge0$ has characteristic function $\exp(iu m-u^2v/2)$ at $u$. For completeness, the standard normal formula follows by differentiating its characteristic-function integral and integrating by parts: if $Z$ has density $(2\pi)^{-1/2}e^{-x^2/2}$, then $\varphi_Z'(u)=-u\varphi_Z(u)$, with $\varphi_Z(0)=1$, so $\varphi_Z(u)=e^{-u^2/2}$. Shifting and scaling give the stated formula, including $v=0$. Apply it to $t^\mathsf{T}X$ at $u=1$ to obtain the display. The uniqueness theorem then gives the final assertion. $\square$

Marginals and Covariance Geometry

A marginal distribution concerns only a selection of coordinates. If $X$ is Gaussian, any coordinate subvector is Gaussian: every linear combination of the selected coordinates is also a linear combination of all the coordinates of $X$, and so is normal by definition. Its mean vector is the corresponding subvector of $\mu$, and its covariance matrix is the matching principal submatrix of $\Sigma$. This identifies the marginal law without needing to integrate out the other coordinates.

The covariance matrix also describes how coordinate variances combine. In two dimensions, for $a=(a_1,a_2)^\mathsf{T}$, the variance is $a_1^2\Sigma_{11}+2a_1a_2\Sigma_{12}+a_2^2\Sigma_{22}$. The cross term records whether the coordinates tend to move together or in opposite directions. If both variances are positive, their correlation is $\rho=\Sigma_{12}/\sqrt{\Sigma_{11}\Sigma_{22}}$. Positive semidefiniteness requires $|\rho|\le1$: the quadratic form in $a_1,a_2$ cannot be negative, so its discriminant cannot be positive when its diagonal coefficients are positive. When $|\rho|=1$, some nonzero direction has zero variance; when $|\rho|<1$, no such direction exists in two dimensions.

Worked Example: A Correlated Gaussian Pair

Let $Z_1,Z_2$ be independent standard normal random variables on a probability space, and set $X_1=Z_1$, $X_2=Z_1+Z_2$. We first check that $X=(X_1,X_2)^\mathsf{T}$ is Gaussian, rather than inferring it just from the coordinate distributions. For any $a=(a_1,a_2)^\mathsf{T}$,

a^\mathsf{T}X=(a_1+a_2)Z_1+a_2Z_2.

Independence and the standard normal characteristic function give a normal distribution for this sum; its characteristic function at $u$ is $\exp[-u^2\{(a_1+a_2)^2+a_2^2\}/2]$. Thus every linear combination is normal, as required.

The means are $\mathbb{E}[X_1]=0$ and $\mathbb{E}[X_2]=0+0=0$. The variances are $\operatorname{Var}(X_1)=1$ and $\operatorname{Var}(X_2)=1+1=2$, using independence. The covariance is $\operatorname{Cov}(Z_1,Z_1+Z_2)=1+0=1$. Therefore

\mu=\begin{bmatrix}0\\0\end{bmatrix}, \qquad \Sigma=\begin{bmatrix}1&1\\1&2\end{bmatrix}.

The marginal distributions are $X_1\sim N(0,1)$ and $X_2\sim N(0,2)$. The joint characteristic function, computed directly from the linear combination, is

\mathbb{E}[e^{\,i(t_1X_1+t_2X_2)}] =\exp\!\left[-\tfrac12\{(t_1+t_2)^2+t_2^2\}\right] =\exp\!\left[-\tfrac12(t_1^2+2t_1t_2+2t_2^2)\right].

Check: Multiplying $t^\mathsf{T}\Sigma t$ gives $t_1^2+2t_1t_2+2t_2^2$, exactly the exponent’s quadratic form. Also, the covariance matrix is positive semidefinite: its diagonal entries are positive and its determinant is $1\cdot2-1\cdot1=1>0$.

Positive semidefiniteness is not merely a necessary check; it is the basic consistency condition for a proposed covariance matrix. In two dimensions, for a symmetric matrix with positive diagonal entries, it requires $\Sigma_{11}\Sigma_{22}-\Sigma_{12}^2\ge0$. A covariance matrix encodes the shape and orientation of variation: directions with larger $a^\mathsf{T}\Sigma a$ have greater projected variance. These geometric facts do not require a density or a nonsingular covariance matrix.

Uncorrelated Coordinates and Independence

For general random variables, zero covariance does not imply independence. For Gaussian vectors, however, a block of zero cross-covariances separates the joint characteristic function into a product. This gives a precise independence criterion.

Theorem: Suppose a Gaussian vector is divided into two coordinate subvectors $U$ and $V$. If every covariance between a coordinate of $U$ and a coordinate of $V$ is zero, then $U$ and $V$ are independent.

Proof. Write the mean as $(\mu_U,\mu_V)$ and the covariance matrix in blocks. The assumed cross-covariances are zero, so the covariance matrix has diagonal blocks $\Sigma_U,\Sigma_V$ and zero off-diagonal blocks. For coefficient vectors $s,t$, the characteristic-function formula gives $\varphi_{(U,V)}(s,t)=\exp(i s^\mathsf{T}\mu_U-\tfrac12s^\mathsf{T}\Sigma_U s)\exp(i t^\mathsf{T}\mu_V-\tfrac12t^\mathsf{T}\Sigma_V t)$. These factors are the marginal characteristic functions of $U$ and $V$. The joint characteristic function therefore equals the characteristic function of the product of the marginal distributions. By uniqueness of characteristic functions, the joint distribution is that product distribution, which is independence. $\square$

Worked Example: Zero Covariance Gives Independence Here

Let $Z_1,Z_2$ again be independent standard normal variables, and define $U=Z_1+Z_2$, $V=Z_1-Z_2$. For any real $s,t$, $sU+tV=(s+t)Z_1+(s-t)Z_2$, which is normal. Hence $(U,V)$ is Gaussian. Its means are both zero, and its variances are $\operatorname{Var}(U)=1+1=2$ and $\operatorname{Var}(V)=1+1=2$. Its covariance is $\operatorname{Cov}(U,V)=\operatorname{Var}(Z_1)-\operatorname{Var}(Z_2)=1-1=0$. Thus its mean and covariance are

\begin{bmatrix}0\\0\end{bmatrix}, \qquad \begin{bmatrix}2&0\\0&2\end{bmatrix}.

More explicitly, its joint characteristic function is $\exp[-\{(s+t)^2+(s-t)^2\}/2]=\exp(-s^2)\exp(-t^2)$. The factors are the characteristic functions of $N(0,2)$ for $U$ and $N(0,2)$ for $V$. The factorization verifies that the joint law is the product of those marginals, so $U$ and $V$ are independent.

Check: The quadratic form from the covariance matrix is $2s^2+2t^2$. The direct calculation gives $(s+t)^2+(s-t)^2=s^2+2st+t^2+s^2-2st+t^2=2s^2+2t^2$, in agreement.

A Vector with Nonzero Means

The mean vector locates the center of the coordinates, while covariance describes variation around that center. For any coefficient vector $a$, the linear combination $a^\mathsf{T}X$ has mean $a^\mathsf{T}\mu$ and variance $a^\mathsf{T}\Sigma a$. Both quantities are needed to specify its normal marginal.

Worked Example: Means, Marginals, and Directional Variances

Let $Z_1,Z_2$ be independent standard normal variables and define $X_1=1+2Z_1$, $X_2=-1+Z_1+\sqrt{2}Z_2$. To check joint Gaussianity, take arbitrary $a_1,a_2$. Then $a_1X_1+a_2X_2$ is a constant plus $(2a_1+a_2)Z_1+\sqrt{2}a_2Z_2$, which is normal by independence and the normal characteristic function.

The means are $\mathbb{E}[X_1]=1$ and $\mathbb{E}[X_2]=-1$. The variances are $\operatorname{Var}(X_1)=2^2=4$ and $\operatorname{Var}(X_2)=1^2+(\sqrt{2})^2=3$. The covariance is $\operatorname{Cov}(2Z_1,Z_1+\sqrt{2}Z_2)=2\cdot1+0=2$. Hence

\mu=\begin{bmatrix}1\\-1\end{bmatrix}, \qquad \Sigma=\begin{bmatrix}4&2\\2&3\end{bmatrix}.

It follows that $X_1\sim N(1,4)$ and $X_2\sim N(-1,3)$. As a directional check, $X_1+X_2$ has mean $1+(-1)=0$ and variance $4+2(2)+3=11$, so $X_1+X_2\sim N(0,11)$. Also $2X_1-X_2$ has mean $2(1)-(-1)=3$ and variance $4(4)+(-1)^2(3)+2(2)(-1)(2)=16+3-8=11$, so $2X_1-X_2\sim N(3,11)$.

Check: Directly, $X_1+X_2=3Z_1+\sqrt{2}Z_2$, with variance $3^2+(\sqrt{2})^2=9+2=11$. Also $2X_1-X_2=3+3Z_1-\sqrt{2}Z_2$, with mean $3$ and variance $3^2+(-\sqrt{2})^2=9+2=11$. The direct calculations agree with the covariance-matrix formula. The matrix is positive definite because its determinant is $4\cdot3-2\cdot2=8>0$ and its first diagonal entry is $4>0$.

Common Pitfall: Covariance Is Not a Universal Independence Test

A common mistake is to conclude that two variables are independent whenever their covariance is zero. Gaussianity is essential to the result proved above. For a concrete counterexample, let $R$ take values $-1,0,1$, each with probability $1/3$, and put $Y=R^2-\tfrac23$. Then $\mathbb{E}[R]=0$, $\mathbb{E}[Y]=\tfrac23-\tfrac23=0$, and $\operatorname{Cov}(R,Y)=\mathbb{E}[R^3]-\mathbb{E}[R]\mathbb{E}[Y]=0$. But $Y$ is determined by $R$: it equals $\tfrac13$ when $R=\pm1$, and $-\tfrac23$ when $R=0$. In particular, the event $\{Y=-\tfrac23\}$ is exactly $\{R=0\}$, so these variables are not independent.

The correction is to ask first whether the vector is jointly Gaussian. If it is, zero cross-covariances imply independence by the characteristic-function argument. If it is not known to be Gaussian, zero covariance alone only says that the centered variables have zero expected product; it does not determine their joint distribution.

Check Your Understanding

Use the definition of a Gaussian vector and the covariance formulas to answer each question.

For $X=(Z_1,Z_1+Z_2)^\mathsf{T}$, where $Z_1,Z_2$ are independent standard normal variables, find $\operatorname{Var}(2X_1-X_2)$ from its covariance matrix.
Explain why every coordinate subvector of a Gaussian vector is Gaussian.
For a Gaussian pair with covariance matrix $\begin{bmatrix}1&-1\\-1&1\end{bmatrix}$, find the variance of $X_1+X_2$. What does the answer say about that direction?
State the condition under which zero cross-covariances between two Gaussian subvectors imply their independence.
Give one reason that zero covariance is insufficient to conclude independence for arbitrary random variables.