Gaussian Vectors and Covariance Parameters

Why a Covariance Matrix Matters

A single numerical measurement has one mean and one variance. A vector of measurements has a mean for each component and a covariance for each pair. Together, these quantities describe the location and linear dependence structure of a Gaussian vector. But not every symmetric matrix can serve as its covariance: the matrix must be positive semidefinite. Whether it is positive definite or singular determines whether the distribution has an ordinary density throughout the full space or is concentrated on a lower-dimensional set.

Definition and Parameters

Definition: A random vector $X=(X_1,\ldots,X_p)^\mathsf{T}$ has a multivariate normal distribution with mean vector $\mu\in\mathbb{R}^p$ and covariance matrix $\Sigma$ if it has the same distribution as $$ X=\mu+\Sigma^{1/2}Z, $$ where $Z\sim N_p(0,I_p)$, and $\Sigma^{1/2}$ is the symmetric positive-semidefinite square root of $\Sigma$. Here $\Sigma$ must be symmetric and positive semidefinite. We write $X\sim N_p(\mu,\Sigma)$.

The mean vector is $\mu=E[X]$. The covariance matrix is $\Sigma=\operatorname{Cov}(X)$, with entries $\Sigma_{ij}=\operatorname{Cov}(X_i,X_j)$. In particular, $\Sigma_{ii}=\operatorname{Var}(X_i)$, and the covariance between components $i$ and $j$ is in the two symmetric entries $\Sigma_{ij}$ and $\Sigma_{ji}$. Where both variances are positive, their correlation is $$ \operatorname{Corr}(X_i,X_j)=\frac{\Sigma_{ij}}{\sqrt{\Sigma_{ii}\Sigma_{jj}}}. $$

The vector $\mu$ has $p$ entries. A symmetric $p\times p$ covariance matrix has $p(p+1)/2$ distinct entries, but these entries cannot be chosen freely: positive semidefiniteness restricts their possible values. The mean controls location; the diagonal of $\Sigma$ controls component-wise spread; and its off-diagonal entries describe linear association. The covariance entries retain the units of the corresponding products, while correlations are unitless.

The defining representation gives $E[X]=\mu$ and $\operatorname{Cov}(X)=\Sigma$. Indeed, $E[Z]=0$ and $\operatorname{Cov}(Z)=I_p$, so $$ E[X]=\mu+\Sigma^{1/2}E[Z]=\mu,\qquad \operatorname{Cov}(X)=\Sigma^{1/2}I_p(\Sigma^{1/2})^\mathsf{T}=\Sigma. $$ It also gives the characteristic function $$ \varphi_X(t)=E[\exp(i t^\mathsf{T}X)] =\exp\!\left(i t^\mathsf{T}\mu-\frac{1}{2}t^\mathsf{T}\Sigma t\right),\qquad t\in\mathbb{R}^p. $$ Thus the distribution is determined by its mean and covariance. Equivalently, every linear combination $a^\mathsf{T}X$ is univariate normal, with mean $a^\mathsf{T}\mu$ and variance $a^\mathsf{T}\Sigma a$; a zero variance is allowed and gives a constant. These facts follow from the representation: $a^\mathsf{T}X=a^\mathsf{T}\mu+(\Sigma^{1/2}a)^\mathsf{T}Z$, which is normal with the stated mean and variance.

Positive-Definite and Singular Covariance Matrices

A symmetric matrix $\Sigma$ is positive semidefinite if $a^\mathsf{T}\Sigma a\geq 0$ for every $a\in\mathbb{R}^p$. It is positive definite if $a^\mathsf{T}\Sigma a>0$ for every nonzero $a$. Because $a^\mathsf{T}\Sigma a=\operatorname{Var}(a^\mathsf{T}X)$, positive semidefiniteness is necessary for a covariance matrix: a variance cannot be negative.

Theorem: If $\Sigma$ is positive definite, $X\sim N_p(\mu,\Sigma)$ has a density with respect to $p$-dimensional Lebesgue measure, $$ f(x)=\frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}} \exp\!\left[-\frac{1}{2}(x-\mu)^\mathsf{T}\Sigma^{-1}(x-\mu)\right],\qquad x\in\mathbb{R}^p. $$ If $\Sigma$ is singular, $X$ is concentrated on a proper affine subspace of $\mathbb{R}^p$ and has no density with respect to $p$-dimensional Lebesgue measure.

To derive the density in the positive-definite case, write $X=\mu+\Sigma^{1/2}Z$. When $\Sigma$ is positive definite, $\Sigma^{1/2}$ is invertible, so the change-of-variables formula applies. Its determinant is $|\Sigma^{1/2}|=|\Sigma|^{1/2}$, and the standard normal density of $Z=\Sigma^{-1/2}(x-\mu)$ transforms to the displayed formula.

For the singular case, use the spectral decomposition of the symmetric positive-semidefinite matrix $\Sigma$. If its rank is $r<p$, then $\Sigma^{1/2}$ has rank $r$, so $\Sigma^{1/2}Z$ lies in its $r$-dimensional column space with probability one. Consequently, $X$ lies in $\mu+\operatorname{range}(\Sigma)$, a proper affine subspace. Such a subspace has $p$-dimensional Lebesgue measure zero. A distribution that assigns probability one to a set of Lebesgue measure zero cannot have a density with respect to that measure. In a null direction $a$, meaning $\Sigma a=0$, the component $a^\mathsf{T}X$ is exactly $a^\mathsf{T}\mu$ with probability one.

Worked Example: A Positive-Definite Covariance and a Linear Contrast

Let $X\sim N_2(\mu,\Sigma)$, with $$ \mu=\begin{pmatrix}1\\-2\end{pmatrix},\qquad \Sigma=\begin{pmatrix}4&1\\1&9\end{pmatrix}. $$ Check that $\Sigma$ is positive definite, find the correlation, and calculate the distribution of $2X_1-X_2$.

The leading diagonal entry is $4>0$, and $$ |\Sigma|=4(9)-1(1)=35>0. $$ For a symmetric $2\times2$ matrix, these two inequalities imply positive definiteness. Thus this Gaussian vector has a density on all of $\mathbb{R}^2$.

The component variances are $4$ and $9$, while their covariance is $1$. Therefore $$ \operatorname{Corr}(X_1,X_2)=\frac{1}{\sqrt{4\cdot9}}=\frac{1}{6}. $$ For $T=2X_1-X_2$, take $a=(2,-1)^\mathsf{T}$. A linear combination of a Gaussian vector is normal, with mean $a^\mathsf{T}\mu$ and variance $a^\mathsf{T}\Sigma a$. Direct calculation gives $$ E[T]=2(1)-(-2)=4,\qquad \operatorname{Var}(T)= \begin{pmatrix}2&-1\end{pmatrix} \begin{pmatrix}4&1\\1&9\end{pmatrix} \begin{pmatrix}2\\-1\end{pmatrix} =2(7)+(-1)(-7)=21. $$ Hence $T\sim N(4,21)$.

Worked Example: A Singular Gaussian Vector on a Line

Let $Z\sim N(0,1)$ and define $X_1=Z$, $X_2=2Z+1$. Find the mean and covariance, and describe where $X$ lies.

The mean is $\mu=(0,1)^\mathsf{T}$. The component variances are $1$ and $4$, and their covariance is $\operatorname{Cov}(Z,2Z+1)=2\operatorname{Var}(Z)=2$. Thus $$ \Sigma=\begin{pmatrix}1&2\\2&4\end{pmatrix},\qquad |\Sigma|=1(4)-2(2)=0. $$ This matrix is positive semidefinite but singular: for any $a=(a_1,a_2)^\mathsf{T}$, $$ a^\mathsf{T}\Sigma a=(a_1+2a_2)^2\geq0. $$ The components satisfy $X_2=2X_1+1$ with probability one. The distribution is concentrated on that line, which has zero area in $\mathbb{R}^2$, so it has no ordinary two-dimensional density. Singularity here records an exact linear constraint, not an error in the model.

Worked Example: Checking a Correlation Parameter

Suppose two components have variance $9$ and $16$, respectively, and proposed correlation $-0.5$. Construct their covariance matrix and check whether it is positive definite.

The covariance must be correlation times the product of standard deviations: $$ \operatorname{Cov}(X_1,X_2)=-0.5\sqrt{9}\sqrt{16}=-0.5(3)(4)=-6. $$ The proposed covariance matrix is therefore $$ \Sigma=\begin{pmatrix}9&-6\\-6&16\end{pmatrix}. $$ Its leading diagonal entry is positive, and its determinant is $9(16)-(-6)^2=144-36=108>0$. Hence it is positive definite and can specify a nonsingular bivariate Gaussian distribution. The corresponding correlation, checked from the matrix entries, is $-6/\sqrt{9\cdot16}=-6/12=-0.5$, as required.

Linear Transformations: A Derived Result

Changing coordinates or forming several linear contrasts changes both the mean and covariance. The following result gives the transformed parameters directly and also shows why a transformation can produce a singular covariance even when the original covariance is positive definite.

Theorem: If $X\sim N_p(\mu,\Sigma)$, $A$ is a fixed $k\times p$ matrix, and $b\in\mathbb{R}^k$, then $$ Y=AX+b\sim N_k(A\mu+b,A\Sigma A^\mathsf{T}). $$ The transformed covariance is positive semidefinite; it is positive definite exactly when $A^\mathsf{T}c\neq0$ for every nonzero $c\in\mathbb{R}^k$ for which this condition is possible, provided $\Sigma$ is positive definite.

To prove the distributional statement, use the characteristic function of $Y$. For any $t\in\mathbb{R}^k$, $$ E[\exp(i t^\mathsf{T}Y)] =\exp\!\left(i t^\mathsf{T}(A\mu+b)-\frac12 t^\mathsf{T}A\Sigma A^\mathsf{T}t\right). $$ This is the characteristic function of the stated Gaussian distribution. Its covariance is positive semidefinite because $$ c^\mathsf{T}A\Sigma A^\mathsf{T}c=(A^\mathsf{T}c)^\mathsf{T}\Sigma(A^\mathsf{T}c)\geq0. $$ If $\Sigma$ is positive definite, the same expression is strictly positive for every nonzero $c$ exactly when $A^\mathsf{T}c\neq0$ for every nonzero $c$. This is equivalent to $A$ having full row rank. Thus a transformation with redundant output coordinates can be Gaussian but singular, even if the input covariance is positive definite. $\square$

Common Mistake and Why the Distinction Matters

A common mistake is to assume that every Gaussian vector has the displayed density involving $\Sigma^{-1}$. That formula requires $\Sigma$ to be positive definite. If $\Sigma$ is singular, its inverse does not exist, and the distribution instead lives on a lower-dimensional affine set. Another mistake is to accept any symmetric matrix as a covariance matrix; a negative value of $a^\mathsf{T}\Sigma a$ would imply a negative variance and makes the proposed model impossible.

The distinction matters when specifying a model, interpreting exact linear relationships, and choosing calculations that require a full-dimensional density or an inverse covariance matrix. A singular matrix is not automatically invalid: it is a valid covariance matrix when it is positive semidefinite. Positive definiteness is the stronger condition that rules out zero-variance linear combinations and permits the usual full-dimensional Gaussian density.

Check Your Understanding

Try these questions by identifying the relevant parameter or matrix property in each case.

For $X\sim N_p(\mu,\Sigma)$, what does the entry $\Sigma_{ij}$ represent, and what does $\Sigma_{ii}$ represent?
Why must $a^\mathsf{T}\Sigma a$ be nonnegative for every vector $a$ if $\Sigma$ is a covariance matrix?
What feature of the distribution changes when a positive-definite covariance matrix is replaced by a singular positive-semidefinite one?
If $X\sim N_p(\mu,\Sigma)$ and $Y=AX+b$, state the mean and covariance of $Y$.
Can a singular covariance matrix specify a valid Gaussian model? Explain what a null direction implies for the corresponding linear combination of components.