Valid Gaussian Covariance Matrices

Why Covariance Matrices Need Validity Conditions

A proposed covariance matrix is more than a table of variances and pairwise covariances: its entries must be mutually compatible with a joint probability distribution. In particular, every linear combination of the variables must have a nonnegative variance. This requirement leads to a precise matrix condition, and, for Gaussian vectors, that condition is also enough to construct a probability law.

The earlier material on means and covariances of Gaussian vectors gives the entries of a covariance matrix and the Gaussian characteristic function. Here the focus is the validity question: which matrices can actually occur as covariance matrices, and can each valid matrix serve as the covariance of a Gaussian vector?

Symmetry and Positive Semidefiniteness

Definition: A real \(p\times p\) matrix \(\Sigma\) is symmetric positive semidefinite if \(\Sigma^T=\Sigma\) and \(a^T\Sigma a\geq 0\) for every \(a\in\mathbb{R}^p\). It is positive definite if \(a^T\Sigma a>0\) for every nonzero \(a\in\mathbb{R}^p\). Positive definiteness is stronger than positive semidefiniteness.

For a random vector \(X=(X_1,\ldots,X_p)^T\) with finite second moments, its covariance matrix is \(\Sigma=\operatorname{Cov}(X)\), whose \((i,j)\) entry is \(\operatorname{Cov}(X_i,X_j)\). The diagonal entries are variances. Symmetry follows from \(\operatorname{Cov}(X_i,X_j)=\operatorname{Cov}(X_j,X_i)\). The quadratic-form condition captures the requirement that no linear combination can have negative variance.

Theorem: If \(X\) is a random vector with finite second moments and covariance matrix \(\Sigma\), then \(\Sigma\) is symmetric positive semidefinite. In particular, every covariance matrix is symmetric, and its diagonal entries are nonnegative.

Proof. Symmetry follows from the symmetry of covariance. For any fixed \(a=(a_1,\ldots,a_p)^T\), linearity of expectation gives

a^T\Sigma a =\sum_{i=1}^p\sum_{j=1}^p a_i a_j\operatorname{Cov}(X_i,X_j) =\operatorname{Var}\!\left(\sum_{i=1}^p a_iX_i\right) \geq 0.

The variance exists because the coordinates have finite second moments and the sum is finite. Since this holds for every \(a\), \(\Sigma\) is positive semidefinite. Taking \(a\) to be the \(i\)th coordinate vector shows that \(\Sigma_{ii}=\operatorname{Var}(X_i)\geq 0\). \(\square\)

Positive semidefiniteness also places restrictions on combinations of off-diagonal entries and variances. For example, for two variables, the covariance matrix has the form

\Sigma= \begin{pmatrix} \sigma_1^2 & c\\ c & \sigma_2^2 \end{pmatrix}.

Its quadratic form must be nonnegative for all pairs of real numbers. In particular, the covariance cannot be chosen independently of the two variances. The theorem gives the general condition without requiring a separate constraint for each pair of variables.

Worked Example: Rejecting an Impossible Covariance Matrix

Consider the proposed matrix

\Sigma= \begin{pmatrix} 1 & 1.4\\ 1.4 & 1 \end{pmatrix}.

It is symmetric and has positive diagonal entries, but those checks are not enough. Take \(a=(1,-1)^T\). Then

a^T\Sigma a = \begin{pmatrix}1&-1\end{pmatrix} \begin{pmatrix}1&1.4\\1.4&1\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix} =1-1.4-1.4+1=-0.8.

If \(\Sigma\) were a covariance matrix, this number would be the variance of \(X_1-X_2\), which cannot be negative. Therefore no random vector with finite second moments can have this covariance matrix. The failure is not merely that a correlation would be outside its allowable range; it is directly a violation of the required quadratic-form condition.

Why the Condition Is Also Sufficient for a Gaussian Law

For Gaussian vectors, symmetry and positive semidefiniteness are not only necessary. They are sufficient. The key construction uses the spectral theorem for real symmetric matrices: there is an orthogonal matrix \(Q\) and a diagonal matrix of real eigenvalues such that \(\Sigma=Q\operatorname{diag}(\lambda_1,\ldots,\lambda_p)Q^T\). If \(\Sigma\) is positive semidefinite, each \(\lambda_i\geq 0\), so its nonnegative square root is defined.

Theorem: Let \(\Sigma\) be any real symmetric positive semidefinite \(p\times p\) matrix, and let \(\mu\in\mathbb{R}^p\). There exists a Gaussian random vector \(X\) with mean \(\mu\) and covariance matrix \(\Sigma\).

Proof. By the spectral theorem, write \(\Sigma=Q\Lambda Q^T\), where \(Q^TQ=I\) and \(\Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_p)\). Positive semidefiniteness implies \(\lambda_j\geq 0\) for every \(j\): for a unit eigenvector \(q_j\), \(\lambda_j=q_j^T\Sigma q_j\geq 0\). Let \(Z=(Z_1,\ldots,Z_p)^T\) have independent \(N(0,1)\) coordinates, and define

X=\mu+Q\Lambda^{1/2}Z, \qquad \Lambda^{1/2}=\operatorname{diag}(\sqrt{\lambda_1},\ldots,\sqrt{\lambda_p}).

Since \(E[Z]=0\), \(E[X]=\mu\). Also \(\operatorname{Cov}(Z)=I\), so direct covariance calculation gives

\operatorname{Cov}(X) =Q\Lambda^{1/2}I\Lambda^{1/2}Q^T =Q\Lambda Q^T =\Sigma.

To verify Gaussianity, take any \(a\in\mathbb{R}^p\). Then \(a^TX=a^T\mu+b^TZ\), where \(b=\Lambda^{1/2}Q^Ta\). Independence and the standard normal characteristic function give

E[e^{it a^TX}] =e^{it a^T\mu}\prod_{j=1}^p e^{-t^2b_j^2/2} =e^{it a^T\mu-t^2a^T\Sigma a/2}.

This is the characteristic function of a univariate normal variable with mean \(a^T\mu\) and variance \(a^T\Sigma a\); when that variance is zero, it is the characteristic function of the constant \(a^T\mu\). Thus every linear combination of coordinates is normal, so \(X\) is Gaussian. This constructs the claimed law. \(\square\)

The construction permits zero eigenvalues. They simply give zero coefficients for some standard normal coordinates. Consequently, a valid covariance matrix need not be positive definite: positive semidefiniteness is the exact condition for the existence of a Gaussian law. Positive definiteness is an additional condition, relevant when a nonsingular density is required.

Worked Example: Constructing a Gaussian Vector With a Singular Covariance

Let \(Z\sim N(0,1)\), and define \(X=(X_1,X_2)^T\) by \(X_1=2Z\) and \(X_2=Z\). Its mean is zero, and the covariance entries are

\operatorname{Var}(X_1)=4,\qquad \operatorname{Var}(X_2)=1,\qquad \operatorname{Cov}(X_1,X_2)=E[2Z^2]=2.

Therefore \(\operatorname{Cov}(X)=\begin{pmatrix}4&2\\2&1\end{pmatrix}\). For any \(a=(u,v)^T\),

a^T\operatorname{Cov}(X)a=4u^2+4uv+v^2=(2u+v)^2\geq 0.

The matrix is positive semidefinite. It is not positive definite, since the nonzero vector \((1,-2)^T\) gives quadratic form zero. This does not prevent a Gaussian law from existing: the explicit vector above has exactly this covariance. The coordinates are perfectly linearly related, \(X_1=2X_2\), but the construction and covariance calculation are valid.

Worked Example: A Three-Dimensional Gaussian Law From a Gram Form

Let \(Z_1,Z_2\) be independent \(N(0,1)\) variables and set \(X_1=Z_1\), \(X_2=Z_1+Z_2\), and \(X_3=Z_2\). The variances are \(1,2,1\), respectively. The covariances are \(\operatorname{Cov}(X_1,X_2)=1\), \(\operatorname{Cov}(X_2,X_3)=1\), and \(\operatorname{Cov}(X_1,X_3)=0\). Hence

\Sigma= \begin{pmatrix} 1&1&0\\ 1&2&1\\ 0&1&1 \end{pmatrix}.

For \(a=(a_1,a_2,a_3)^T\), the quadratic form is

a^T\Sigma a =a_1^2+2a_1a_2+2a_2^2+2a_2a_3+a_3^2 =(a_1+a_2)^2+(a_2+a_3)^2\geq 0.

Thus \(\Sigma\) is positive semidefinite, and the displayed construction verifies directly that it is a Gaussian covariance matrix. The form can equal zero for nonzero \(a\), for example \(a=(1,-1,1)^T\), so the matrix is not positive definite. The example illustrates why positive semidefinite, rather than positive definite, is the general validity condition.

Common Pitfall and Practical Checks

A common mistake is to check only that a matrix is symmetric and that its diagonal entries are nonnegative, then conclude that it is a covariance matrix. The first example shows why this reasoning fails: those checks hold, yet a linear combination has negative variance. The correction is to verify the entire positive-semidefinite condition, \(a^T\Sigma a\geq 0\) for every \(a\), or use an equivalent valid matrix test.

Conversely, a zero determinant or a zero eigenvalue does not by itself invalidate a covariance matrix. It means the matrix is not positive definite, but it may still be positive semidefinite and may still specify a Gaussian law. The construction theorem includes this case. Whether such a law has a density with respect to ordinary \(p\)-dimensional volume is a separate question.

These conditions are useful when specifying a multivariate Gaussian model, checking an estimated or proposed covariance matrix, or deciding whether a stated collection of variances and covariances is internally possible. The quadratic-form interpretation gives a direct conceptual check: every weighted sum of coordinates must have a legitimate, nonnegative variance.

Key takeaway: A covariance matrix must be symmetric positive semidefinite. Conversely, every real symmetric positive semidefinite matrix is the covariance matrix of a Gaussian vector, for any chosen mean vector. Zero eigenvalues are allowed; they do not make the covariance specification invalid.

Check Your Understanding

Use the definitions and construction in this tutorial to answer the following.

Why must \(a^T\Sigma a\) be nonnegative for every \(a\) when \(\Sigma\) is a covariance matrix?
A symmetric matrix has positive diagonal entries. Is that sufficient to guarantee that it is a covariance matrix? Explain.
What does the sufficiency theorem say about a symmetric positive semidefinite matrix with a zero eigenvalue?
In the three-dimensional example, verify that \(a=(1,-1,1)^T\) gives \(a^T\Sigma a=0\). What does this imply about positive definiteness?
In the Gaussian construction, why does \(\lambda_j=0\) cause no problem when forming \(\Lambda^{1/2}\)?

Valid Gaussian Covariance Matrices

What You'll Learn