Positive Semidefiniteness of Covariance Matrices

Why Covariance Matrices Must Have a Positivity Property

A covariance matrix collects variances on its diagonal and covariances away from the diagonal. Those entries describe uncertainty in individual coordinates and how coordinates vary together. But the matrix has a further constraint: no linear combination of the coordinates can have negative variance. The algebraic expression that captures this constraint is a quadratic form.

This property is useful because it gives a concise way to check whether a proposed matrix could represent covariance structure. It also explains the meaning of the matrix beyond its individual entries: each quadratic form corresponds to the variance of a particular weighted sum.

Definition: A real symmetric \(p \times p\) matrix \(A\) is positive semidefinite if \(a^T A a \ge 0\) for every vector \(a \in \mathbb{R}^p\). The inequality is allowed to be an equality; positive semidefiniteness does not require every nonzero \(a\) to give a strictly positive value.

Let \(X=(X_1,\ldots,X_p)^T\) be a random vector whose coordinates have finite second moments, so \(E[X_i^2] < \infty\) for each \(i\). Its covariance matrix is \(\Sigma=(\operatorname{Cov}(X_i,X_j))_{i,j=1}^p\), where \(\operatorname{Cov}(X_i,X_j)=E[(X_i-E[X_i])(X_j-E[X_j])]\). These moment conditions ensure that every entry is finite. Covariance is symmetric, so \(\Sigma\) is a real symmetric matrix.

The Quadratic Form Is a Variance

Take any fixed vector \(a=(a_1,\ldots,a_p)^T \in \mathbb{R}^p\), and form the scalar random variable \(S_a=a^T X=\sum_{i=1}^p a_iX_i\). Because the sum has finitely many terms with finite second moments, \(S_a\) also has a finite second moment. Its variance is nonnegative, and expanding that variance gives the central identity.

Theorem: If \(X\) has finite second moments and covariance matrix \(\Sigma\), then for every \(a \in \mathbb{R}^p\), \[ a^T\Sigma a=\operatorname{Var}(a^T X)\ge 0. \] Consequently, every covariance matrix is positive semidefinite.

Proof. Write \(\mu_i=E[X_i]\). By the definition of variance and the fact that a finite sum can be expanded term by term,

\begin{aligned} \operatorname{Var}(a^T X) &=E\left[\left(\sum_{i=1}^p a_i(X_i-\mu_i)\right)^2\right]\\ &=E\left[\sum_{i=1}^p\sum_{j=1}^p a_i a_j(X_i-\mu_i)(X_j-\mu_j)\right]\\ &=\sum_{i=1}^p\sum_{j=1}^p a_i a_j E[(X_i-\mu_i)(X_j-\mu_j)]\\ &=\sum_{i=1}^p\sum_{j=1}^p a_i a_j\operatorname{Cov}(X_i,X_j)\\ &=a^T\Sigma a. \end{aligned}

Each expectation in the expansion is finite: finite second moments imply the required cross-products are integrable by the Cauchy–Schwarz inequality. Finally, a variance is the expectation of a squared centered random variable, so it is nonnegative. Thus \(a^T\Sigma a\ge 0\) for every \(a\), which is precisely positive semidefiniteness. \(\square\)

The identity is more informative than a statement about matrix entries alone. Choosing \(a\) chooses a weighted sum of coordinates; the quadratic form gives exactly the variance of that sum. For instance, \(a=(1,1)^T\) asks about the variance of \(X_1+X_2\), while \(a=(1,-1)^T\) asks about the variance of \(X_1-X_2\).

Worked Example: A Covariance Matrix From Three Outcomes

Suppose \((X_1,X_2)\) takes the values \((1,2)\), \((3,0)\), and \((2,4)\), each with probability \(1/3\). Find its covariance matrix and check a quadratic form using the variance identity.

The coordinate means are \(E[X_1]=(1+3+2)/3=2\) and \(E[X_2]=(2+0+4)/3=2\). The centered pairs are \((-1,0)\), \((1,-2)\), and \((0,2)\). Therefore,

\begin{aligned} \operatorname{Var}(X_1)&=\frac{(-1)^2+1^2+0^2}{3}=\frac{2}{3},\\ \operatorname{Var}(X_2)&=\frac{0^2+(-2)^2+2^2}{3}=\frac{8}{3},\\ \operatorname{Cov}(X_1,X_2)&=\frac{(-1)(0)+(1)(-2)+(0)(2)}{3}=-\frac{2}{3}. \end{aligned}

Thus \(\Sigma=\begin{pmatrix}2/3&-2/3\\-2/3&8/3\end{pmatrix}\). For \(a=(1,2)^T\), direct evaluation gives

a^T\Sigma a=\frac{2}{3}-\frac{8}{3}+\frac{32}{3}=\frac{26}{3}.

To check this as a variance, \(a^TX=X_1+2X_2\) takes values \(5,3,10\), with mean \(6\). Its variance is \(\bigl((5-6)^2+(3-6)^2+(10-6)^2\bigr)/3=(1+9+16)/3=26/3\). The matrix calculation and the variance calculation agree.

More Checks Through the Variance Interpretation

The theorem applies to every coefficient vector, not just a particular choice. In some examples it is possible to display the quadratic form as a sum of nonnegative terms, which makes the property especially transparent.

Worked Example: Independent Bernoulli Coordinates

Let \(X_1\sim\operatorname{Bernoulli}(1/2)\) and \(X_2\sim\operatorname{Bernoulli}(1/4)\), independently. For a Bernoulli variable with success probability \(r\), the variance is \(r(1-r)\). Hence \(\operatorname{Var}(X_1)=1/4\) and \(\operatorname{Var}(X_2)=(1/4)(3/4)=3/16\). Independence gives \(\operatorname{Cov}(X_1,X_2)=0\), so

\Sigma= \begin{pmatrix} 1/4&0\\ 0&3/16 \end{pmatrix}.

For any \(a=(a_1,a_2)^T\), the quadratic form is \(a^T\Sigma a=(1/4)a_1^2+(3/16)a_2^2\), which is nonnegative. For example, with \(a=(2,-1)^T\), \(a^T\Sigma a=4(1/4)+3/16=19/16\). The weighted sum \(2X_1-X_2\) has variance \(4\operatorname{Var}(X_1)+\operatorname{Var}(X_2)-4\operatorname{Cov}(X_1,X_2)=1+3/16=19/16\), confirming the calculation.

Worked Example: A Three-Coordinate Covariance Matrix

Let \(U,V,W\) be independent random variables, each taking \(1\) and \(-1\) with probability \(1/2\). Define \(X_1=U\), \(X_2=U+V\), and \(X_3=V+W\). All three means are zero. Each of \(U,V,W\) has variance \(1\), and independence makes the cross-covariances between distinct variables zero. It follows that

\begin{aligned} \operatorname{Var}(X_1)&=1,& \operatorname{Var}(X_2)&=2,& \operatorname{Var}(X_3)&=2,\\ \operatorname{Cov}(X_1,X_2)&=E[U(U+V)]=1,& \operatorname{Cov}(X_1,X_3)&=E[U(V+W)]=0,& \operatorname{Cov}(X_2,X_3)&=E[(U+V)(V+W)]=1. \end{aligned}

For the last covariance, expanding the product gives \(E[UV]+E[UW]+E[V^2]+E[VW]=0+0+1+0=1\). Therefore the covariance matrix is \(\Sigma=\begin{pmatrix}1&1&0\\1&2&1\\0&1&2\end{pmatrix}\). For an arbitrary \(a=(a_1,a_2,a_3)^T\), the corresponding weighted sum is \(a^TX=(a_1+a_2)U+(a_2+a_3)V+a_3W\). Independence and zero means give

a^T\Sigma a=\operatorname{Var}(a^TX) =(a_1+a_2)^2+(a_2+a_3)^2+a_3^2\ge 0.

As a numerical check, choose \(a=(1,-2,1)^T\). The sum-of-squares expression gives \(1+1+1=3\). Direct matrix multiplication gives \(a^T\Sigma a=1-4+2+8-4=3\), so both calculations agree.

Principal Submatrices Inherit the Property

A principal submatrix is formed by selecting the same set of indices for rows and columns. For example, selecting coordinates 1 and 3 of a three-dimensional covariance matrix gives the covariance matrix of \((X_1,X_3)^T\). This selection preserves positive semidefiniteness.

Theorem: Every principal submatrix of a positive semidefinite matrix is positive semidefinite. In particular, every principal submatrix of a covariance matrix is positive semidefinite.

Proof. Let \(I\) be a set of selected indices and let \(\Sigma_{II}\) be the corresponding principal submatrix. Take any coefficient vector \(b\) with one coordinate for each index in \(I\). Construct a full-length vector \(a\) by putting the entries of \(b\) in the positions indexed by \(I\) and putting zero in every other position. Expanding the quadratic form, all terms involving an index outside \(I\) vanish, so \(b^T\Sigma_{II}b=a^T\Sigma a\). Since \(\Sigma\) is positive semidefinite, \(a^T\Sigma a\ge 0\). Thus \(b^T\Sigma_{II}b\ge 0\) for every \(b\), as required. \(\square\)

This result lets us examine selected coordinates without losing the positivity requirement. It does not say that checking a few selected entries is enough to establish the property for the whole matrix; the definition still requires a nonnegative quadratic form for every coefficient vector.

Common Mistake: Checking Only the Diagonal

The diagonal entries of a covariance matrix are variances and therefore cannot be negative. But that fact alone does not guarantee positive semidefiniteness: the off-diagonal entries also affect quadratic forms. For example, consider \(B=\begin{pmatrix}1&2\\2&1\end{pmatrix}\). Its diagonal entries are both positive, yet for \(a=(1,-1)^T\),

$$ a^TBa=1-2-2+1=-2<0. $$

So \(B\) is not positive semidefinite and cannot be a covariance matrix. The error is to treat nonnegative variances of individual coordinates as if they controlled the variance of every weighted sum. The correction is to check the full quadratic form, or to establish that it is a variance as in the theorem.

Key takeaway: A covariance matrix is positive semidefinite because every quadratic form \(a^T\Sigma a\) is the variance of the weighted sum \(a^TX\). Its diagonal variances are only part of this requirement; the covariances must also combine consistently.

Check Your Understanding

Use the variance interpretation to answer each question.

State the moment condition on the coordinates of \(X\) used to define a finite covariance matrix.
For a vector \(a\), what random variable has variance \(a^T\Sigma a\)?
Let \(X_1\) and \(X_2\) have variances \(2\) and \(5\), with covariance \(-1\). Calculate the quadratic form for \(a=(1,1)^T\), and identify the corresponding weighted sum.
Why does selecting the same coordinate indices for the rows and columns of a positive semidefinite matrix produce a positive semidefinite principal submatrix?
Give a reason that nonnegative diagonal entries alone are not enough to establish positive semidefiniteness.