Positive Semidefinite Matrices

Why Positive Semidefiniteness Matters

Many quantities in multivariate statistics are quadratic: the variance of a linear combination, the squared length of a transformed vector, or the curvature of a quadratic function. A matrix is positive semidefinite when its quadratic form never takes a negative value. The condition allows zero, which matters: a matrix can be positive semidefinite and still have a nontrivial null space.

This tutorial gives four equivalent ways to identify such a matrix. The quadratic-form definition states the property directly; eigenvalues give a spectral test; principal minors give a determinant-based test; and a Gram representation expresses the matrix as inner products of vectors.

Symmetry is part of the definition. For any real vector \(x\), the value \(x^{T}Ax\) is unchanged if \(A\) is replaced by its symmetric part \((A+A^{T})/2\), but the matrix itself may not be symmetric. The characterizations below concern real symmetric matrices.

Worked Examples: Reading the Quadratic Form

Four Equivalent Tests

For a real symmetric matrix, these descriptions are not separate assumptions: each is equivalent to positive semidefiniteness. In the statement, “all principal minors” includes minors of every size, not only those in the upper-left corner.

Proof. First, use the spectral theorem for real symmetric matrices to write \(A=Q\Lambda Q^{T}\), where \(Q\) is orthogonal and \(\Lambda\) is diagonal with the eigenvalues \(\lambda_1,\ldots,\lambda_n\) on its diagonal. If all \(\lambda_i\geq0\), then for any \(x\), setting \(y=Q^{T}x\) gives \(x^{T}Ax=y^{T}\Lambda y=\sum_{i=1}^{n}\lambda_i y_i^{2}\geq0\). Conversely, if \(x^{T}Ax\geq0\) for every \(x\), take a unit eigenvector \(q_i\) for \(\lambda_i\). Then \(\lambda_i=q_i^{T}Aq_i\geq0\). This proves the equivalence of (1) and (2).

Suppose (1) holds. For any index set \(S\), a vector \(z\) on the coordinates in \(S\) can be extended to a vector \(x\in\mathbb{R}^{n}\) by putting zero in all other coordinates. Then \(z^{T}A[S]z=x^{T}Ax\geq0\). Thus every principal submatrix is positive semidefinite. Its eigenvalues are nonnegative by the equivalence just proved, so its determinant, the product of those eigenvalues, is nonnegative. This proves (1) implies (3).

Now suppose every principal minor of \(A\) is nonnegative. For \(t>0\), expansion of the determinant gives \(\det(A+tI)=\sum_{S\subseteq\{1,\ldots,n\}}\det(A[S])t^{n-|S|}\), where the empty-set term is \(t^{n}\). Every term is nonnegative and the empty-set term is strictly positive, so \(\det(A+tI)>0\). If \(A\) had a negative eigenvalue \(\lambda\), choosing \(t=-\lambda>0\) would make \(A+tI\) singular, contradicting this strict inequality. Therefore all eigenvalues are nonnegative, proving (3) implies (2), and hence (3) implies (1).

If (2) holds, define \(B=\Lambda^{1/2}Q^{T}\), where \(\Lambda^{1/2}\) is diagonal with entries \(\sqrt{\lambda_i}\). These square roots are real because \(\lambda_i\geq0\). Direct multiplication yields \(B^{T}B=Q\Lambda^{1/2}\Lambda^{1/2}Q^{T}=Q\Lambda Q^{T}=A\). Thus (2) implies (4). Finally, if (4) holds, then \(x^{T}Ax=x^{T}B^{T}Bx=(Bx)^{T}(Bx)=\sum_j(Bx)_j^{2}\geq0\) for every \(x\). This proves (4) implies (1) and completes the equivalences. \(\square\)

Principal Minors: A Careful Test

The theorem makes the principal-minor criterion exact, but completeness matters. For an \(n\times n\) matrix, a leading principal minor uses the first \(k\) rows and columns for some \(k\). Testing only these nested submatrices can miss a negative principal minor elsewhere. The next example shows why the distinction matters when nonnegative, rather than strictly positive, values are allowed.

Gram Representations and Statistical Meaning

The representation \(A=B^{T}B\) says that the entries of \(A\) are inner products of the columns of \(B\). If those columns are \(b_1,\ldots,b_n\), then \(A_{ij}=b_i^{T}b_j\). The quadratic form becomes the squared length of a linear combination: \(x^{T}Ax=\|Bx\|^{2}\). This explains directly why it cannot be negative. Conversely, the theorem shows that every real symmetric positive semidefinite matrix can be built from such inner products, even when some of the vectors are linearly dependent.

This structure appears naturally in multivariate statistics. For example, if \(X=(X_1,\ldots,X_n)^{T}\) is a random vector with finite second moments, its covariance matrix has entries \(\operatorname{Cov}(X_i,X_j)\). For any fixed vector \(a\), \(a^{T}\operatorname{Cov}(X)a=\operatorname{Var}(a^{T}X)\geq0\). So a covariance matrix is positive semidefinite. If some nonzero linear combination has zero variance, the matrix can be singular; that does not contradict semidefiniteness.

Common Mistake

• Wrong reasoning: “All the leading principal minors are nonnegative, so the matrix must be positive semidefinite.” The test in the theorem requires every principal minor, and the matrix \(C\) above passes the leading-minor check while failing the full test.
• Correction: For a symmetric matrix, establish nonnegativity of the quadratic form, verify that all eigenvalues are nonnegative, check every principal minor, or exhibit a Gram representation. Do not replace “every principal minor” with “every leading principal minor.”
• Another common error: Treating a zero eigenvalue as evidence against semidefiniteness. Zero is allowed: it indicates a direction in which the quadratic form can vanish.