Rank Deficiency and Conditioning of the Sample Covariance

Why Rank and Conditioning Matter

A covariance matrix is often inverted when constructing a multivariate procedure, but a sample covariance may fail to have an inverse even when the population covariance does. The issue is partly geometric: after centering, the observations occupy a space of dimension at most \(n-1\). When the number of variables \(p\) is large relative to \(n\), this constraint forces the sample covariance to be singular. Even when it is invertible, a very small eigenvalue can make its inverse extremely sensitive to small changes in the data.

This tutorial derives the rank restriction, establishes the exact rank under a nonsingular Gaussian model, and examines what small eigenvalues imply for inversion. The distinction is important: having an inverse is not the same as having a stable inverse.

Centered Sample Covariance and Its Rank

Let \(X_1,\ldots,X_n\) be observations in \(\mathbb{R}^p\), and write \(\bar X=n^{-1}\sum_{i=1}^n X_i\). For \(n\geq 2\), the usual centered sample covariance matrix is

Let \(Y\) be the \(n\times p\) matrix whose \(i\)th row is \(X_i^\mathsf{T}\), let \(\mathbf{1}\) denote the \(n\)-vector of ones, and define the centering matrix \(H=I_n-\mathbf{1}\mathbf{1}^\mathsf{T}/n\). Then \(HY\) has rows \((X_i-\bar X)^\mathsf{T}\), and \(S=(HY)^\mathsf{T}(HY)/(n-1)\). The matrix \(H\) is a projection onto the subspace orthogonal to \(\mathbf{1}\), so its rank is \(n-1\).

The first part is a deterministic bound. The second part says that, under the specified Gaussian model, this bound is attained except on an event of probability zero. The Gaussian assumption and positive definiteness of \(\Sigma\) are relevant to this exact-rank claim.

Proof of the Rank Results

For any matrix \(A\), the matrices \(A^\mathsf{T}A\) and \(A\) have the same null space: \(A^\mathsf{T}Av=0\) implies \(0=v^\mathsf{T}A^\mathsf{T}Av=\|Av\|^2\), hence \(Av=0\), and the converse is immediate. Thus \(\operatorname{rank}(A^\mathsf{T}A)=\operatorname{rank}(A)\). Apply this with \(A=HY\). Since \(H\) has rank \(n-1\), \(\operatorname{rank}(HY)\leq n-1\); also \(HY\) has \(p\) columns, so its rank is at most \(p\). Multiplication by the positive scalar \(1/(n-1)\) does not change rank. This proves the deterministic bound.

For the Gaussian claim, write \(Y=\mathbf{1}\mu^\mathsf{T}+Z\Sigma^{1/2}\), where the rows of \(Z\) are independent \(N_p(0,I_p)\), and \(\Sigma^{1/2}\) is an invertible square root. Since \(H\mathbf{1}=0\), \(HY=HZ\Sigma^{1/2}\), and multiplication on the right by \(\Sigma^{1/2}\) preserves rank. Choose an \(n\times(n-1)\) matrix \(U\) whose columns form an orthonormal basis for the range of \(H\). Then \(H=UU^\mathsf{T}\), and \(\operatorname{rank}(HZ)=\operatorname{rank}(U^\mathsf{T}Z)\). The entries of \(U^\mathsf{T}Z\) are independent standard normal random variables: orthogonal transformations of a standard multivariate normal vector preserve its distribution, and the original columns of \(Z\) are independent.

An \((n-1)\times p\) matrix of independent standard normal entries has rank \(\min(n-1,p)\) with probability one. To see why, set \(r=\min(n-1,p)\) and consider the determinant of its upper-left \(r\times r\) block. This determinant is a polynomial in the matrix entries and is not identically zero, since it equals 1 when that block is the identity. The zero set of a nonzero polynomial has Lebesgue measure zero, while the Gaussian matrix has a density. Thus this minor is nonzero with probability one, giving rank at least \(r\); the dimensions give rank at most \(r\). Combining these facts proves the theorem. \(\square\)

Invertibility Is Not the Same as Stability

For a positive definite matrix \(A\), let \(\lambda_{\min}(A)\) and \(\lambda_{\max}(A)\) be its smallest and largest eigenvalues. Its spectral condition number is \(\kappa_2(A)=\lambda_{\max}(A)/\lambda_{\min}(A)\). A large condition number means that some directions are much less variable than others. Inverting \(A\) replaces each eigenvalue by its reciprocal, so directions associated with small eigenvalues receive large weights.

The next bound makes this sensitivity precise. For a matrix \(B\), \(\|B\|_2\) denotes its spectral norm, the largest amount by which it can stretch a unit vector.

For any unit vector \(v\), the Rayleigh quotient gives \(v^\mathsf{T}(A+E)v\geq\lambda_{\min}(A)-\delta>0\). Thus \(A+E\) is positive definite, with smallest eigenvalue at least \(\lambda_{\min}(A)-\delta\). The inverse identity \((A+E)^{-1}-A^{-1}=-A^{-1}E(A+E)^{-1}\) follows by multiplying out the right-hand side. Taking spectral norms and using \(\|BC\|_2\leq\|B\|_2\|C\|_2\) yields

Here the inverse norms equal the reciprocals of the smallest eigenvalues because both matrices are positive definite. This proves the bound. \(\square\)

The restriction \(\delta<\lambda_{\min}(A)\) is substantive: it ensures that the perturbation cannot remove positive definiteness. The bound also shows why an absolute error that looks small can still be consequential when the smallest eigenvalue is comparably small.

Common Pitfall and Practical Implications

A common mistake is to conclude that a positive definite population covariance guarantees an invertible sample covariance. That reasoning confuses a population property with a restriction imposed by the observed sample dimensions. If \(p>n-1\), the centered sample covariance is necessarily singular, even when the population covariance is positive definite. If \(p\leq n-1\), the Gaussian rank result gives invertibility with probability one, but it does not guarantee a well-conditioned matrix.

A second mistake is to treat numerical invertibility as evidence of stability. A matrix may have a nonzero smallest eigenvalue and still have such a large condition number that small data changes produce large changes in the inverse. In analyses that require an inverse, the rank and the spectrum should therefore be checked separately. When the sample covariance is singular or poorly conditioned, the direct inverse is not a dependable substitute for a method designed for that setting; choosing such a method requires additional assumptions and is a separate estimation question.

The geometric rank bound is useful before any computation: it identifies when direct inversion is impossible from the dimensions alone. When inversion is possible, the smallest eigenvalue and condition number give complementary information about numerical and statistical sensitivity. Neither check by itself establishes that a covariance estimate is accurate, but together they reveal two basic obstacles to using its inverse.