Singular Covariance Matrices and Exact Linear Relations

Why Singularity Signals an Exact Constraint

A covariance matrix summarizes the variances of individual components and their pairwise covariances. When it is singular, some direction in the space of components has no variation at all. This is stronger than saying that two variables are highly correlated: it means a nonzero linear combination is constant with probability one. The constant need not be zero, so the resulting constraint is generally an affine relation.

For example, if one measurement is always three times another minus four, the two measurements vary, but their values are confined to a line. The covariance matrix cannot distinguish movement along that line from movement in the full two-dimensional plane, because there is no variation perpendicular to the line. The precise equivalence below makes this geometric observation useful in calculations.

The finite-second-moment condition ensures that every covariance entry and every variance of a linear combination used below is finite. For a fixed vector \(\boldsymbol c=(c_1,\ldots,c_p)^\mathsf{T}\), the scalar \(\boldsymbol c^\mathsf{T}\boldsymbol X=\sum_{j=1}^p c_jX_j\) has variance

This is the quadratic-form version of the covariance rule for linear transformations. It explains why singularity is connected to zero variance: for a singular covariance matrix, some nonzero direction is annihilated by the matrix, and the corresponding quadratic form is zero.

The Singularity Equivalence

Proof. A square matrix is singular exactly when its null space contains a nonzero vector. Thus, if \(\Sigma\) is singular, there is a nonzero \(\boldsymbol c\) such that \(\Sigma\boldsymbol c=\boldsymbol 0\). Using symmetry of a covariance matrix and the quadratic-form identity,

Conversely, if a nonzero \(\boldsymbol c\) satisfies \(\operatorname{Var}(\boldsymbol c^\mathsf{T}\boldsymbol X)=0\), then \(\boldsymbol c^\mathsf{T}\Sigma\boldsymbol c=0\). Since \(\Sigma\) is a covariance matrix, it is positive semidefinite. By the spectral theorem, write \(\Sigma=Q\Lambda Q^\mathsf{T}\), where \(Q\) is orthogonal and \(\Lambda\) is diagonal with nonnegative entries. Then

Every term in this sum is nonnegative. Therefore, for each \(j\) with \(\lambda_j>0\), the corresponding coordinate \((Q^\mathsf{T}\boldsymbol c)_j\) is zero. It follows that \(\Lambda Q^\mathsf{T}\boldsymbol c=\boldsymbol 0\), hence \(\Sigma\boldsymbol c=\boldsymbol 0\). Since \(\boldsymbol c\ne\boldsymbol 0\), \(\Sigma\) is singular.

It remains to connect zero variance to an almost-sure relation. Set \(Y=\boldsymbol c^\mathsf{T}\boldsymbol X\) and \(m=E[Y]=\boldsymbol c^\mathsf{T}\boldsymbol\mu\). If \(\operatorname{Var}(Y)=0\), then \(E[(Y-m)^2]=0\). For every positive integer \(k\), nonnegativity gives

Thus \(P(|Y-m|\ge 1/k)=0\) for every \(k\). The event \(\{Y\ne m\}\) is the countable union of the events \(\{|Y-m|\ge 1/k\}\), so it also has probability zero. Therefore \(Y=m\) almost surely. In the other direction, if \(Y=d\) almost surely, then \(E[Y]=d\) and \(\operatorname{Var}(Y)=E[(Y-d)^2]=0\). This proves the equivalence and shows that the constant is \(d=\boldsymbol c^\mathsf{T}\boldsymbol\mu\). \(\square\)

The almost-sure qualification matters: an exact relation need not hold at every point in the underlying sample space. It holds except on an event of probability zero, which is enough to give the linear combination zero variance. In vector terms, the random vector lies almost surely in the affine hyperplane \(\{\boldsymbol x:\boldsymbol c^\mathsf{T}\boldsymbol x=d\}\). A singular covariance matrix therefore indicates that the distribution is confined to a lower-dimensional affine set, though that set may have dimension lower than \(p-1\) if there are several independent exact relations.

Worked Examples

Interpreting the Relation and Avoiding a Common Mistake

If \(\boldsymbol c^\mathsf{T}\boldsymbol X=d\) almost surely, centering the components gives the equivalent relation \(\boldsymbol c^\mathsf{T}(\boldsymbol X-\boldsymbol\mu)=0\) almost surely. The covariance matrix records variation around the mean, so it is the centered relation that corresponds directly to a null direction. The uncentered form is often easier to interpret because it states the constraint in the original units.

A common mistake is to treat a very small variance as if it were zero. If \(\operatorname{Var}(\boldsymbol c^\mathsf{T}\boldsymbol X)\) is small but positive, the combination still varies with positive probability; the theorem does not give an exact relation. The covariance matrix may be close to singular, but it is not singular. In applications, near-singularity can make calculations sensitive, but it should not be reported as a mathematical identity without exact justification.

A related mistake is to conclude that a singular covariance matrix means two particular components must be equal or one must be a multiple of the other. The null vector may involve several components, and the relation may include a nonzero constant. For instance, \(X_3=X_1+X_2\) is a relation among three components, not a pairwise equality. To identify the constraint, find a nonzero vector in the null space of \(\Sigma\), then interpret its coefficients as a linear combination.