Valid Covariance Matrices for Latent Effects

Why Covariance Validity Matters

In a GLMM, a vector of latent effects can represent dependence among observations that share a group, location, or time structure. For example, a conditional linear predictor may contain \(Zb\), where \(b\) is a random vector and \(Z\) maps its components to observations. The covariance matrix assigned to \(b\) determines how its components vary together. If that matrix is mathematically impossible, it cannot describe any probability distribution for the latent effects, regardless of the response distribution or link function.

This tutorial concerns a basic but essential question: when can a matrix be a covariance matrix? The answer rests on symmetry and positive semidefiniteness. The second condition is more than a technical constraint: every linear combination of random effects must have a nonnegative variance.

Definition: Let \(b=(b_1,\ldots,b_q)^\mathsf{T}\) be a random vector with finite second moments and mean \(\mu_b\). Its covariance matrix is \(\Sigma=\operatorname{Cov}(b)=E[(b-\mu_b)(b-\mu_b)^\mathsf{T}]\). A real matrix \(\Sigma\) is positive semidefinite if it is symmetric and \(a^\mathsf{T}\Sigma a\geq 0\) for every real vector \(a\) of matching dimension. It is positive definite if it is symmetric and \(a^\mathsf{T}\Sigma a>0\) for every nonzero \(a\).

For a GLMM latent-effect vector, the covariance entries have direct meanings: diagonal entries are variances, and off-diagonal entries are covariances. If effects have units such as log-odds, each variance has squared log-odds units, and a covariance has the product of the two effects' units. Validity is required whether or not \(b\) is Gaussian; Gaussianity is not needed for the covariance argument.

Deriving the Required Conditions

The covariance matrix is symmetric because, for every pair \(j,k\), \(\operatorname{Cov}(b_j,b_k)=\operatorname{Cov}(b_k,b_j)\). Its diagonal entries are nonnegative because each is a variance. But nonnegative diagonal entries alone are not enough: the entries must also combine to give nonnegative variance for every linear combination.

\operatorname{Var}(a^\mathsf{T}b) =E\!\left[\{a^\mathsf{T}(b-\mu_b)\}^2\right] =a^\mathsf{T}E[(b-\mu_b)(b-\mu_b)^\mathsf{T}]a =a^\mathsf{T}\Sigma a\geq 0.

This identity proves that a covariance matrix must be positive semidefinite. It also gives an immediate test: if even one vector \(a\) produces \(a^\mathsf{T}\Sigma a<0\), that candidate cannot be a covariance matrix. A zero quadratic form is permitted. It means that the corresponding linear combination has variance zero, so it equals its mean almost surely.

For symmetric matrices, eigenvalues provide a complete test. The result follows from the spectral decomposition, which writes a real symmetric matrix as \(Q\Lambda Q^\mathsf{T}\), where \(Q\) is orthogonal and \(\Lambda\) is diagonal with the eigenvalues on its diagonal.

Theorem: A real symmetric matrix \(\Sigma\) is positive semidefinite if and only if every eigenvalue of \(\Sigma\) is nonnegative. It is positive definite if and only if every eigenvalue is strictly positive.

To see both directions, write any vector \(a\) as \(a=Qc\). Then

a^\mathsf{T}\Sigma a =c^\mathsf{T}\Lambda c =\sum_{j=1}^{q}\lambda_j c_j^2.

If every \(\lambda_j\geq 0\), this sum is nonnegative for every \(c\), hence for every \(a\). Conversely, if some eigenvalue \(\lambda_k<0\), choose \(a\) to be a corresponding unit eigenvector. The quadratic form is then \(a^\mathsf{T}\Sigma a=\lambda_k<0\), contradicting the requirement for a covariance matrix. Strict positivity for every nonzero \(a\) holds exactly when all eigenvalues are strictly positive.

Worked Matrix Checks

Worked Example: A Candidate That Fails Symmetry

Suppose two centered latent effects, \(b_1\) and \(b_2\), are proposed to have the covariance matrix

\Sigma= \begin{pmatrix} 1 & 0.30\\ 0.20 & 1 \end{pmatrix}.

The diagonal entries propose unit variances. But the upper-right entry says \(\operatorname{Cov}(b_1,b_2)=0.30\), whereas the lower-left entry says \(\operatorname{Cov}(b_2,b_1)=0.20\). Since covariance is symmetric, these two values must be equal. The matrix therefore fails before any eigenvalue or quadratic-form calculation: it cannot be a covariance matrix.

If the intended covariance is \(0.30\), both off-diagonal entries must be \(0.30\). If the intended value is \(0.20\), both must be \(0.20\). Symmetry is not an optional modeling convention or a rounding preference; it follows from the definition of covariance.

Worked Example: A Symmetric Matrix With a Negative Variance

Consider the symmetric candidate

\Sigma= \begin{pmatrix} 1 & 1.2\\ 1.2 & 1 \end{pmatrix}.

Both proposed variances are \(1\), but the proposed covariance is \(1.2\). Test the contrast \(a=(1,-1)^\mathsf{T}\), which corresponds to \(b_1-b_2\):

a^\mathsf{T}\Sigma a = \begin{pmatrix}1&-1\end{pmatrix} \begin{pmatrix}1&1.2\\1.2&1\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix} =1-1.2-1.2+1=-0.40.

The candidate assigns a negative variance, so it is invalid. The eigenvalues give the same conclusion. The vectors \((1,1)^\mathsf{T}\) and \((1,-1)^\mathsf{T}\) are eigenvectors, with eigenvalues \(2.2\) and \(-0.20\), respectively. Since an eigenvalue is negative, the matrix is not positive semidefinite.

For a two-effect covariance matrix with positive variances, the covariance must also satisfy \(|\operatorname{Cov}(b_1,b_2)|\leq \sqrt{\operatorname{Var}(b_1)\operatorname{Var}(b_2)}\). Here the upper bound is \(1\), yet the proposed covariance is \(1.2\). That quick check flags the problem; the quadratic form shows exactly which contrast has the impossible variance.

Worked Example: A Valid but Singular Covariance Matrix

Suppose three centered latent effects have the proposed covariance matrix

\Sigma= \begin{pmatrix} 1 & 1 & 0\\ 1 & 1 & 0\\ 0 & 0 & 0.25 \end{pmatrix}.

This matrix is symmetric. Its eigenvalues are \(2\), \(0\), and \(0.25\). For example, \((1,1,0)^\mathsf{T}\) has eigenvalue \(2\), \((1,-1,0)^\mathsf{T}\) has eigenvalue \(0\), and \((0,0,1)^\mathsf{T}\) has eigenvalue \(0.25\). All are nonnegative, so the matrix is positive semidefinite and is a valid covariance matrix.

It is not positive definite because one eigenvalue is zero. In particular,

\operatorname{Var}(b_1-b_2) = \begin{pmatrix}1&-1&0\end{pmatrix} \Sigma \begin{pmatrix}1\\-1\\0\end{pmatrix} =0.

Thus \(b_1=b_2\) almost surely when the effects are centered, even though each has variance \(1\). This is a perfectly valid, but degenerate, joint distribution. Some fitting routines require a positive definite matrix because they use its inverse or a nonsingular density representation. Their computational restriction does not make a singular positive semidefinite covariance mathematically invalid.

Practical Assessment and Diagnostics

For a proposed covariance matrix, first confirm that its entries are real, finite, and arranged symmetrically. Then assess positive semidefiniteness. For a small matrix, a carefully chosen quadratic form can demonstrate invalidity, but one successful nonnegative quadratic form does not prove validity: the condition must hold for every vector. Computing all eigenvalues is a systematic test for a symmetric matrix.

Another exact criterion is that every principal minor of a symmetric matrix be nonnegative. In practice, eigenvalues are often easier to inspect for a modest-sized matrix. Checking only the leading principal minors is not sufficient for positive semidefiniteness in general; that shortcut can miss a problematic submatrix. For positive definiteness, Sylvester's criterion does use strictly positive leading principal minors, but it is a different condition.

Numerical calculations require care. A computed eigenvalue of \(-10^{-12}\), for instance, may be a rounding artifact when the entries are on a much larger scale; it should be judged relative to numerical precision and matrix scale, not automatically treated as meaningful. Conversely, a materially negative eigenvalue is evidence of invalidity. Symmetrizing a matrix by averaging its two triangles may address tiny numerical discrepancies, but it does not repair a genuinely invalid covariance specification without changing the model.

Negative diagonal entry: immediately invalid, because a diagonal entry is a variance.
All diagonal entries nonnegative: necessary, but not sufficient; the second worked example has positive diagonal entries and is invalid.
Failed Cholesky factorization: may indicate indefiniteness, but can also occur for a valid singular positive semidefinite matrix. Do not treat failure alone as proof of invalidity.
Pairwise checks pass: still inspect the full matrix in dimensions greater than two. Individually plausible pairs do not guarantee a jointly valid covariance matrix.

Parameterizations, Alternatives, and Limits

One approach is to specify covariance entries directly and check the resulting matrix. This is transparent, but an unconstrained optimizer can propose values that violate positive semidefiniteness. Another approach is to build validity into the parameterization. For example, a matrix of the form \(\Sigma=LL^\mathsf{T}\) is positive semidefinite for every real matrix \(L\), since \(a^\mathsf{T}LL^\mathsf{T}a=(L^\mathsf{T}a)^\mathsf{T}(L^\mathsf{T}a)\geq 0\). If \(L\) is nonsingular and square, the result is positive definite. Such parameterizations help maintain validity, although they do not by themselves determine whether the structure is scientifically appropriate.

A diagonal covariance structure is another option when independent latent components are substantively justified. It is automatically positive semidefinite if its diagonal variances are nonnegative, but it rules out covariance between components. That simplification is not interchangeable with a general covariance structure. A precision-matrix representation can be convenient when the inverse covariance exists, but it requires a positive definite covariance; a singular covariance has no ordinary inverse.

Positive semidefiniteness is a mathematical validity condition, not a guarantee of a good GLMM. A valid matrix may still impose an unsuitable dependence pattern, be poorly identified from the available data, or create numerical difficulties. Conversely, a nearly singular matrix can be valid but signal that some linear combination of effects has very little variation. The condition alone does not select a spatial or temporal structure; it only rules out impossible covariance specifications.

Common Mistake

A frequent error is to conclude that a matrix is valid because its diagonal entries are positive, or because every pairwise covariance seems plausible. The reasoning overlooks variances of combinations such as \(b_1-b_2\), which depend on the off-diagonal entries as well. The correction is to verify symmetry and then establish that the full matrix is positive semidefinite, using the eigenvalue criterion or an equivalent complete test. A second error is to reject every singular matrix: zero eigenvalues are allowed for covariance matrices, although particular algorithms may require strict positive definiteness.

Key takeaway: A latent-effect covariance matrix must be symmetric and have no negative eigenvalues. Equivalently, every linear combination of latent effects must have a nonnegative variance. Zero eigenvalues indicate degeneracy, not automatic invalidity.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Why must the covariance matrix of a random vector be symmetric?
A symmetric matrix has eigenvalues \(3\), \(0.5\), and \(-0.01\). Is it a valid covariance matrix? Explain.
A symmetric matrix has eigenvalues \(4\), \(1\), and \(0\). Is it positive definite, positive semidefinite, both, or neither?
For \(\Sigma=\begin{pmatrix}4&1\\1&1\end{pmatrix}\), calculate the variance of \(b_1-b_2\) if this is the covariance matrix of \(b\).

Solutions. (1) By definition, \(\operatorname{Cov}(b_j,b_k)=E[(b_j-\mu_j)(b_k-\mu_k)]\), and multiplication is commutative, so this equals \(\operatorname{Cov}(b_k,b_j)\). (2) No. The negative eigenvalue means a corresponding eigenvector \(a\) gives \(a^\mathsf{T}\Sigma a=-0.01\|a\|^2<0\), an impossible variance. (3) It is positive semidefinite, since all eigenvalues are nonnegative, but not positive definite, since one is zero. (4) Use \(a=(1,-1)^\mathsf{T}\): \(\operatorname{Var}(b_1-b_2)=a^\mathsf{T}\Sigma a=4-1-1+1=3\) squared units of the effect. This nonnegative value checks that contrast; the full matrix's eigenvalues or another complete criterion are needed to establish validity in general.