Covariance Failures Through Eigenstructure

Why Look Beyond a Positive-Definite Warning?

A random-effects covariance matrix describes both the variability of individual effects and their covariation. When estimation reports that this matrix is not positive definite, the warning identifies a mathematical problem but does not, by itself, reveal its structure. One possibility is a single weak combination of random effects: most directions have substantial variability, while one particular contrast is nearly redundant. Another is broader numerical instability, in which several directions are poorly determined or the reported spectrum changes materially with numerical settings.

Eigenstructure makes that distinction visible. The eigenvalues quantify variability along orthogonal directions, and their eigenvectors identify the corresponding combinations of random effects. This is a diagnostic, not a substitute for checking the fitted model, units, and estimation behavior. In particular, a small eigenvalue does not by itself establish why the direction is weak.

For example, if the random effects are \(\eta_1\) and \(\eta_2\), the vector \(v=(1,-1)^\mathsf{T}/\sqrt{2}\) represents the normalized contrast \((\eta_1-\eta_2)/\sqrt{2}\). Its variance is \(v^\mathsf{T}Gv\). If this value is small, the fitted covariance structure assigns little variation to that contrast, even if the marginal variances of \(\eta_1\) and \(\eta_2\) are not small.

What the Spectrum Shows

For a real symmetric matrix, the spectral decomposition is \(G=V\Lambda V^\mathsf{T}\), where the columns of \(V\) are orthonormal eigenvectors and \(\Lambda\) is diagonal with the eigenvalues. If a random-effects vector \(\eta\) has mean zero and covariance \(G\), then the variance of its projection onto any unit direction \(u\) is \(u^\mathsf{T}Gu\). Expanding that direction in the eigenvector basis gives a precise reason the eigenvalues are useful.

The last equality follows because \(Gv_k=\lambda_kv_k\) and \(v_j^\mathsf{T}v_k=0\) for \(j\ne k\). Thus the variance in any unit direction is a weighted average of the eigenvalues, with nonnegative weights \(a_j^2\) summing to one. The minimum possible variance is the smallest eigenvalue, attained in its eigenvector direction. A negative eigenvalue therefore implies a direction with negative quadratic form, which is incompatible with a covariance matrix.

When all eigenvalues are positive, the spectral condition number in the Euclidean norm is \(\kappa_2(G)=\lambda_{\max}/\lambda_{\min}\). A large value indicates that the matrix has directions with very different variances and can be sensitive to perturbations. It is not a universal pass/fail criterion: the relevant scale depends on parameter units, numerical precision, and the estimation procedure.

Worked Example: One Weak Contrast

Use Relative Scale, Not Just Raw Eigenvalues

Eigenvalues have units that depend on the random-effects coordinates. If one effect is expressed in a different unit, its numerical variance changes, and so can the eigenvalues of the covariance matrix. Consequently, comparing raw eigenvalues across differently scaled parameters can be misleading. A large raw condition number may reflect a scale disparity rather than a weak correlation structure.

One diagnostic is to standardize by marginal standard deviations. Let \(D\) be diagonal with entries \(\sqrt{G_{ii}}\), assuming these are positive. The corresponding correlation matrix is \(R=D^{-1}GD^{-1}\). Its spectrum describes dependence after each coordinate has been scaled to unit marginal variance. This transformation is useful for diagnosing correlation structure, but it does not replace examination of \(G\): a very small marginal variance may itself be substantively or numerically important.

One Weak Direction or Several?

A useful first comparison is between the smallest eigenvalues relative to the largest eigenvalue, preferably after considering an appropriate parameter scaling. If one eigenvalue is much smaller than the others, its eigenvector isolates a candidate weak direction. If several are small, more than one independent direction is weak, which is more consistent with broad loss of numerical support in the covariance structure. “Consistent with” is important: a spectrum diagnoses geometry, not the cause of that geometry.

Numerical Tolerance and Failure Modes

Computed eigenvalues are subject to finite-precision error. A value near zero must therefore be interpreted relative to matrix scale and the accuracy of the calculation. A common diagnostic convention is to compare each eigenvalue with a relative threshold such as \(\tau\lambda_{\max}\), where \(\tau\) is chosen for the numerical method and precision in use. This is a tolerance for diagnosis, not a universal statistical cutoff. A slightly negative eigenvalue within a defensible numerical tolerance may be indistinguishable from zero in that computation; it is not evidence that the exact fitted matrix is positive definite.

For a symmetric matrix, a materially negative eigenvalue indicates non-positive-definiteness. Before attributing it to rounding, check that the matrix is symmetric as intended, parameters use the expected units, and the reported values have not been rounded so heavily that the reconstructed matrix differs from the one used in estimation. If a negative value is close to the tolerance boundary, recompute from full-precision values and assess whether reasonable numerical changes alter the sign or magnitude.

Eigenvectors also require care. The signs of an eigenvector and its negative represent the same direction, so sign flips across runs are not meaningful by themselves. When eigenvalues are repeated or nearly repeated, individual eigenvectors within that eigenspace may rotate substantially even when the subspace is stable. In that case, interpret the weak subspace as a whole rather than claiming one uniquely identified contrast.

How to Use the Diagnosis

Eigenstructure complements, rather than replaces, the earlier material on non-positive-definite random-effects covariance. A direct positive-definiteness check answers whether a matrix meets the mathematical condition. Eigenanalysis adds information about how many directions are weak and which combinations they involve. The covariance matrix's marginal variances and correlations remain useful summaries, but neither reveals every weak contrast as directly as the eigenvectors do.

Common failure modes include treating a small eigenvalue as proof that a random effect is unnecessary, interpreting a raw condition number without checking units, and calling any small negative computed eigenvalue a substantive model defect. The correction is to use the eigenvalue and eigenvector together, assess scale and numerical tolerance, and describe the result narrowly: for example, “one contrast has much less estimated variance than the other direction,” not “the model is invalid.”

This diagnostic has limits. A covariance spectrum does not establish whether the data support a parameter combination, identify the cause of weak estimation, or determine which model change is appropriate. It also does not guarantee that a positive-definite matrix is numerically well behaved. Its value is to localize the geometry of the covariance problem and guide more targeted investigation.