Random-Effect Correlations Near Plus Or Minus One

Why a Random-Effect Correlation Near an Extreme Deserves Attention

An NLME model can estimate a random-effect correlation close to $+1$ or $-1$ even when the optimizer reports convergence. Such an estimate is not automatically an error: the data may support a strong relationship between individual deviations. But it can also signal that the fitted covariance matrix is nearly singular, so the model has little information about variation in one direction of the random-effects space. The estimate may then be sensitive to small changes in data or model specification.

The key is to examine more than the reported correlation. Check whether both random-effect variances are meaningfully above zero, inspect the covariance matrix's eigenvalues (and, when scales differ, its standardized correlation matrix), and distinguish the random-effect correlation from correlations among the estimated model parameters. These checks describe related but different features of a fit.

Definition: For two subject-level random effects $\eta_1$ and $\eta_2$, let their standard deviations be $\omega_1>0$ and $\omega_2>0$, and their correlation be $\rho$. Their covariance matrix is $$ \Omega = \begin{pmatrix} \omega_1^2 & \rho\omega_1\omega_2 \\ \rho\omega_1\omega_2 & \omega_2^2 \end{pmatrix}. $$ The random-effect correlation $\rho$ describes how the two random effects vary together across subjects. It is defined only when both standard deviations are positive.

The usual interpretation of this matrix assumes the model represents the subject-level random effects with a finite covariance matrix, commonly through a multivariate normal distribution such as $\boldsymbol{\eta}_i\sim N(\mathbf{0},\Omega)$. An estimated covariance matrix must be positive semidefinite: its variances cannot be negative, and its correlation must satisfy $-1\leq\rho\leq 1$. The diagnostic discussion below concerns the fitted covariance, not a claim that every dataset or every random-effects distribution is exactly normal.

What the Extreme Correlation Means Geometrically

For a two-effect covariance matrix, its determinant is $$ \det(\Omega)=\omega_1^2\omega_2^2(1-\rho^2). $$ If both variances are positive, the determinant approaches zero as $|\rho|$ approaches one. At $\rho=+1$, the standardized effects move together perfectly; at $\rho=-1$, they move in opposite directions perfectly. In either case, there is no random variation in one linear direction, and the covariance matrix has rank one rather than two. A correlation near either extreme therefore implies a direction with very little modeled variation.

The eigenvalues quantify variation along the principal directions of the covariance matrix. For a symmetric matrix with diagonal entries $a=\omega_1^2$, $c=\omega_2^2$, and off-diagonal entry $b=\rho\omega_1\omega_2$, they are

\lambda_{\pm} = \frac{a+c\pm\sqrt{(a-c)^2+4b^2}}{2}.

A small $\lambda_-$ relative to $\lambda_+$ indicates that the fitted random effects vary much less in one principal direction than in the other. Because covariance eigenvalues depend on the units and scales of the random effects, also inspect the eigenvalues of the standardized correlation matrix:

R = \begin{pmatrix} 1 & \rho\\ \rho & 1 \end{pmatrix}, \qquad \lambda_-(R)=1-|\rho|, \qquad \lambda_+(R)=1+|\rho|.

This standardized comparison is useful when the two random effects have very different units or standard deviations. There is no universal eigenvalue cutoff that diagnoses a failed fit: the ratio of the small to large eigenvalue, numerical precision, model scale, and the scientific purpose all matter.

Worked Example: Calculating Near-Singularity From a Positive Correlation

Suppose a hypothetical NLME fit estimates $\omega_1=0.30$, $\omega_2=0.20$, and $\rho=0.995$. The random-effect standard deviations are in the units of their respective model parameters. Calculate the covariance matrix, determinant, and standardized eigenvalues.

The variances are $0.30^2=0.0900$ and $0.20^2=0.0400$. The covariance is $0.995(0.30)(0.20)=0.0597$. Thus

\Omega = \begin{pmatrix} 0.0900 & 0.0597\\ 0.0597 & 0.0400 \end{pmatrix}, \qquad \det(\Omega)=0.0900(0.0400)-(0.0597)^2 =0.00003591.

The standardized eigenvalues are $1-0.995=0.005$ and $1+0.995=1.995$. Their ratio is $0.005/1.995\approx0.00251$. The small eigenvalue is only about 0.25% of the large one, indicating very little variation along one standardized direction. In this example, the estimated relationship is strongly positive, but the matrix is also close to singular. That is a reason to investigate the fit, not by itself proof that the estimate is wrong.

Two Different Kinds of Correlation

A frequent source of confusion is the distinction between correlation among the random effects and correlation among parameter estimates. The first is $\rho$ in $\Omega$: it describes the modeled subject-to-subject relationship between $\eta_1$ and $\eta_2$. The second is calculated from the estimated sampling covariance matrix of model parameters, often obtained from a local approximation based on the likelihood's curvature. For parameters $\theta_j$ and $\theta_k$, their estimated correlation is

\operatorname{Corr}(\hat{\theta}_j,\hat{\theta}_k) = \frac{\operatorname{Cov}(\hat{\theta}_j,\hat{\theta}_k)} {\operatorname{SE}(\hat{\theta}_j)\operatorname{SE}(\hat{\theta}_k)}.

A parameter-estimate correlation near $+1$ or $-1$ indicates that uncertainty in two estimates is strongly coupled under that approximation. It is not the random-effect correlation, and it does not by itself establish that the random-effect covariance matrix is near-singular. Conversely, an extreme random-effect correlation can occur without an equally extreme correlation between every pair of reported parameter estimates.

Worked Example: Computing a Parameter-Estimate Correlation

In a hypothetical fit, suppose the estimated standard errors of two log-standard-deviation parameters are $0.080$ and $0.110$, and their estimated covariance is $-0.00836$. Calculate the correlation between their estimates and interpret it.

Using the standardization formula,

\operatorname{Corr}(\hat{\theta}_1,\hat{\theta}_2) = \frac{-0.00836}{(0.080)(0.110)} = \frac{-0.00836}{0.00880} = -0.950.

The two parameter estimates have a strong negative correlation in the local uncertainty approximation: values of one estimate above its fitted value tend to accompany values of the other below its fitted value. Since these are log-standard-deviation parameters, the correlation is dimensionless. This result prompts inspection of their uncertainty and the fit's stability. It does not mean that the two subject-level random effects have correlation $-0.950$.

How to Diagnose a Near-Singular Fit

A near-extreme random-effect correlation can arise when the observed data mainly inform one combination of random effects rather than their separate directions. For example, over the observed sampling times, changes in two individual parameters may have similar effects on predictions, or one parameter may compensate for the other. Limited information per subject, a small number of subjects, or a complex random-effects structure can also leave the covariance weakly determined. These are possible explanations, not conclusions that can be drawn from the correlation alone.

Before acting on an extreme estimate, check whether the corresponding standard deviations are clearly distinguishable from zero. If one is near zero, its correlation with another effect may be poorly determined: when an effect has essentially no variation, the direction of its relationship with another effect is not well informed. This links the diagnosis to the earlier material on boundary estimates for variance components, but the question here is whether the joint covariance has a poorly supported direction.

A practical assessment uses several pieces of evidence together:

Inspect the estimated standard deviations, covariance, and correlation; verify that they are mutually consistent and that the matrix is positive semidefinite.
Calculate or inspect the eigenvalues of $\Omega$, and compare the smallest with the largest. Also inspect the standardized correlation matrix when the random effects have different scales.
Review parameter standard errors and correlations from the estimated parameter covariance matrix, if that approximation is available and reliable. Large parameter correlations can reveal coupled estimation uncertainty, but are not a substitute for examining $\Omega$.
Check whether estimates change markedly with reasonable alternative initial values or modest changes in model specification. Strong sensitivity is evidence that the covariance is not robustly determined.
Compare model fit and relevant predictions under a scientifically defensible simpler covariance structure. Do not decide solely from whether the optimizer prints a convergence message.

These checks have limits. Eigenvalues diagnose the geometry of the estimated covariance, not whether the model is scientifically appropriate. Parameter correlations based on local curvature can be unreliable when the likelihood is highly non-quadratic or the fit is near a boundary. And a high absolute correlation may be plausible when the underlying random effects truly move together. Interpret diagnostics in the context of the data, model, and purpose of the analysis.

Worked Example: Considering a Diagonal Covariance Structure

Suppose a hypothetical model for two random effects estimates $\omega_1=0.30$, $\omega_2=0.20$, and $\rho=0.995$. On the same dataset and likelihood scale, an unstructured covariance model has $-2\log L=412.6$, while a diagonal covariance model has $-2\log L=414.0$. Describe the structural change and what these results do and do not establish.

The unstructured two-effect matrix estimates three covariance parameters: two variances and their covariance (equivalently, two standard deviations and a correlation). The diagonal structure sets the covariance to zero and estimates two variances:

\Omega_{\mathrm{diag}} = \begin{pmatrix} 0.30^2 & 0\\ 0 & 0.20^2 \end{pmatrix} = \begin{pmatrix} 0.0900 & 0\\ 0 & 0.0400 \end{pmatrix}.

The objective difference is $414.0-412.6=1.4$, so the unstructured model has the smaller negative twice log-likelihood by 1.4 units. The diagonal model is simpler by one covariance parameter and, in this example, gives a relatively similar objective value. That comparison makes the diagonal structure worth considering; it does not prove that the simpler structure is scientifically correct or that the covariance is exactly zero. A formal model comparison requires a suitable inferential procedure and its assumptions. The choice should also consider scientific plausibility, estimate stability, and whether the simpler structure gives adequate predictions.

Choosing a Response and Avoiding Common Mistakes

When a covariance appears nearly singular, a defensible response is to investigate and document the issue rather than automatically deleting the correlation. Depending on the model and scientific context, possible structures include an unstructured covariance, a diagonal covariance (zero correlation), or a more restricted structure that estimates only scientifically motivated covariances. A simpler structure can reduce the number of covariance parameters and improve numerical stability, but it also changes what subject-level relationships the model permits.

Do not interpret a diagonal structure as saying the random effects do not exist: it retains their variances while setting their covariance to zero. Likewise, do not regard an estimated correlation of $0.999$ as a precise biological finding simply because it is close to one. Its apparent precision may be misleading if the covariance matrix is nearly singular or if one variance is near zero.

Common pitfall: “The random-effect correlation is $0.99$, so the two subject-level effects are definitively almost perfectly related, and I should report that as a stable feature.” The correction is to check both variances, the covariance eigenvalues, and the uncertainty and sensitivity of the fit. A near-extreme estimate is a diagnostic signal; it is not, on its own, evidence of a stable scientific relationship.

The main alternative to an unstructured covariance is not universally better or worse. An unstructured model allows the data to estimate every covariance but may be difficult to determine. A diagonal model is more parsimonious but imposes zero correlations. Restricted structures make intermediate assumptions and should be chosen for a scientific or design-based reason, not simply because they produce a preferred estimate. Whatever structure is used, report it clearly and assess whether its estimates and predictions are adequately supported.

Check Your Understanding

Answer each question, then compare with the solutions below.

For two positive random-effect standard deviations, what happens to the determinant of their covariance matrix as the absolute correlation approaches one?
A standardized two-effect covariance matrix has $\rho=-0.98$. What are its eigenvalues, and what do they indicate?
Explain the difference between a random-effect correlation of $0.95$ and a parameter-estimate correlation of $0.95$.
A fitted covariance has one standard deviation close to zero and a correlation near $-1$. Why should that correlation be interpreted cautiously?
What does a diagonal covariance structure impose, and what does it retain?

Solutions. (1) Since $\det(\Omega)=\omega_1^2\omega_2^2(1-\rho^2)$, it approaches zero as $|\rho|$ approaches one, provided both variances remain positive. (2) The eigenvalues are $1-0.98=0.02$ and $1+0.98=1.98$. The small eigenvalue indicates little standardized variation in one direction. (3) A random-effect correlation describes the modeled relationship between subject-level deviations; a parameter-estimate correlation describes coupled uncertainty between two estimated model parameters. They are different quantities. (4) With a standard deviation near zero, the corresponding random effect has little estimated variation, so its relationship with another effect may not be well informed; the extreme correlation alone is not reliable evidence of a stable relationship. (5) A diagonal structure sets the covariance, and hence the correlation when both variances are positive, to zero. It retains the two random-effect variances.