Random-Effects Covariance Parameterizations

Why the Random-Effects Covariance Structure Matters

A longitudinal mixed model can allow subjects to vary in their baseline biomarker level and in how quickly that biomarker changes. The covariance structure for those random effects determines not only how much subjects vary in each feature, but also whether the features tend to vary together. For example, subjects with higher-than-average baselines might tend to have steeper increases, flatter increases, or no systematic difference in slope.

Two common choices are a diagonal covariance matrix, which sets all distinct random-effect covariances to zero, and an unstructured covariance matrix, which estimates each variance and covariance. The unstructured choice is more flexible, but that flexibility requires information from the data. In particular, repeated measurements from many occasions on only a few subjects do not necessarily provide enough independent information to estimate a complicated subject-level covariance matrix precisely.

Model and Definitions

For subject \(i\) at occasion \(j\), write a linear mixed model as

Y_{ij}=X_{ij}^{\mathsf T}\beta+Z_{ij}^{\mathsf T}b_i+\varepsilon_{ij}.

Here \(Y_{ij}\) is the measured biomarker, \(X_{ij}\) and \(Z_{ij}\) are the fixed-effect and random-effect design vectors, \(\beta\) is the vector of fixed effects, \(b_i\) is the subject’s vector of random effects, and \(\varepsilon_{ij}\) is residual error. Let \(q\) be the number of random effects per subject. The standard covariance specification assumes subjects’ random effects are independent draws from a multivariate normal distribution with mean zero and common covariance matrix \(D\):

b_i\sim N_q(0,D).

The random-effects covariance matrix \(D\) must be symmetric and positive semidefinite. Its diagonal entries are random-effect variances; its off-diagonal entries are covariances. This tutorial compares specifications for \(D\), not choices for the residual covariance. For the basic interpretation below, assume random effects and residual errors are independent, the residual errors have mean zero, and the same residual structure is used when comparing the two choices for \(D\).

Definition: A diagonal random-effects covariance matrix estimates a variance for each random effect and constrains every covariance between distinct random effects to zero. An unstructured random-effects covariance matrix estimates all variances and all distinct covariances, subject to symmetry and positive semidefiniteness.

For \(q\) random effects, a diagonal matrix has \(q\) free covariance parameters. A symmetric unstructured matrix has \(q\) variances and \(q(q-1)/2\) distinct covariances, for a total of \(q(q+1)/2\). The zero-covariance restriction makes the diagonal specification a special case of the unstructured one.

Random Intercepts and Slopes

For a random intercept and random slope, let \(b_i=(b_{0i},b_{1i})^{\mathsf T}\), where the slope is for time measured in years. The unstructured choice is

D= \begin{pmatrix} \sigma_0^2 & \sigma_{01}\\ \sigma_{01} & \sigma_1^2 \end{pmatrix}, \qquad \operatorname{Corr}(b_{0i},b_{1i}) = \frac{\sigma_{01}}{\sqrt{\sigma_0^2\sigma_1^2}}.

The diagonal choice has the same two variances but sets \(\sigma_{01}=0\). The intercept variance is measured in squared biomarker units; the slope variance is measured in squared biomarker-units-per-year; and the covariance is measured in biomarker-units squared per year. The correlation is unitless and must lie between \(-1\) and \(1\).

Under this model, the random contribution to the trajectory at time \(t\) is \(b_{0i}+t b_{1i}\). Its variance is

\operatorname{Var}(b_{0i}+t b_{1i}) = \sigma_0^2+2t\sigma_{01}+t^2\sigma_1^2.

Thus the covariance affects how between-subject variation changes over time. Setting it to zero is a substantive restriction about the relationship between baseline deviations and slope deviations, not merely a computational shortcut.

Worked Example: Counting Covariance Parameters

Suppose a model includes a random intercept, a random linear slope, and a random quadratic-time effect, so \(q=3\). How many covariance parameters are estimated under each specification?

Diagonal. Estimate one variance for each of the three random effects: \(q=3\) covariance parameters.

Unstructured. Estimate three variances and one covariance for each of the \(\binom{3}{2}=3\) pairs. The total is \(3+3=6\), equivalently \(q(q+1)/2=3(4)/2=6\).

The unstructured model therefore estimates three additional covariance parameters. This count describes the covariance matrix alone; it does not include fixed-effect parameters or residual variance parameters.

What the Covariance Adds: A Calculation

The following calculation shows the implication of a nonzero covariance while holding the random-effect variances fixed. Suppose the random intercept has standard deviation 2 biomarker units, the random slope has standard deviation 1 biomarker unit per year, and their correlation is \(-0.5\). Then the variances are 4 and 1, and the covariance is \((-0.5)(2)(1)=-1\).

Worked Example: Comparing Trajectory Variation

Using those values, compare the variance in the random trajectory at year 2 under the unstructured model and under a diagonal model with the same variances.

Unstructured. The covariance matrix is \(D=\begin{pmatrix}4&-1\\-1&1\end{pmatrix}\). Substituting into the trajectory-variance formula gives \(4+2(2)(-1)+(2^2)(1)=4-4+4=4\). The standard deviation of the random trajectory contribution at year 2 is \(\sqrt{4}=2\) biomarker units.

Diagonal. Setting the covariance to zero gives \(4+2(2)(0)+(2^2)(1)=4+0+4=8\). The corresponding standard deviation is \(\sqrt{8}\approx2.828\) biomarker units.

The difference arises because the negative covariance means higher intercept deviations tend to accompany lower slope deviations, and vice versa. At year 2, those tendencies partly offset. This is a model-based implication, not evidence that the covariance is negative in any particular application.

Derivation and Interpretation of the Trade-Off

For any random-effect design vector \(z\), the random contribution is \(z^{\mathsf T}b_i\). Since \(b_i\) has covariance \(D\), its variance is \(z^{\mathsf T}Dz\). In the random-intercept and random-slope case, \(z=(1,t)^{\mathsf T}\). Expanding the matrix product gives

\begin{aligned} z^{\mathsf T}Dz &= \begin{pmatrix}1&t\end{pmatrix} \begin{pmatrix}\sigma_0^2&\sigma_{01}\\\sigma_{01}&\sigma_1^2\end{pmatrix} \begin{pmatrix}1\\t\end{pmatrix}\\ &=\sigma_0^2+2t\sigma_{01}+t^2\sigma_1^2. \end{aligned}

The factor \(2\) appears because the symmetric off-diagonal covariance contributes once from each cross-product. More generally, the marginal covariance between two measurements from the same subject includes the term \(Z_{ij}^{\mathsf T}D Z_{ik}\), in addition to any residual covariance. Therefore, a different \(D\) changes the model-implied relationship among repeated measurements even if the fixed-effects formula is unchanged.

The diagonal model is more economical: it estimates fewer parameters and can yield more stable estimates when the number of independent subjects is limited. But if the true random effects are correlated, forcing their covariance to zero can distort the implied trajectory variation and may affect other model estimates. The unstructured model avoids that particular restriction, but its additional parameters can be imprecise, especially with few subjects or weak information about one random effect.

Parameter count alone does not determine which structure is better. A useful decision considers the scientific plausibility of the restriction, the number and distribution of subjects’ observation times, convergence and boundary behavior, uncertainty in the covariance estimates, and whether conclusions are sensitive to the choice. Information criteria or a likelihood-based comparison can be considered when their assumptions are appropriate; a more flexible model’s improved fit by itself does not establish that its extra covariance estimates are reliable.

Worked Example: An Illustrative Covariance Estimate

For a small numerical illustration, suppose three hypothetical random-effect vectors, expressed as intercept deviations in biomarker units and slope deviations in biomarker units per year, are \((-2,0)\), \((0,1)\), and \((2,-1)\). Treat these as illustrative draws, not as the output of a fitted mixed model. Calculate their sample covariance matrix and compare its diagonal and unstructured forms.

Both sample means are zero. Using the sample covariance denominator \(m-1=2\), the intercept variance is \(\{(-2)^2+0^2+2^2\}/2=8/2=4\). The slope variance is \(\{0^2+1^2+(-1)^2\}/2=2/2=1\). The covariance is \(\{(-2)(0)+(0)(1)+(2)(-1)\}/2=-2/2=-1\).

The resulting unstructured sample covariance matrix is \(\begin{pmatrix}4&-1\\-1&1\end{pmatrix}\). Its correlation is \(-1/\sqrt{4\cdot1}=-0.5\). It is positive definite because its determinant is \(4(1)-(-1)^2=3>0\), and its diagonal entries are positive.

The diagonal version retains the two sample variances but sets the covariance to zero: \(\begin{pmatrix}4&0\\0&1\end{pmatrix}\). In an actual mixed model, covariance parameters are estimated through the model likelihood, not by simply taking the sample covariance of predicted subject effects. This small calculation illustrates what the restriction removes; it is not a recommended estimator for a fitted model.

Practical Checks, Comparisons, and Limitations

Before interpreting either structure, confirm that the random effects and their design columns are meaningful and that the time scale is stated. In a random-intercept and slope model, changing the time origin changes the intercept’s interpretation and can change the intercept-slope covariance. For example, replacing \(t\) by \(t-c\) rewrites the random intercept as \(b_{0i}+c b_{1i}\), so its covariance with the slope need not equal the covariance at the original time origin. Centering time can make the intercept more interpretable, but it does not make a covariance structure universally correct.

Useful warning signs include convergence problems, covariance estimates with very large uncertainty, correlations near \(-1\) or \(1\), or substantial changes in estimates under modest changes to the specification. These signs do not by themselves identify the cause. Check that the model is coded as intended, that the estimated matrix is admissible, and that the data contain useful information about each random effect. A diagonal fit can be a useful comparison, but it should not be selected solely because it converges more easily.

Feature	Diagonal	Unstructured
Covariance parameters for \(q\) random effects	\(q\)	\(q(q+1)/2\)
Distinct random effects allowed to covary	No; all covariances are fixed at zero	Yes; all distinct covariances are estimated
Main advantage	Fewer parameters; can be more stable with limited information	Flexible representation of between-subject variation
Main risk	May impose an unrealistic zero-covariance restriction	May estimate covariance parameters imprecisely or unstably

Both choices also share limitations. They typically assume one common random-effects covariance matrix across subjects and a specified distribution for the random effects. Neither choice repairs a misspecified fixed-effects mean structure, and neither determines the residual covariance structure. Those are separate modeling decisions.

Common Mistakes

Confusing the two covariance structures. A diagonal \(D\) does not mean there is no variation; it retains a variance for every random effect. It means only that distinct random effects have zero covariance.
Calling an unstructured covariance “assumption-free.” It still assumes a common covariance matrix, positive semidefiniteness, and the other model conditions. It is flexible about the entries, not free of assumptions.
Equating more parameters with better estimation. More parameters permit a wider range of covariance patterns, but may increase uncertainty when the data provide limited information.
Reading covariance as correlation. Covariance has units and depends on scale; correlation is unitless. Convert using \(\sigma_{01}/\sqrt{\sigma_0^2\sigma_1^2}\) when both variances are positive.
Treating a fitted covariance as a causal relationship. A random-effects covariance describes modeled between-subject variation; it does not show that one subject-level feature causes another.

Key takeaway: The diagonal specification estimates random-effect variances while constraining their covariances to zero. The unstructured specification estimates all variances and covariances. Choose between them by balancing the plausibility of that restriction against the information available to estimate the additional parameters.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For four random effects, how many covariance parameters are estimated under a diagonal specification and under an unstructured specification?
A random-intercept and slope model has variances 9 and 4 and covariance 3. What is the intercept-slope correlation?
At time \(t=1\), calculate the random trajectory variance for intercept variance 4, slope variance 1, and covariance \(-0.5\). What would it be under the diagonal restriction with the same variances?
Why might a researcher prefer a diagonal covariance structure when there are few subjects, and what risk does that choice carry?

Solutions. (1) The diagonal model has \(q=4\) parameters. The unstructured model has \(q(q+1)/2=4(5)/2=10\). (2) The correlation is \(3/\sqrt{9\cdot4}=3/6=0.5\). (3) The unstructured variance is \(4+2(1)(-0.5)+1^2(1)=4\); the diagonal variance is \(4+1=5\). (4) The diagonal model estimates fewer covariance parameters, which can improve stability when information is limited. Its risk is that a true nonzero covariance is constrained to zero, potentially misrepresenting between-subject trajectory variation.