Random-Slope Error Covariance Structures

Why Random Slopes Change the Covariance Pattern

In repeated observations from the same unit, a random intercept lets each unit have its own level around the population regression line. A random slope also lets its response to a covariate vary. This second source of variation makes the covariance between two observations depend on the covariate values at which they were recorded. The resulting pattern is more flexible than the constant within-unit covariance produced by a random intercept alone.

The covariance pattern matters because GLS weights residuals according to their joint variability. To construct those weights, we first express the random effects and their design, then calculate the marginal covariance after accounting for the unobserved unit-specific effects.

Model and Assumptions

Let \(i=1,\ldots,N\) index units, and let unit \(i\) have \(m_i\) observations. For observation \(j\), let \(q_{ij}\) be the covariate whose slope varies across units. A random-intercept and random-slope model can be written as

y_{ij}=x_{ij}'\beta+u_i+v_iq_{ij}+\varepsilon_{ij}.

Here \(x_{ij}\) is the fixed-effects regressor vector, \(\beta\) is the fixed-effects coefficient vector, \(u_i\) is the unit-specific random intercept, and \(v_i\) is the unit-specific random slope. The covariate \(q_{ij}\) may also appear among the fixed-effects regressors. The random slope describes deviations from the population slope; it does not replace that fixed slope.

Write \(b_i=(u_i,v_i)'\). Conditional on the observed covariates, assume \(E(b_i)=0\), \(\operatorname{Var}(b_i)=G\), and \(E(\varepsilon_i)=0\), \(\operatorname{Var}(\varepsilon_i)=R_i\), where \(\varepsilon_i=(\varepsilon_{i1},\ldots,\varepsilon_{im_i})'\). Assume random effects and residual errors are uncorrelated, and that the unit-level error vectors are uncorrelated across distinct units. For the usual independent residual-error special case, \(R_i=\sigma^2 I_{m_i}\), with \(\sigma^2>0\). The covariance matrix of the random effects is

G= \begin{pmatrix} g_{00}&g_{01}\\ g_{01}&g_{11} \end{pmatrix}, \qquad g_{00}\geq 0,\quad g_{11}\geq 0,\quad g_{00}g_{11}-g_{01}^2\geq 0.

The restrictions make \(G\) positive semidefinite. In particular, \(g_{00}\) is the random-intercept variance, \(g_{11}\) is the random-slope variance, and \(g_{01}\) is their covariance. No normality assumption is needed to derive the covariance matrix or define the GLS criterion.

Constructing the Marginal Covariance Matrix

For unit \(i\), collect the responses into \(y_i\), the fixed-effects rows into \(X_i\), and the covariate values into the random-effects design matrix

Z_i= \begin{pmatrix} 1&q_{i1}\\ \vdots&\vdots\\ 1&q_{im_i} \end{pmatrix}.

Then \(y_i=X_i\beta+Z_ib_i+\varepsilon_i\). Its conditional mean is \(X_i\beta\). Since the random effects and residual errors have mean zero and are uncorrelated, the marginal covariance matrix is

V_i=\operatorname{Var}(y_i\mid X_i,Z_i)=Z_iGZ_i'+R_i.

The entry in row \(j\), column \(k\) follows by multiplying the corresponding rows of \(Z_i\) by \(G\):

\operatorname{Cov}(y_{ij},y_{ik}\mid X_i,Z_i) =g_{00}+g_{01}(q_{ij}+q_{ik})+g_{11}q_{ij}q_{ik}+(R_i)_{jk}.

With independent residual errors of variance \(\sigma^2\), the last term is \(\sigma^2\) when \(j=k\), and zero otherwise. Thus the random-effects contribution to the covariance depends on both covariate values. The variance of an observation at covariate value \(q\), including independent residual error, is \(g_{00}+2g_{01}q+g_{11}q^2+\sigma^2\). The random-slope component therefore changes both the variances and the off-diagonal covariances.

Worked Example: Building a Three-Observation Covariance Block

Suppose one unit is measured at \(q=0,1,2\). Let \(G=\begin{pmatrix}4&0.6\\0.6&0.25\end{pmatrix}\), and suppose independent residual errors have variance \(\sigma^2=1\). First, \(G\) is a valid covariance matrix because its determinant is \(4(0.25)-0.6^2=0.64>0\), and both diagonal entries are positive.

For the observation at \(q=0\), the variance is \(4+1=5\). At \(q=1\), it is \(4+2(0.6)(1)+0.25(1)^2+1=6.45\). At \(q=2\), it is \(4+2(0.6)(2)+0.25(2)^2+1=8.4\).

The covariance between the observations at \(q=0\) and \(q=1\) is \(4+0.6(0+1)+0.25(0)(1)=4.6\). Between \(q=0\) and \(q=2\), it is \(4+0.6(0+2)+0.25(0)(2)=5.2\). Between \(q=1\) and \(q=2\), it is \(4+0.6(1+2)+0.25(1)(2)=6.3\). Residual errors contribute no off-diagonal covariance.

Combining these entries gives

V_i= \begin{pmatrix} 5&4.6&5.2\\ 4.6&6.45&6.3\\ 5.2&6.3&8.4 \end{pmatrix}.

The off-diagonal entries are not all equal: observations at different covariate values share different amounts of random-intercept and random-slope variation. This is the key distinction from a model in which the only shared unit effect is a random intercept.

Covariate Centering and the Meaning of the Random Intercept

The intercept and slope components depend on the chosen origin of \(q\). If \(q^*=q-c\), then \(u_i+v_iq=(u_i+cv_i)+v_iq^*\). The random intercept in the centered parameterization is therefore \(u_i^*=u_i+cv_i\), while the random slope remains \(v_i\). Its covariance matrix is

G^*= \begin{pmatrix} g_{00}+2cg_{01}+c^2g_{11}&g_{01}+cg_{11}\\ g_{01}+cg_{11}&g_{11} \end{pmatrix}.

The fitted random-effects contribution and the marginal covariance for the same observations do not change under this reparameterization. But the numerical values called the random-intercept variance and intercept-slope covariance do change. Consequently, the random-intercept variance describes variation at \(q=0\) in the original parameterization, not a covariate-independent feature of the data.

Worked Example: Centering Without Changing the Covariance

Use the covariance parameters and observations from the previous example, and center at \(c=1\), so the original values \(q=0,1,2\) become \(q^*=-1,0,1\). The transformed parameters are

g_{00}^*=4+2(1)(0.6)+1^2(0.25)=5.45,\qquad g_{01}^*=0.6+1(0.25)=0.85,\qquad g_{11}^*=0.25.

For the two observations with \(q^*=-1\) and \(q^*=1\), the random-effects covariance is \(5.45+0.85(-1+1)+0.25(-1)(1)=5.2\). In the original parameterization, those are the observations at \(q=0\) and \(q=2\), whose covariance was \(4+0.6(0+2)+0.25(0)(2)=5.2\). The covariance agrees, as it must.

The variance at \(q^*=-1\), including residual error, is \(5.45+2(0.85)(-1)+0.25(-1)^2+1=5\), also matching the original variance at \(q=0\). Centering changes the parameter labels and values, not the underlying covariance of the observed responses.

From Covariance Blocks to the GLS Criterion

Stack all units’ observations into \(y\) and their fixed-effects design matrices into \(X\). Under the assumed lack of covariance between distinct units, the marginal covariance matrix \(V\) is block diagonal, with blocks \(V_1,\ldots,V_N\). If each \(V_i\) is positive definite, then \(V\) is positive definite and can be used to weight the residuals. For known covariance parameters, GLS chooses \(\beta\) to minimize

Q(\beta)=(y-X\beta)'V^{-1}(y-X\beta) =\sum_{i=1}^{N}(y_i-X_i\beta)'V_i^{-1}(y_i-X_i\beta).

This is a covariance-weighted sum of squared residuals. Residual directions with high variance receive less weight, while directions with low variance receive more. If \(X\) has full column rank, differentiating the criterion with respect to \(\beta\) gives the GLS normal equations and solution:

\frac{\partial Q(\beta)}{\partial\beta} =-2X'V^{-1}(y-X\beta)=0, \qquad \hat{\beta}_{GLS}=(X'V^{-1}X)^{-1}X'V^{-1}y.

This expression defines the GLS estimator when \(V\) is known. If the covariance parameters are unknown, a covariance estimate or a method for estimating them is needed before the criterion can be evaluated; that estimation is a separate task. The criterion itself does not require normally distributed errors. Under a Gaussian model it is also the part of the negative log likelihood that depends on \(\beta\) when \(V\) is fixed.

Worked Example: Evaluating a GLS Residual Criterion

Consider two observations from one unit at \(q=0\) and \(q=1\), with \(G=\begin{pmatrix}1&0.2\\0.2&0.25\end{pmatrix}\) and independent residual variance \(\sigma^2=0.5\). The determinant of \(G\) is \(0.25-0.04=0.21>0\), so it is positive definite. The marginal covariance entries are \(V_{11}=1+0.5=1.5\), \(V_{22}=1+2(0.2)(1)+0.25+0.5=2.15\), and \(V_{12}=1+0.2=1.2\). Thus

V= \begin{pmatrix}1.5&1.2\\1.2&2.15\end{pmatrix}, \qquad V^{-1}=\frac{1}{1.785} \begin{pmatrix}2.15&-1.2\\-1.2&1.5\end{pmatrix},

because \(\det(V)=1.5(2.15)-1.2^2=1.785>0\). Suppose a candidate coefficient vector gives residuals \(e=y-X\beta=(1,-1)'\), in the response’s units. Its GLS criterion contribution is

e'V^{-1}e =\frac{1}{1.785} \begin{pmatrix}1&-1\end{pmatrix} \begin{pmatrix}2.15&-1.2\\-1.2&1.5\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix} =\frac{2.15+1.5+2.4}{1.785} \approx 3.389.

For comparison, the unweighted sum of squared residuals is \(1^2+(-1)^2=2\). The GLS criterion is not expected to equal that sum: it evaluates this residual vector relative to the specified covariance pattern. The value is a criterion contribution, not by itself a test statistic or a p-value.

Common Mistake: Treating All Within-Unit Covariances as Equal

A frequent error is to assume that every pair of observations from the same unit has the same covariance just because they share a unit. That conclusion follows for the shared random-intercept contribution alone, but not generally when slopes also vary. The random-slope contribution \(g_{11}q_{ij}q_{ik}\), and the intercept-slope contribution \(g_{01}(q_{ij}+q_{ik})\), depend on the pair’s covariate values. Replacing the resulting \(V_i\) with a constant-covariance block discards this structure and changes the GLS weights.

Another related error is to interpret a negative covariance contribution as proof that the full covariance matrix is invalid. Individual off-diagonal covariances may be negative; validity requires the covariance matrix as a whole to be positive semidefinite. In applications, the parameters must define a valid \(G\), and the resulting \(V_i\) must be positive definite for ordinary GLS inversion.

Key takeaway: The random-effects design \(Z_i\) determines how the random-intercept and random-slope covariance matrix \(G\) translates into observed-response covariance: \(V_i=Z_iGZ_i'+R_i\). These covariance blocks, assembled across units, determine the GLS residual criterion.

Check Your Understanding

Use the model and definitions above to answer the following questions.

For a random-intercept and random-slope model with independent residual errors, write the covariance between two distinct observations at covariate values \(q_{ij}\) and \(q_{ik}\).
What does the parameter \(g_{01}\) represent, and what restriction involving \(g_{00}\), \(g_{01}\), and \(g_{11}\) ensures that \(G\) is positive semidefinite?
Explain why centering the slope covariate changes the random-intercept variance but does not change the covariance of the observed responses.
State the block-diagonal form of \(V\) when units are uncorrelated, and write the GLS criterion in terms of the unit-specific blocks.
Why is it incorrect to assume that the off-diagonal entries in \(V_i\) must all be equal when a random slope is present?