Prediction of Random Effects · Tutorial 453 of 698
Predicting Correlated Random Intercepts and Slopes
You will derive and compute a group’s conditional random-intercept and random-slope distribution, including how their covariance affects predictions.
Advanced
11 min read
What You'll Learn
State the Gaussian mixed-model assumptions needed to predict a correlated random-intercept and random-slope vector.
Derive the conditional mean and covariance using both precision and covariance forms.
Calculate a group’s conditional random-effects distribution from observed residuals.
Predict a group-specific deviation at a chosen covariate value and calculate its conditional variance.
Recognize how centering, correlation assumptions, and residual structure affect prediction.
Distinguish conditional random-effect prediction from a fixed-effects-only prediction.
Why Correlated Intercepts and Slopes Need Joint Prediction
A group’s random intercept describes how its outcome level differs from the population level; its random slope describes how its covariate relationship differs. When these effects are correlated, information about one can change the prediction of the other. Predicting them separately as if they were independent can therefore misstate both the group-specific trajectory and its uncertainty.
We derive the conditional distribution for a group in a Gaussian linear mixed model. The derivation assumes the model parameters are known; in applications they are usually replaced by estimates. The result is a joint distribution for the group’s random effects given its observed outcomes, not a distribution for the fixed-effect parameters.
Model and Assumptions
Definition: For group \(i\), let \(y_i\) be its vector of observed outcomes, \(X_i\) its fixed-effects design matrix, and \(Z_i\) its random-effects design matrix. The Gaussian mixed model is
$$
y_i = X_i\beta + Z_i b_i + \varepsilon_i,
$$
where \(b_i=(b_{0i},b_{1i})'\) is the random-intercept and random-slope vector. Assume \(b_i\sim N(0,D)\), \(\varepsilon_i\sim N(0,R_i)\), and \(b_i\) is independent of \(\varepsilon_i\). The covariance matrix is
$$
D=
\begin{pmatrix}
\tau_0^2 & \tau_{01}\\
\tau_{01} & \tau_1^2
\end{pmatrix}.
$$
Here \(b_{0i}\) is measured in outcome units, while \(b_{1i}\) is measured in outcome units per unit of the covariate. Thus \(\tau_{01}\) has the corresponding product units. The matrix \(D\) must be a valid covariance matrix: in the non-degenerate case, \(\tau_0^2>0\), \(\tau_1^2>0\), and \(\tau_{01}^2<\tau_0^2\tau_1^2\). Groups are typically assumed to have independent random-effect vectors with this same covariance matrix.
For a random intercept and slope, a row of \(Z_i\) is \((1,x_{ij})\), where \(x_{ij}\) is the covariate value for observation \(j\) in group \(i\). The residual covariance \(R_i\) describes the within-group variation not explained by the fixed and random effects. It need not be an identity matrix, but it must be specified appropriately for the model. The formulas below require \(R_i\) to be positive definite and \(D\) to be positive definite when using matrix inverses.
Conditional Mean and Covariance
Write \(r_i=y_i-X_i\beta\) for the group’s residual vector relative to the fixed-effects mean. The conditional distribution of \(b_i\) given \(y_i\) is multivariate normal. Its mean is the group’s predicted random-effects vector, and its covariance describes uncertainty about that vector conditional on the model and observed group data.
Theorem: Under the Gaussian mixed-model assumptions,
$$
b_i\mid y_i \sim N(m_i,C_i),
$$
where the conditional mean and covariance have the precision forms
$$
C_i=(D^{-1}+Z_i'R_i^{-1}Z_i)^{-1},
\qquad
m_i=C_iZ_i'R_i^{-1}r_i.
$$
Equivalently, with \(V_i=Z_iDZ_i'+R_i\),
$$
m_i=DZ_i'V_i^{-1}r_i,
\qquad
C_i=D-DZ_i'V_i^{-1}Z_iD.
$$
The precision form shows how prior information and group data combine: \(D^{-1}\) is the prior precision, and \(Z_i'R_i^{-1}Z_i\) is the information contributed by the observations. The covariance form makes clear that conditioning on the observations reduces uncertainty from the unconditional covariance \(D\). These expressions are equivalent; they are not different prediction methods.
Derivation by Completing the Square
For this derivation, suppress the group subscript and treat \(\beta,D,R\) as fixed. Conditional on \(b\), the residual vector \(r=y-X\beta\) has mean \(Zb\) and covariance \(R\). The likelihood contribution for \(b\) is proportional to \(\exp\{-\tfrac12(r-Zb)'R^{-1}(r-Zb)\}\), while its normal prior contributes \(\exp\{-\tfrac12 b'D^{-1}b\}\). Multiplying these kernels gives
$$
p(b\mid y)\ \propto\ \exp\left\{-\frac12\left[b'(D^{-1}+Z'R^{-1}Z)b-2b'Z'R^{-1}r\right]\right\}.
$$
Let \(A=D^{-1}+Z'R^{-1}Z\) and \(h=Z'R^{-1}r\). Since \(A\) is positive definite, \(C=A^{-1}\) exists. The quadratic expression can be completed exactly:
$$
b'Ab-2b'h=(b-A^{-1}h)'A(b-A^{-1}h)-h'A^{-1}h.
$$
The last term does not depend on \(b\). The remaining kernel is a normal density with mean \(A^{-1}h\) and covariance \(A^{-1}\). This proves the precision formulas. Alternatively, \(b\) and \(y\) are jointly Gaussian, with \(\operatorname{Cov}(b,y)=DZ'\) and \(\operatorname{Var}(y)=ZDZ'+R=V\). The conditional Gaussian formula then gives the covariance form in the theorem. In particular, the conditional mean is centered on the observed residuals \(r\), not on the raw outcomes \(y\).
Worked Example: A Group with Two Covariate Values
Suppose a hypothetical group has two observations, at \(x=-1\) and \(x=1\). The fixed-effects model predicts 10 outcome units at both covariate values; the observed outcomes are 11 and 12. Thus \(r=(1,2)'\). Suppose the residual variance is 1 outcome-unit squared for each observation, with independent residuals, and
$$
Z=
\begin{pmatrix}
1&-1\\
1&1
\end{pmatrix},
\qquad
R=I,
\qquad
D=
\begin{pmatrix}
1&0.6\\
0.6&1
\end{pmatrix}.
$$
First, \(D^{-1}=\begin{pmatrix}1.5625&-0.9375\\-0.9375&1.5625\end{pmatrix}\), since the determinant of \(D\) is \(1-0.6^2=0.64\). Also, \(Z'Z=2I\), so
$$
C=(D^{-1}+Z'Z)^{-1}
=\begin{pmatrix}3.5625&-0.9375\\-0.9375&3.5625\end{pmatrix}^{-1}
=\begin{pmatrix}0.3016&0.07937\\0.07937&0.3016\end{pmatrix}.
$$
The predicted random intercept is 0.9841 outcome units, and the predicted random slope is 0.5397 outcome units per covariate unit. The conditional covariance between them is positive, 0.07937 in the corresponding product units. The calculation uses the specified positive-definite \(D\) and \(R\), and the two residuals are consistent with the stated fixed-effects predictions and observations.
Worked Example: One Observation and a Correlated Effects Pair
Consider a hypothetical group with one observation at \(x=2\), whose residual from the fixed-effects prediction is 1 outcome unit. Let \(Z=(1,2)\), \(R=1\), and \(D=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\). Here \(D\) is positive definite because its determinant is \(0.75>0\), and the residual variance is positive.
For one observation, the covariance-form denominator is \(R+ZDZ'\). Compute \(DZ'=(2,2.5)'\), so \(ZDZ'=1(2)+2(2.5)=7\), and the denominator is \(1+7=8\). The conditional mean is
$$
m=DZ'(R+ZDZ')^{-1}r
=\begin{pmatrix}2\\2.5\end{pmatrix}\frac{1}{8}(1)
=\begin{pmatrix}0.25\\0.3125\end{pmatrix}.
$$
The conditional covariance is
$$
C=D-\frac{(DZ')(ZD)}{8}
=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}
-\frac{1}{8}
\begin{pmatrix}4&5\\5&6.25\end{pmatrix}
=\begin{pmatrix}0.5&-0.125\\-0.125&0.2188\end{pmatrix}.
$$
The single observation informs the combination \(b_0+2b_1\), not the two effects independently. The positive unconditional correlation affects how the observed residual is allocated between intercept and slope; after conditioning, their covariance is negative. The observation does not identify both effects exactly, as shown by the nonzero conditional variances.
Predicting a Group Deviation at a Covariate Value
The predicted group-specific deviation in the linear predictor at covariate value \(x\) is \(b_{0i}+xb_{1i}\). If \(a(x)=(1,x)'\), its conditional mean and variance follow by linear transformation of the conditional Gaussian vector:
$$
E\{b_{0i}+xb_{1i}\mid y_i\}=a(x)'m_i,
\qquad
\operatorname{Var}\{b_{0i}+xb_{1i}\mid y_i\}=a(x)'C_i a(x).
$$
In expanded form, the variance is \(C_{00}+2xC_{01}+x^2C_{11}\). The covariance term matters: omitting it can either overstate or understate uncertainty, depending on its sign and the value of \(x\).
Worked Example: Predicting the Group Deviation at \(x=2\)
Use the conditional mean and covariance from the first example. At \(x=2\), the predicted random deviation is
$$
(1,2)\begin{pmatrix}0.9841\\0.5397\end{pmatrix}
=0.9841+2(0.5397)=2.0635
$$
outcome units. Its conditional variance is
$$
(1,2)
\begin{pmatrix}0.3016&0.07937\\0.07937&0.3016\end{pmatrix}
\begin{pmatrix}1\\2\end{pmatrix}
=0.3016+4(0.07937)+4(0.3016)=1.825
$$
outcome units squared, giving a conditional standard deviation of \(\sqrt{1.825}=1.351\) outcome units. This quantifies uncertainty about the group’s random deviation at \(x=2\), conditional on the model parameters and the group’s data. It is not the residual standard deviation and does not include uncertainty from estimating \(\beta\), \(D\), or \(R_i\).
Assumptions, Diagnostics, and Alternatives
The exact Gaussian conditional distribution requires a linear Gaussian outcome model with the stated covariance structure. In practice, examine whether residual patterns are compatible with the chosen \(R_i\), whether a linear random slope is reasonable across the observed covariate range, and whether the assumed distribution and shared covariance \(D\) are plausible across groups. A group with little covariate variation may provide weak information about its slope; the formula still returns a prediction, but that prediction can remain strongly influenced by the distribution \(D\).
Centering the covariate changes the meaning of the intercept. If \(x\) is replaced by \(x-c\), the new intercept is \(b_{0i}+cb_{1i}\); its variance and covariance with the slope change accordingly. Predictions for the same covariate values remain equivalent when the design and covariance are transformed consistently. A common error is to center the covariate but keep the old intercept-slope covariance matrix. That changes the model rather than merely changing the coordinate system.
Another pitfall is to set \(\tau_{01}=0\) automatically because the effects have different units. Different units do not imply independence. Constraining the covariance to zero is a modeling choice, and it changes \(D\), hence both conditional predictions and their uncertainty. Likewise, fitting a separate regression within each group does not generally give the same prediction: it ignores the shared distribution across groups and can be unstable for groups with few observations. A fixed-effects-only prediction instead sets the group deviation to zero; it targets the population-level linear predictor, not the observed group’s conditional prediction.
These formulas are specific to Gaussian outcomes with an identity-scale linear predictor. In non-Gaussian GLMMs, the conditional distribution of the random effects given the observations is generally not Gaussian, so this exact calculation does not carry over. Even in the Gaussian case, treating fitted parameters as known makes the conditional covariance an incomplete account of total uncertainty. Predictions also depend on the chosen random-effects structure: a poor covariance or residual model can produce precise-looking but misleading results.
Key takeaway: For correlated Gaussian random intercepts and slopes, predict the vector jointly using \(m_i=DZ_i'V_i^{-1}(y_i-X_i\beta)\) and retain the full conditional covariance \(C_i=D-DZ_i'V_i^{-1}Z_iD\). Its off-diagonal term can affect both the predicted trajectory and its uncertainty.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
In the precision formula, identify the prior precision and the information contributed by the observations.
For a predicted deviation \(b_0+xb_1\), write its conditional variance in terms of the entries of \(C\).
Why does one observation at a single covariate value generally fail to determine both random effects exactly?
What must change in the random-effects covariance when the covariate is recentered, if the model is to represent the same trajectories?
Solutions. (1) The prior precision is \(D^{-1}\), and the data contribution is \(Z_i'R_i^{-1}Z_i\); they add to give the conditional precision. (2) The variance is \(C_{00}+2xC_{01}+x^2C_{11}\), including the covariance term. (3) One observation informs only the linear combination \(b_0+xb_1\); uncertainty can remain in other combinations of the two effects. (4) Transform the intercept and slope together under the recentering, and transform their covariance matrix accordingly; retaining the old covariance generally specifies a different model.