Deriving Covariance Across Covariate Values

Why Variation Depends on the Covariate Value

With a random intercept and random slope, clusters can differ both in their overall level and in how their conditional linear predictor changes with a covariate. Consequently, the amount of between-cluster variation need not be the same at every covariate value. It also matters whether we compare predictors evaluated at the same value or at two different values: their covariance depends on both values.

This tutorial derives those quantities on the linear-predictor scale. The formulas apply to a GLMM with a single covariate having a random slope, whether the link is the identity, log, or logit. They describe variation induced by the random effects, not variation in the response conditional on those effects.

Model and Assumptions

Definition: For cluster $j$ and covariate value $x$, write the conditional linear predictor as $$ \eta_j(x)=\beta_0+\beta_1x+b_{0j}+b_{1j}x. $$ Here, $\beta_0+\beta_1x$ is the fixed part, $b_{0j}$ is the cluster-specific random-intercept deviation, and $b_{1j}$ is the cluster-specific random-slope deviation. Assume $E(b_{0j})=E(b_{1j})=0$, $\operatorname{Var}(b_{0j})=\tau_{00}$, $\operatorname{Var}(b_{1j})=\tau_{11}$, and $\operatorname{Cov}(b_{0j},b_{1j})=\tau_{01}$. The random effects are independent across clusters. The covariance derivations require finite second moments; normality is not needed for these formulas.

The covariate is treated as fixed when taking these moments. The fixed part of $\eta_j(x)$ is therefore constant across clusters at a given $x$, so it contributes no between-cluster variance. Define the random contribution as $u_j(x)=b_{0j}+b_{1j}x$. Then $\eta_j(x)=\beta_0+\beta_1x+u_j(x)$, and the desired variance and covariance are those of $u_j(\cdot)$.

The random-effects covariance matrix must be positive semidefinite:

\begin{pmatrix} \tau_{00} & \tau_{01}\\ \tau_{01} & \tau_{11} \end{pmatrix}, \qquad \tau_{00}\geq 0,\quad \tau_{11}\geq 0,\quad \tau_{01}^{\,2}\leq \tau_{00}\tau_{11}.

The last inequality is the covariance bound. It ensures that the intercept and slope variances and their covariance can belong to a valid joint distribution. If either variance is zero, the bound requires the covariance to be zero as well.

Deriving Variance and Covariance Across Values

At a single covariate value, the random contribution is a linear combination of the random intercept and random slope. Its variance includes the variances of both terms and twice their covariance.

\begin{aligned} \operatorname{Var}\{\eta_j(x)\} &=\operatorname{Var}\{u_j(x)\}\\ &=\operatorname{Var}(b_{0j}+xb_{1j})\\ &=\tau_{00}+2x\tau_{01}+x^2\tau_{11}. \end{aligned}

For two possibly different covariate values, use the covariance rule for linear combinations. The same random intercept and slope enter both predictors, but with different multipliers:

\begin{aligned} \operatorname{Cov}\{\eta_j(x),\eta_j(z)\} &=\operatorname{Cov}\{b_{0j}+xb_{1j},\,b_{0j}+zb_{1j}\}\\ &=\operatorname{Var}(b_{0j}) +z\operatorname{Cov}(b_{0j},b_{1j}) +x\operatorname{Cov}(b_{1j},b_{0j}) +xz\operatorname{Var}(b_{1j})\\ &=\tau_{00}+(x+z)\tau_{01}+xz\tau_{11}. \end{aligned}

This derivation shows why the covariance is not generally just the random-intercept variance. The random slopes contribute $xz\tau_{11}$, while intercept–slope dependence contributes $(x+z)\tau_{01}$. Setting $z=x$ recovers the variance formula. In particular, the variance at $x$ must be nonnegative for every $x$, as follows from the valid random-effects covariance matrix.

If $x$ and $z$ are different, the covariance can be positive, zero, or negative. A negative covariance means that clusters with a relatively high predictor at one value tend, in the random-effects distribution, to have a relatively low predictor at the other value. It does not mean either predictor has negative variance.

Worked Example: Between-Cluster Variance at One Value

Suppose a hypothetical model uses $x$ measured in tens of treatment visits. Its random-effects estimates are $\tau_{00}=0.36$, $\tau_{11}=0.04$, and $\tau_{01}=-0.06$. Find the between-cluster variance and standard deviation of the conditional linear predictor at $x=2$.

First check that the covariance parameters are admissible. The variances are nonnegative, and $ \tau_{01}^{\,2}=(-0.06)^2=0.0036 \leq (0.36)(0.04)=0.0144 $. Thus the covariance matrix passes the positive-semidefinite check.

Substitute $x=2$ into the variance formula:

\begin{aligned} \operatorname{Var}\{\eta_j(2)\} &=\tau_{00}+2(2)\tau_{01}+2^2\tau_{11}\\ &=0.36+4(-0.06)+4(0.04)\\ &=0.36-0.24+0.16\\ &=0.28. \end{aligned}

The corresponding between-cluster standard deviation is $\sqrt{0.28}\approx 0.5292$ linear-predictor units. The negative intercept–slope covariance reduces the variance at this positive covariate value relative to the sum of the intercept and slope variance contributions alone.

Worked Example: Covariance and Correlation at Two Values

Consider a hypothetical model with $\tau_{00}=0.25$, $\tau_{11}=0.04$, and $\tau_{01}=0.05$. Calculate the covariance and correlation between cluster-specific linear predictors at $x=-1$ and $z=2$.

The covariance matrix is valid because $0.05^2=0.0025 \leq (0.25)(0.04)=0.0100$. Apply the cross-value formula:

\begin{aligned} \operatorname{Cov}\{\eta_j(-1),\eta_j(2)\} &=\tau_{00}+(-1+2)\tau_{01}+(-1)(2)\tau_{11}\\ &=0.25+(1)(0.05)+(-2)(0.04)\\ &=0.25+0.05-0.08\\ &=0.22. \end{aligned}

To obtain the correlation, calculate each variance. At $x=-1$, $0.25+2(-1)(0.05)+(-1)^2(0.04)=0.19$. At $z=2$, $0.25+2(2)(0.05)+2^2(0.04)=0.61$. Therefore,

\operatorname{Corr}\{\eta_j(-1),\eta_j(2)\} =\frac{0.22}{\sqrt{(0.19)(0.61)}} =\frac{0.22}{\sqrt{0.1159}} \approx \frac{0.22}{0.3404} \approx 0.646.

Across clusters, the two linear predictors have a positive correlation of about 0.646 in this random-effects distribution. This is a correlation between predictors for the same cluster evaluated at two covariate values, not a correlation between observed responses.

Worked Example: A Negative Covariance Across Values

Suppose a hypothetical model has $\tau_{00}=0.25$, $\tau_{11}=0.09$, and $\tau_{01}=-0.10$. Find the covariance between predictors at $x=0$ and $z=3$, and interpret its sign.

The covariance matrix is admissible because $(-0.10)^2=0.0100 \leq (0.25)(0.09)=0.0225$. The cross-value calculation is

\begin{aligned} \operatorname{Cov}\{\eta_j(0),\eta_j(3)\} &=0.25+(0+3)(-0.10)+(0)(3)(0.09)\\ &=0.25-0.30+0\\ &=-0.05. \end{aligned}

The individual variances are nonnegative: at $x=0$, the variance is $0.25$; at $z=3$, it is $0.25+2(3)(-0.10)+3^2(0.09)=0.46$. The covariance is negative because the negative intercept–slope contribution outweighs the shared intercept contribution for this pair of values. The corresponding correlation is $ -0.05/\sqrt{(0.25)(0.46)}\approx -0.147 $. Thus clusters tending to have higher predictors at $x=0$ tend to have lower predictors at $z=3$, although the relationship is modest in this example.

What the Formulas Do—and Do Not—Describe

These quantities are between-cluster moments of the conditional linear predictors under the random-effects distribution, with the covariate values held fixed. They are not response-scale variances. In a logistic GLMM, for example, $\eta_j(x)$ is a conditional log-odds; its variance does not equal the variance of the conditional probability. In a log-link model, the predictor is a conditional log mean, and its variance is not the variance of the conditional mean on the response scale. Transforming to response values requires separate calculations.

The formulas also treat the model parameters as given. If $\tau_{00}$, $\tau_{11}$, and $\tau_{01}$ are estimated, plugging their estimates into these expressions gives an estimated random-effects variance or covariance. It does not, by itself, account for uncertainty in those parameter estimates.

For different clusters $j$ and $k$, the random-effects independence assumption gives zero covariance between their random contributions. This is distinct from the within-cluster covariance across $x$ and $z$, which arises because the same cluster-specific intercept and slope appear at both values.

Checks, Alternatives, and Common Pitfalls

The main diagnostic checks are algebraic and substantive. Verify that the estimated covariance matrix is positive semidefinite, calculate variances at covariate values relevant to the data, and check that computed variances are not negative beyond numerical rounding. A materially negative variance signals an invalid covariance matrix, a formula or substitution error, or an estimation problem. Very large variances at covariate values far outside the observed range may also warn that the fitted linear random-slope structure is being extrapolated beyond the data.

A random-intercept-only model is a simpler alternative: it sets the random-slope variance and intercept–slope covariance to zero, so the predictor variance is constant at $\tau_{00}$ and the within-cluster covariance across any two covariate values is also $\tau_{00}$. A random-intercept/random-slope model with independent random effects instead sets $\tau_{01}=0$. In that case, the variance is $\tau_{00}+x^2\tau_{11}$, and the cross-value covariance is $\tau_{00}+xz\tau_{11}$. These simplifications change the implied pattern of cluster variation; they should not be assumed merely to make arithmetic easier.

Omitting the covariance term. The variance is not generally $\tau_{00}+x^2\tau_{11}$; it also contains $2x\tau_{01}$. The cross-value covariance contains $(x+z)\tau_{01}$.
Confusing covariance and correlation. Covariance retains squared linear-predictor units and depends on scale. Correlation is dimensionless and requires division by the product of standard deviations.
Assuming covariance must be positive. A valid random-effects covariance matrix ensures nonnegative variance, but covariance between predictors at distinct values can be negative.
Ignoring the covariate scale. The numerical values of $x$ and $z$ must use the same units and coding used to define the fitted random slope.

The central practical point is that random-slope variation is location-dependent on the covariate scale. The intercept–slope covariance can increase variation at some values and reduce it at others, while the slope variance contributes quadratically to a single-value variance and through $xz$ to a cross-value covariance.

Check Your Understanding

Use the formulas above to answer each question. Solutions are provided below the questions.

If $\tau_{00}=0.16$, $\tau_{11}=0.09$, and $\tau_{01}=0.06$, what is the predictor variance at $x=1$?
For the same parameters, what is the covariance between predictors at $x=0$ and $z=2$?
Can $\tau_{00}=0.10$, $\tau_{11}=0.20$, and $\tau_{01}=0.16$ form a valid random-effects covariance matrix? Explain.
In one sentence, distinguish the covariance of predictors at two covariate values from the covariance of responses.

Solutions. (1) The covariance matrix is valid because $0.06^2=0.0036\leq(0.16)(0.09)=0.0144$. The variance is $0.16+2(1)(0.06)+1^2(0.09)=0.37$. (2) The covariance is $0.16+(0+2)(0.06)+(0)(2)(0.09)=0.28$. (3) No: $0.16^2=0.0256$, whereas $(0.10)(0.20)=0.0200$, so the covariance bound is violated. (4) Predictor covariance describes variation in the random-effects contribution to conditional linear predictors; response covariance also depends on the response distribution and link.