Covariance From a Nested Random-Intercept Model

Why a Shared Random Intercept Creates Covariance

A random-intercept model gives observations in the same cluster a shared source of variation. Even when those observations are independent after conditioning on the cluster effect, their outcomes can be associated marginally: learning that one outcome is high provides information about the shared effect and therefore about the other outcome. The key distinction is between conditional independence, given the random effect, and the marginal covariance after that effect is integrated out.

The target here is a two-level nested structure: observations $i$ are nested in clusters $j$, with one random intercept per cluster. The derivation applies to a generalized linear mixed model (GLMM) when its assumptions hold, and the examples show how the covariance depends on the response distribution and link.

Model and assumptions: Let $Y_{ij}$ be the response for observation $i$ in cluster $j$, and let $b_j$ be that cluster's random intercept. A random-intercept GLMM specifies $$ g\{E(Y_{ij}\mid b_j)\}=x_{ij}^{\mathsf T}\beta+b_j,\qquad b_j\sim N(0,\tau^2), $$ where $g$ is the link function, $x_{ij}$ is the covariate vector, and $\beta$ is the fixed-effect coefficient vector. Assume distinct clusters have independent random intercepts, and observations are conditionally independent given their cluster random effects and covariates. The conditional response distribution must also be specified; for example, it may be Gaussian or Poisson. The marginal covariance is obtained by averaging over the random effects.

Write $m_{ij}(b_j)=E(Y_{ij}\mid b_j)$. Under the conditional-independence assumption, two distinct observations in the same cluster have zero conditional covariance. Their marginal covariance is not generally zero, because their conditional means both depend on $b_j$. For observations in different clusters, both the random effects and the responses are independent under the stated model, so their marginal covariance is zero.

The Law of Total Covariance

For any two random variables $Y_1,Y_2$ and conditioning information $B$, the law of total covariance separates the covariance into a conditional component and a component due to changing conditional means:

\operatorname{Cov}(Y_1,Y_2) = E\{\operatorname{Cov}(Y_1,Y_2\mid B)\} + \operatorname{Cov}\{E(Y_1\mid B),E(Y_2\mid B)\}.

Apply this identity to two distinct observations $Y_{ij}$ and $Y_{kj}$ in cluster $j$, taking $B=b_j$. Conditional independence makes the first term zero. Thus

\operatorname{Cov}(Y_{ij},Y_{kj}) = \operatorname{Cov}\{m_{ij}(b_j),m_{kj}(b_j)\}, \qquad i\ne k.

This result is on the response scale. In a nonlinear GLMM, it is the covariance of the inverse-link conditional means over the distribution of $b_j$, not simply the variance $\tau^2$ of the random intercept. For observations from different clusters $j\ne \ell$, the corresponding conditional means depend on independent random effects. With conditional independence across clusters, their marginal covariance is zero.

Key result: In a two-level random-intercept model with conditionally independent observations, distinct observations in the same cluster have marginal covariance $\operatorname{Cov}\{m_{ij}(b_j),m_{kj}(b_j)\}$. Observations in different clusters have marginal covariance zero, provided the clusters' random effects are independent. The variance of a single observation additionally includes its expected conditional variance: $$ \operatorname{Var}(Y_{ij})=E\{\operatorname{Var}(Y_{ij}\mid b_j)\}+\operatorname{Var}\{m_{ij}(b_j)\}. $$

Worked Covariance Calculations

Worked Example: Poisson Responses With a Log Link

Suppose two distinct observations in one cluster have conditional distributions $Y_{1j}\mid b_j\sim\operatorname{Poisson}(2e^{b_j})$ and $Y_{2j}\mid b_j\sim\operatorname{Poisson}(3e^{b_j})$, independently given $b_j$, where $b_j\sim N(0,0.25)$. Find their marginal covariance and marginal means.

For a normal random variable $b\sim N(0,\tau^2)$, $E(e^{tb})=e^{t^2\tau^2/2}$. Here $E(Y_{1j}\mid b_j)=2e^{b_j}$ and $E(Y_{2j}\mid b_j)=3e^{b_j}$. Therefore the marginal means are

E(Y_{1j})=2e^{0.25/2}=2e^{0.125}\approx2.266,\qquad E(Y_{2j})=3e^{0.25/2}=3e^{0.125}\approx3.399.

Conditional independence means $E(Y_{1j}Y_{2j}\mid b_j)=E(Y_{1j}\mid b_j)E(Y_{2j}\mid b_j)=6e^{2b_j}$. Hence

\begin{aligned} \operatorname{Cov}(Y_{1j},Y_{2j}) &=E(Y_{1j}Y_{2j})-E(Y_{1j})E(Y_{2j})\\ &=6e^{2(0.25)}-\{2e^{0.125}\}\{3e^{0.125}\}\\ &=6(e^{0.5}-e^{0.25})\\ &\approx6(1.648721-1.284025)\\ &\approx2.188. \end{aligned}

Thus the two counts have marginal covariance about $2.188$ counts squared. The positive value arises because a larger $b_j$ raises both conditional means. The conditional Poisson variances do not enter this covariance for two distinct observations: conditional independence makes their conditional covariance zero.

Worked Example: Gaussian Random Intercept

Consider the Gaussian model $Y_{ij}=x_{ij}^{\mathsf T}\beta+b_j+\varepsilon_{ij}$, where $b_j\sim N(0,4)$, $\varepsilon_{ij}\sim N(0,9)$, and all errors are mutually independent and independent of the random effects. Find the variance of one observation, the covariance of two distinct observations in the same cluster, and the covariance for observations in different clusters.

For a single observation, the fixed part is nonrandom, and the two random components are independent. Therefore

\operatorname{Var}(Y_{ij})=\operatorname{Var}(b_j)+\operatorname{Var}(\varepsilon_{ij})=4+9=13.

For distinct observations in the same cluster, both include $b_j$, while their residual errors are independent:

\operatorname{Cov}(Y_{ij},Y_{kj}) =\operatorname{Cov}(b_j+\varepsilon_{ij},b_j+\varepsilon_{kj}) =\operatorname{Var}(b_j)=4,\qquad i\ne k.

For observations in different clusters, the random intercepts and residual errors are independent, so their covariance is $0$. The marginal covariance is therefore $4$ measurement-units squared for a same-cluster pair and $0$ measurement-units squared for a different-cluster pair. In this Gaussian identity-link model, the within-cluster correlation is $4/13\approx0.3077$; that simple ratio should not be transferred automatically to a nonlinear GLMM.

Worked Example: Reading a Nested Covariance Matrix

Suppose four Gaussian observations are ordered as $Y_{1A},Y_{2A},Y_{1B},Y_{2B}$, with two observations in each of clusters A and B. Let the random-intercept variance be $0.16$ and the independent residual variance be $0.49$, both in squared measurement units. Construct the marginal covariance matrix.

Each diagonal entry is the marginal variance $0.16+0.49=0.65$. Two different observations in the same cluster share the random intercept, so their covariance is $0.16$. Observations in different clusters have covariance zero. Thus

\operatorname{Var} \begin{pmatrix} Y_{1A}\\Y_{2A}\\Y_{1B}\\Y_{2B} \end{pmatrix} = \begin{pmatrix} 0.65&0.16&0&0\\ 0.16&0.65&0&0\\ 0&0&0.65&0.16\\ 0&0&0.16&0.65 \end{pmatrix}.

The two nonzero off-diagonal entries $0.16$ identify the within-cluster pairs. The matrix is block diagonal because the two clusters are independent under this model. Its diagonal values are variances; its off-diagonal values are covariances.

What the Covariance Says—and What It Does Not

The covariance describes how two outcomes vary together, in squared response units. A positive covariance means that deviations of the two outcomes from their marginal means tend to have the same sign. It is not itself a correlation: covariance depends on the units and marginal scales of both outcomes. In the Gaussian random-intercept model above, the covariance for distinct same-cluster observations equals $\tau^2$, and the marginal variance is $\tau^2+\sigma^2$, where $\sigma^2$ is the residual variance. Consequently, the within-cluster correlation is $\tau^2/(\tau^2+\sigma^2)$ under those specific assumptions.

For a log-link Poisson random-intercept model with conditional means $\lambda_i e^{b_j}$ and $\lambda_k e^{b_j}$, where $b_j\sim N(0,\tau^2)$, the same calculation gives

\operatorname{Cov}(Y_{ij},Y_{kj}) =\lambda_i\lambda_k\{e^{2\tau^2}-e^{\tau^2}\},\qquad i\ne k.

This makes clear why the response-scale covariance depends both on the random-effect variance and on the conditional means. For other links, the covariance is still the covariance of the two conditional-mean functions over the random-effect distribution, but a closed-form expression may not be available.

Assumptions, Diagnostics, and Alternatives

The result relies on a correctly specified cluster membership structure, independent random intercepts across clusters, and conditional independence of distinct responses given the random effects and covariates. The random-effect distribution and the conditional response distributions must also be appropriate for the application. If there is residual dependence after conditioning, the first term in the law of total covariance need not be zero, and the random intercept alone will not describe the full covariance.

A useful diagnostic question is whether the observed within-cluster dependence is broadly compatible with a common cluster shift. After accounting for covariates, look for remaining systematic patterns within clusters, such as dependence that changes with observation order or substantial cluster-specific variation in covariate effects. For a Gaussian model, residual plots and estimated within-cluster correlations can help reveal a poor fit. For a non-Gaussian GLMM, compare observed and model-based patterns using diagnostics suited to the response distribution, including simulated residual checks where appropriate. Sparse clusters or few clusters can make the covariance pattern difficult to assess reliably.

The main alternatives answer different modeling needs. An ordinary model that treats all observations as independent does not represent the shared cluster effect and may give misleading uncertainty for fixed effects. A Gaussian random-intercept model provides the simple variance-component covariance described above, but its response-scale covariance formula does not generally apply to a nonlinear GLMM. Cluster-robust standard errors can adjust some inferential uncertainty without specifying this same detailed covariance structure; they do not supply the model's predicted joint distribution. A random-intercept model is also limited when cluster dependence cannot reasonably be represented by a single shared shift. For example, residual dependence may remain after fitting it, or the data may not have the assumed nesting and independence structure.

Identify the pair.
Decide whether the observations share a cluster random intercept. This determines which random effect enters both conditional means.

Separate conditional and marginal dependence.
Use the law of total covariance. Under conditional independence, distinct observations have zero conditional covariance, but their random conditional means may covary.

Integrate over the random effect.
Calculate the covariance of the conditional means under the random-effect distribution. For different clusters, independent random effects and conditional independence imply zero covariance.

Common Pitfall

A frequent error is to say, “The observations are conditionally independent, so their covariance is zero.” That conclusion confuses conditional and marginal covariance. Conditional independence establishes $\operatorname{Cov}(Y_{ij},Y_{kj}\mid b_j)=0$, not $\operatorname{Cov}(Y_{ij},Y_{kj})=0$. The correction is to use the total-covariance identity: the marginal covariance can be positive because the two conditional means vary together as the shared $b_j$ varies. Conversely, do not report $\tau^2$ as the covariance for every GLMM: it equals the distinct-observation covariance in the Gaussian identity-link random-intercept model, but not generally in nonlinear models.

Check Your Understanding

Answer each question, then compare with the solutions.

Two distinct observations in one cluster are conditionally independent given $b_j$. Which term in the law of total covariance remains for their marginal covariance?
In the Gaussian model, let $\tau^2=2$ and $\sigma^2=6$. Find the marginal variance, the covariance of two distinct same-cluster observations, and their correlation.
Under the Poisson-lognormal model in the worked example, what is the covariance between observations in two different clusters?
Give one reason that a random-intercept model could fail to represent the dependence in the data.

Solutions. (1) The conditional covariance term is zero, leaving $\operatorname{Cov}\{m_{ij}(b_j),m_{kj}(b_j)\}$, the covariance of the conditional means over the random-effect distribution. (2) The marginal variance is $2+6=8$; the same-cluster covariance is $2$; the correlation is $2/8=0.25$. (3) It is zero under independent random effects across clusters and conditional independence. (4) For example, dependence may remain after conditioning on the shared intercept, or cluster-specific covariate effects may be needed rather than a common shift alone.