Why covariance structure follows the design
When observations come from the same site, person, plot, or other cluster, they may be more alike than observations from different clusters. A model must represent that dependence: treating every response as independent can understate uncertainty and misrepresent the information in the data. The design indicates which observations share an experimental or sampling unit; the covariance structure describes how their responses are expected to vary together.
There is not one universally correct structure for every clustered study. A random intercept is plausible when clusters differ in their overall level. A random slope is useful when the effect of a measured predictor can vary across clusters. A correlated-error structure can represent residual dependence that remains even after the modeled cluster effects are included. These choices are informed by assignment, measurement schedule, and subject-matter knowledge, not just by which model is easiest to fit.
A general clustered model
Let \(Y_{ij}\) be the response from cluster \(i\) at observation \(j\). A useful general form is
Here \(\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}\) is the fixed-effects mean, \(\mathbf{b}_i\) contains cluster-specific random effects, and \(\varepsilon_{ij}\) is the residual error. For the covariance formulas below, assume clusters are independent; random effects have mean zero and covariance matrix \(\mathbf{G}\); and residuals have mean zero and covariance matrix \(\mathbf{R}_i\) within cluster \(i\). Assume also that random effects and residuals are independent of one another. Normal distributions are often used for model-based inference, but the covariance calculations themselves follow from these means, variances, and independence conditions.
For two observations \(j\) and \(k\) in the same cluster, the model implies
The first term is dependence induced by shared random effects; the second is residual dependence. A model can include one source, the other, or both when justified. For distinct clusters, the covariance is zero under the assumed independence between clusters.
Random intercepts: common cluster shifts
A random-intercept model gives each cluster its own shift in outcome level, while holding the modeled predictor effects constant across clusters. For a model with an intercept and fixed predictors, write the cluster-specific term as \(b_{0i}\), with variance \(\tau_{00}\). If residual errors are independent with variance \(\sigma^2\), then two different observations in one cluster share the same random intercept.
Under these assumptions, each response has variance \(\tau_{00}+\sigma^2\), and any two distinct responses in the same cluster have covariance \(\tau_{00}\). Their correlation is therefore \(\tau_{00}/(\tau_{00}+\sigma^2)\). This constant within-cluster correlation is often called the intraclass correlation for this simple model.
Worked Example: A Random Intercept for Repeated Clinic Measurements
Suppose a hypothetical study measures a continuous health score twice for each patient. The score is measured in points. The fixed-effects part describes the mean score at each visit. Assume different patients are independent, and assume that within a patient there is a normally distributed random intercept with variance \(9\) points squared and independent residual errors with variance \(16\) points squared. A shared patient-level tendency to have higher or lower scores makes a random intercept plausible.
For one measurement, the total variance is the random-intercept variance plus the residual variance:
For the two measurements on the same patient, the residual errors are assumed independent, but both responses contain the same patient intercept. Thus their covariance is \(9\ \text{points}^2\), and their correlation is
This model represents a common patient-specific elevation or reduction in score. It does not, by itself, represent a dependence that gets weaker as the gap between visits grows. The assumed equal covariance for every pair of distinct visits is a substantive feature of the model, not a consequence of the word “repeated.”
Random slopes: cluster-specific responses to a predictor
A random slope allows the effect of a predictor to vary across clusters. If \(x_{ij}\) is a quantitative predictor, a random-intercept and random-slope model has cluster-specific intercept \(b_{0i}\) and slope \(b_{1i}\). Their variances and covariance are represented by \(\mathbf{G}\). The random-effects contribution is \(b_{0i}+b_{1i}x_{ij}\).
The covariance now depends on the predictor values. If \(\tau_{11}=0\), there is no slope variation; if \(\tau_{01}=0\), intercepts and slopes are uncorrelated. A valid covariance matrix must be positive semidefinite, which for this two-by-two matrix requires \(\tau_{00}\geq0\), \(\tau_{11}\geq0\), and \(\tau_{00}\tau_{11}-\tau_{01}^2\geq0\).
A random slope is substantively plausible when the predictor’s effect could differ across the sampled clusters and the design provides information about that variation. For example, a time trend might differ among clinics, or the response to a treatment might differ among sites. Including a random slope does not mean that every cluster’s slope is estimated precisely; it means the model allows slope variation in the population of clusters represented by the random-effects assumption.
Worked Example: Covariance Under a Random Time Slope
Consider hypothetical scores measured at times \(x=0,1,2\) months for each clinic. Suppose clinic intercepts have variance \(9\) points squared, clinic time slopes have variance \(0.25\) points squared per month squared, and their covariance is \(1\) point squared per month. Assume independent residual errors with variance \(4\) points squared. The random-effects covariance matrix is valid because its determinant is \(9(0.25)-1^2=1.25>0\), and both variances are positive. The time predictor is measured in months, and observations from different clinics are assumed independent.
For a measurement at time \(x\), its variance is \(9+2(1)x+0.25x^2+4\). The residual variance is added because it contributes to each observation’s variance. At \(x=0,1,2\), the variances are \(13\), \(15.25\), and \(18\) points squared, respectively.
For the covariance of two distinct observations at times \(x\) and \(x'\), the residual covariance is zero by assumption. At months 0 and 2, for example,
At months 1 and 2, it is \(9+1(1+2)+0.25(1)(2)=12.5\ \text{points}^2\). These values differ because shared slope variation contributes according to both measurement times. The model is defensible if clinic-specific time trends are scientifically plausible; the time-indexed covariance pattern would not follow from a random intercept alone.
Correlated errors: dependence left in the residuals
Sometimes the main concern is not just that clusters have different levels or slopes. Nearby measurements may remain more alike than distant measurements, even after accounting for the modeled cluster effects. A residual covariance structure can describe this pattern. It is important to specify whether a proposed correlation belongs to the random effects, the residuals, or both, because the combination determines the total covariance.
Two common patterns help make the distinction concrete. In a compound-symmetry structure, every pair of distinct observations within a cluster has the same covariance (and, in the equal-variance case, the same correlation). A random intercept with independent, equal-variance residuals implies this pattern. In a first-order autoregressive residual structure, often denoted AR(1), residual correlations decrease with the number of equally spaced steps between observations: the correlation at lag \(h\) is \(\rho^h\), where \(|\rho|<1\).
Worked Example: AR(1) Residual Dependence Across Visits
Suppose a hypothetical device records a continuous response at four equally spaced visits for each participant. After the fixed mean and any justified random effects have been accounted for, suppose residual variance is \(16\) units squared and the residual correlation between adjacent visits is \(\rho=0.60\). Assume participants are independent and residual variances are equal across visits. AR(1) is a candidate if the residual association is expected to decline by visit separation.
The lag-one residual covariance is \(16(0.60)=9.60\) units squared. The lag-two correlation is \(0.60^2=0.36\), giving covariance \(16(0.36)=5.76\) units squared. At lag three the correlation is \(0.60^3=0.216\), giving covariance \(16(0.216)=3.456\) units squared. The resulting within-participant residual covariance matrix is
The equal spacing matters: a one-step lag must represent the same time separation throughout this AR(1) specification. If visits are irregularly spaced, using the visit number as if each interval were equal may not match the scientific dependence pattern. The numerical pattern is a model assumption to justify, not something established merely by observing repeated data.
How assignment and measurement guide the choice
The design provides a starting point for deciding which candidate structures make sense. Randomization identifies which units were assigned to conditions, while the observation schedule identifies which responses share a unit or occur in a meaningful order. These facts constrain plausible dependence; they do not uniquely determine a covariance model.
Use the assignment and sampling scheme to identify which observations belong to the same cluster and which clusters can reasonably be treated as independent.
A common baseline difference suggests a random intercept. A cluster-specific response to a predictor, such as time or treatment, can motivate a random slope when the design supports it.
After accounting for plausible random effects, ask whether residual association should remain, and whether it is constant across pairs or changes with time or distance.
Describe which observations covary, what determines the covariance, and which independence or equal-variance assumptions are being made. Reject structures that conflict with the design or scientific setting.
This reasoning prevents the model formula from becoming a list of terms chosen by habit. For example, if a treatment is assigned to whole sites, multiple measurements within each site do not create additional independently randomized sites. A site-level random effect may represent shared site variation, but it does not change the assignment unit or make the treatment effect supported by more site assignments.
A short derivation and practical cautions
The covariance formula follows by expanding the model and using the stated independence conditions. For two observations in the same cluster, subtract their fixed means. The remaining terms are \(\mathbf{z}_{ij}^{\mathsf T}\mathbf{b}_i+\varepsilon_{ij}\) and \(\mathbf{z}_{ik}^{\mathsf T}\mathbf{b}_i+\varepsilon_{ik}\). Their covariance is the sum of the random-effect covariance and residual covariance because the cross-covariances between \(\mathbf{b}_i\) and the errors are zero. The random-effect contribution is
Adding \(\operatorname{Cov}(\varepsilon_{ij},\varepsilon_{ik})\) gives the stated result. For a random intercept, both design vectors for that effect equal 1, so its contribution is \(\tau_{00}\) for every pair. For a random slope, the design vector includes the predictor value, so the contribution changes with the pair of values.
A frequent mistake is to say, “These are repeated observations, so I will use a random intercept,” without checking what pattern that implies. The correction is to name the implied pattern: with independent residual errors, a random intercept gives equal covariance for all distinct within-cluster pairs. If the scientific expectation is that correlation fades with time separation, a residual structure such as AR(1) may be a more direct candidate. Conversely, an AR(1) residual structure does not automatically represent cluster-specific baseline differences or treatment slopes. Both sources may be needed, but each needs a design-based and scientific justification.
No covariance structure can compensate for a mismatch between the model and assignment scheme. Nor should every possible random effect and residual correlation be added automatically. More flexible structures make more assumptions and require enough informative clusters and observations to estimate their parameters. The goal at this stage is to articulate defensible candidate structures and their implications; formal comparison of fitted models is a separate step.
Check Your Understanding
Answer each question by identifying the implied dependence pattern, not just naming a model term.
- With independent residual errors, what covariance does a random-intercept model imply for two distinct observations from the same cluster?
- Why can a random slope for time produce different covariances for different pairs of measurement times?
- In an AR(1) residual model with equal visit spacing, what is the residual correlation at lag three if the lag-one correlation is \(0.5\)?
- Give one design or scientific reason to include a random slope and one reason to consider correlated residuals.
- Why does repeated measurement alone not establish that a random-intercept structure is appropriate?