Within-Subject Residual Correlation Choices

Why Specify Residual Correlation?

Repeated measurements from one person may be related for more than one reason. A subject’s random intercept or slope can make their measurements move together, while residuals can also remain correlated after accounting for those subject effects. These are different sources of dependence. Choosing a residual correlation structure means specifying the latter: the covariance of unexplained within-subject deviations, conditional on the model’s random effects and covariates.

The distinction matters because the model uses the full covariance of the repeated outcomes to estimate fixed effects and their uncertainty. Treating residuals as independent when they remain correlated can misrepresent that covariance. Conversely, adding a complicated residual structure when random effects already account for much of the dependence can make parameters hard to distinguish and estimate.

Definition: For subject \(i\), write the repeated-outcome model as \(y_i=X_i\beta+Z_i b_i+\epsilon_i\), where \(y_i\) is the vector of outcomes, \(X_i\) and \(Z_i\) are design matrices, \(\beta\) is the fixed-effect vector, \(b_i\) is the subject-specific random-effect vector, and \(\epsilon_i\) is the residual vector. Assume \(E(b_i)=0\), \(E(\epsilon_i)=0\), \(\operatorname{Cov}(b_i)=G\), \(\operatorname{Cov}(\epsilon_i)=R_i\), and \(\operatorname{Cov}(b_i,\epsilon_i)=0\). The matrix \(R_i\) describes within-subject residual covariance; it is not, by itself, the marginal covariance of \(y_i\).

Under these assumptions, the marginal covariance is \(V_i=Z_iGZ_i^\mathsf{T}+R_i\). The first term is dependence induced by random effects; the second is residual dependence. The earlier material on marginal covariance decomposition established this addition rule. Here the focus is how to choose and interpret \(R_i\).

Three Common Residual Structures

Suppose subject \(i\) has \(m_i\) observations and, for the structures below, a common residual variance \(\sigma^2\). The structures differ in how residual covariance changes across pairs of observations.

Definition: The independent residual structure sets \(R_i=\sigma^2 I_{m_i}\). Residuals from different occasions are uncorrelated, conditional on the random effects and covariates.

Independence is a statement about residuals, not necessarily about the observed outcomes. If the subject has a random intercept, for example, two outcomes can be correlated even when \(R_i\) is diagonal.

Definition: The compound-symmetric (CS) residual structure sets every residual variance to \(\sigma^2\) and every pairwise residual covariance to \(\sigma^2\rho\), where \(\rho\) is the common residual correlation. Thus \(R_i=\sigma^2\{(1-\rho)I_{m_i}+\rho J_{m_i}\}\), where \(J_{m_i}\) is the matrix of ones. For a subject with \(m_i\) observations, positive definiteness requires \(-1/(m_i-1)<\rho<1\) when \(m_i>1\).

CS treats every pair of occasions alike, whether they are adjacent or far apart in time. The same \(\rho\) applies to all pairs. This can be a useful parsimonious summary when equal residual correlation is plausible, but it does not represent correlation that decays with temporal separation.

Definition: The first-order autoregressive residual structure, or AR(1), sets \((R_i)_{jk}=\sigma^2\rho^{|j-k|}\), where \(j\) and \(k\) index equally spaced occasions and \(-1<\rho<1\). The residual variance is \(\sigma^2\) at every occasion, and residual correlation decays geometrically with the number of occasion steps between measurements.

For example, under AR(1), adjacent occasions have correlation \(\rho\), occasions two steps apart have correlation \(\rho^2\), and those three steps apart have correlation \(\rho^3\). In the common case of positive \(\rho\), this expresses stronger residual association for nearby occasions. Negative \(\rho\) is mathematically possible and produces alternating signs across lags.

The AR(1) formula uses occasion number as its distance scale. If observations are irregularly spaced in actual time, a one-step gap need not represent the same duration for every pair. Applying the discrete AR(1) formula then asserts dependence by occasion index, not by elapsed time. A continuous-time correlation model can instead base correlation on time separation, but it is a different structure and should not be silently substituted for discrete AR(1).

Why These Matrices Are Valid Covariances

A covariance matrix must be symmetric and positive semidefinite; a nonsingular Gaussian likelihood ordinarily requires positive definiteness. The parameter restrictions above are not merely formal. They ensure that the proposed variances and covariances can describe a valid joint distribution.

Check compound symmetry.
The CS matrix has eigenvalue \(\sigma^2(1-\rho)\) for each of the \(m_i-1\) directions orthogonal to the vector of ones, and eigenvalue \(\sigma^2\{1+(m_i-1)\rho\}\) in the direction of the vector of ones.

Require positive eigenvalues.
With \(\sigma^2>0\), the first eigenvalue is positive when \(\rho<1\), and the second is positive when \(\rho>-1/(m_i-1)\). Together these give the CS bounds.

Check AR(1).
For equally spaced occasions, define a stationary residual process by \(\epsilon_j=\rho\epsilon_{j-1}+u_j\), with independent innovations \(u_j\) of variance \(\sigma^2(1-\rho^2)\), independent of the preceding residual. If \(|\rho|<1\), the stationary variance is \(\sigma^2\), and repeated substitution gives \(\operatorname{Cov}(\epsilon_j,\epsilon_k)=\sigma^2\rho^{|j-k|}\). This construction demonstrates that the AR(1) matrix is a valid covariance matrix.

The independent structure is the special case \(R_i=\sigma^2 I_{m_i}\), with all off-diagonal elements zero. It is also the \(\rho=0\) case of either CS or AR(1). That algebraic overlap does not make the structures interchangeable: away from zero, CS keeps all correlations equal, whereas AR(1) makes them depend on lag.

Worked Comparisons

Worked Example: Residual Correlation Versus Random-Intercept Correlation

Suppose a hypothetical model has a random intercept with variance \(0.36\), no random slope, and residual variance \(0.64\). Compare the marginal covariance of two measurements under independent residuals and under CS residuals with \(\rho=0.25\). Assume both measurements have the same fixed-effect design for the random intercept.

For a random intercept, the random-effect contribution to the covariance of any pair is \(0.36\). Under independent residuals, the residual covariance is zero, so the marginal covariance is \(0.36+0=0.36\). Each marginal variance is \(0.36+0.64=1.00\), giving marginal correlation \(0.36/1.00=0.36\).

Under CS residuals, the residual covariance is \(0.64(0.25)=0.16\). The marginal covariance is therefore \(0.36+0.16=0.52\). The marginal variances remain \(1.00\), so the marginal correlation is \(0.52/1.00=0.52\).

The calculation shows why “independent residuals” does not mean “independent repeated outcomes.” Even with independent residuals, the random intercept creates marginal correlation. The additional CS component raises the marginal correlation from \(0.36\) to \(0.52\).

Worked Example: Comparing CS and AR(1) at Three Occasions

Consider three equally spaced hypothetical occasions, residual variance \(\sigma^2=0.50\), and positive correlation parameter \(\rho=0.60\). Write the residual covariance matrices under CS and AR(1).

For CS, each off-diagonal covariance is \(0.50(0.60)=0.30\). Thus

R_{\mathrm{CS}}= \begin{pmatrix} 0.50&0.30&0.30\\ 0.30&0.50&0.30\\ 0.30&0.30&0.50 \end{pmatrix}.

For AR(1), adjacent occasions have covariance \(0.50(0.60)=0.30\), while occasions two steps apart have covariance \(0.50(0.60)^2=0.50(0.36)=0.18\). Therefore

R_{\mathrm{AR(1)}}= \begin{pmatrix} 0.50&0.30&0.18\\ 0.30&0.50&0.30\\ 0.18&0.30&0.50 \end{pmatrix}.

Both structures agree about adjacent residual covariances but differ for the first and third occasions. CS says their residual correlation is still \(0.60\); AR(1) says it is \(0.60^2=0.36\). These are residual covariances, not necessarily the marginal covariances of the observed outcomes.

Worked Example: Random Slopes and Residual AR(1)

Suppose hypothetical measurements occur at times \(t=0\) and \(t=1\), and the random effects are an intercept and a linear slope. Let the random-intercept variance be \(0.25\), the random-slope variance be \(0.04\), and their covariance be \(0.02\). Let the residual variance be \(0.49\), with AR(1) parameter \(\rho=0.40\). Calculate the marginal covariance and correlation for the two outcomes.

At times \(t\) and \(s\), the random-effect covariance is \(0.25+0.02(t+s)+0.04ts\). At times 0 and 1 this is \(0.25+0.02(0+1)+0.04(0)(1)=0.27\). The AR(1) residual covariance for adjacent occasions is \(0.49(0.40)=0.196\). Hence the marginal covariance is \(0.27+0.196=0.466\).

The random-effect variance at time \(t\) is \(0.25+2(0.02)t+0.04t^2\). At time 0 it is \(0.25\), so the marginal variance is \(0.25+0.49=0.74\). At time 1 it is \(0.25+0.04+0.04=0.33\), so the marginal variance is \(0.33+0.49=0.82\). The marginal correlation is

\frac{0.466}{\sqrt{0.74(0.82)}}= \frac{0.466}{\sqrt{0.6068}} \approx \frac{0.466}{0.7789} \approx 0.598.

Thus the marginal correlation is approximately \(0.598\), not \(0.40\). The AR(1) parameter is the residual correlation between adjacent occasions; random intercept and slope variation also contributes to the observed association.

Choosing and Assessing a Structure

Structure selection should begin with the observation schedule and the scientific meaning of residual dependence, not with a preference for the most complicated model. The main distinctions are summarized below.

Structure	Residual covariance pattern	Useful when	Main limitation
Independent	Zero for distinct occasions	Remaining residual dependence is plausibly negligible after random effects and covariates	Misses dependence not captured elsewhere
Compound symmetric	Same covariance for every distinct pair	A common residual correlation across occasions is plausible	Does not decay with time or occasion separation
AR(1)	Covariance decays geometrically with occasion lag	Occasions are equally spaced and nearby residuals are expected to be more alike	Its lag-based pattern can be unsuitable for irregular time gaps

A practical comparison fits plausible alternatives with the same fixed-effects structure, then checks whether the estimated covariance pattern and model fit support the added structure. Likelihood-based criteria can help compare candidate covariance models; when using restricted maximum likelihood, comparisons should keep the fixed-effects structure the same. Better fit alone is not proof that the covariance assumptions are scientifically correct. The estimated residual correlations should also be plausible and stable, and the model should be identifiable from the available repeated measurements.

Useful warning signs include estimates near the allowed parameter bounds, convergence problems, large changes in covariance estimates across modest model changes, or an estimated AR(1) pattern that does not make sense for the visit schedule. A subject with only one observation provides no within-subject residual pair to inform \(\rho\). If most subjects have very few repeated measurements, several different structures may fit similarly because the data contain little information to distinguish them. Residual summaries can also be noisy, especially when based on estimated random effects; they should be read as evidence about model adequacy, not as direct observations of the unobserved errors.

The three structures here all assume a common residual variance across occasions. They differ in correlation pattern, not in their variance assumptions. A model may have the right correlation shape but still be inadequate if residual variance changes over time; that is a separate modeling issue. Likewise, adding random slopes and adding residual correlation are not equivalent remedies: random effects represent subject-level variation in trajectories, while \(R_i\) represents remaining within-subject association after that variation is accounted for.

A Common Misinterpretation

A frequent mistake is to estimate a positive random-intercept variance and conclude that residuals must be correlated, or to set residual correlation to zero and conclude that repeated outcomes are independent. Neither conclusion follows. The random-intercept variance contributes covariance through \(Z_iGZ_i^\mathsf{T}\); the residual structure contributes covariance through \(R_i\). Only their sum gives the marginal covariance of the outcomes.

Key takeaway: State clearly whether a correlation refers to residuals conditional on random effects or to the observed outcomes marginally. Independent residuals can coexist with correlated outcomes, and a residual correlation parameter is not generally the marginal correlation.

Check Your Understanding

For each question, give your answer before reading the brief solution.

A model has independent residuals and a random intercept with variance \(0.20\); the residual variance is \(0.80\). What is the marginal correlation between two measurements with the same random-intercept design?
For three equally spaced occasions, an AR(1) model has residual variance \(0.40\) and \(\rho=0.50\). What is the residual covariance between the first and third occasions?
With four occasions, is a CS residual correlation of \(\rho=-0.40\) positive definite? Show the relevant lower bound.
In one sentence, distinguish the residual correlation parameter in AR(1) from the marginal correlation of two observed outcomes.
Why might discrete-time AR(1) be a poor choice when actual measurement times are irregular?

Solutions. (1) Each marginal variance is \(0.20+0.80=1.00\), and the covariance is \(0.20\), so the marginal correlation is \(0.20/1.00=0.20\). (2) The occasions are two steps apart, so the covariance is \(0.40(0.50)^2=0.10\). (3) For \(m_i=4\), the lower bound is \(-1/(4-1)=-1/3\). Since \(-0.40<-1/3\), the proposed CS matrix is not positive definite. (4) The AR(1) parameter gives residual correlation conditional on random effects for adjacent equally spaced occasions; marginal correlation also includes the random-effect covariance and depends on the marginal variances. (5) Discrete AR(1) counts occasion steps, so equal step distances may correspond to unequal elapsed times and fail to represent time-based decay.