Why Model Residual Covariance Across Endpoints?
When several outcomes are measured repeatedly on the same individual, their fitted means do not generally explain all their dependence. Even after accounting for covariates and shared random effects, an individual may have a residual deviation in one endpoint that is associated with a deviation in another endpoint at the same visit. Residual deviations may also persist across visits within an endpoint. A covariance model represents these remaining associations in a defined way.
The distinction from shared random effects is important. A shared random effect can induce dependence across endpoints because the endpoints respond to the same subject-level latent quantity. Residual covariance describes dependence that remains conditional on the modeled means and random effects. These components answer different questions and should not be treated as interchangeable.
For continuous outcomes with a Gaussian conditional model, the residual vector is often specified as multivariate normal with mean zero and covariance \(R_i\). For a non-continuous endpoint, residual covariance must be interpreted on the scale defined by its model. For example, some models introduce latent response variables; a covariance specified on that latent scale is not automatically the covariance of the observed categories or counts. The scale and meaning of the residuals therefore need to be explicit.
Separating the Two Types of Dependence
Suppose endpoints A and B are observed at visits \(t_1,\ldots,t_m\). Two kinds of residual association are especially useful to distinguish. First, residuals from the same endpoint at different visits can be serially correlated. Second, residuals from different endpoints at the same visit can be correlated. A model may also imply association between different endpoints at different visits.
A simple, interpretable construction is a separable covariance model. Let the endpoint covariance matrix be \(\Sigma_E\), and let the temporal correlation matrix be \(T_i\). For two endpoints, define \(\Sigma_E\) using endpoint-specific residual standard deviations \(\sigma_A,\sigma_B\) and same-visit residual correlation \(\gamma\). For equally spaced visits, use an AR(1) temporal correlation with lag-one correlation \(\phi\):
Here \(\otimes\) denotes the Kronecker product. With residuals ordered by endpoint, the vector is \((e_{i,A,1},\ldots,e_{i,A,m},e_{i,B,1},\ldots,e_{i,B,m})^\mathsf{T}\). The implied covariances are
Thus \(\sigma_A^2\) and \(\sigma_B^2\) are residual variances; \(\phi\) describes within-endpoint serial correlation; and \(\gamma\) is the cross-endpoint correlation at the same visit. The model also specifies cross-endpoint, cross-visit covariance. Under this separable structure, that covariance is the same-visit cross-endpoint covariance multiplied by the temporal correlation for the visit separation.
Why the Covariance Matrix Is Valid
A proposed covariance matrix must be positive semidefinite; otherwise it cannot be the covariance matrix of a random vector. For the separable construction, validity follows from the validity of its two component matrices. A symmetric matrix is positive semidefinite when every quadratic form \(v^\mathsf{T}Mv\) is nonnegative.
If \(\sigma_A,\sigma_B>0\) and \(|\gamma|<1\), then \(\Sigma_E\) is positive definite: its determinant is \(\sigma_A^2\sigma_B^2(1-\gamma^2)>0\), and its diagonal elements are positive. For equally spaced visits, the AR(1) matrix \(T_i\) is positive definite when \(|\phi|<1\). The Kronecker product of two positive-semidefinite matrices is positive semidefinite, so \(R_i=\Sigma_E\otimes T_i\) is a valid covariance matrix. This gives useful parameter constraints for estimation and interpretation.
The restriction to a shared \(\phi\) is also substantive: the separable model gives both endpoints the same temporal correlation pattern. It is not a universal property of repeated endpoints. More flexible structures can allow endpoint-specific serial correlations or less constrained cross-lag associations, but they need their own positive-definiteness conditions and may require substantially more information to estimate.
Worked Example: Building a Two-Endpoint, Two-Visit Matrix
Consider hypothetical residuals for endpoints A and B measured at visits 1 and 2. Suppose \(\sigma_A=2.00\) units, \(\sigma_B=3.00\) units, the same-visit cross-endpoint correlation is \(\gamma=0.400\), and the lag-one temporal correlation is \(\phi=0.600\). These values meet the conditions \(|\gamma|<1\) and \(|\phi|<1\).
The endpoint covariance matrix is \(\Sigma_E=\begin{pmatrix}4.00&2.40\\2.40&9.00\end{pmatrix}\), because \(\gamma\sigma_A\sigma_B=0.400(2.00)(3.00)=2.40\). The temporal matrix is \(T=\begin{pmatrix}1&0.600\\0.600&1\end{pmatrix}\). For endpoint-major ordering \((A_1,A_2,B_1,B_2)\), the Kronecker product gives
For example, the covariance between A at visit 1 and A at visit 2 is \(\sigma_A^2\phi=(2.00)^2(0.600)=2.40\) squared units. The same-visit covariance between A and B is \(\gamma\sigma_A\sigma_B=(0.400)(2.00)(3.00)=2.40\) in the product of their units. The covariance between A at visit 1 and B at visit 2 is \(\gamma\sigma_A\sigma_B\phi=(0.400)(2.00)(3.00)(0.600)=1.44\) in the product of their units.
The equal numerical values of two entries here are coincidental consequences of the chosen parameters; they represent different endpoint-visit pairs. The matrix diagonal also checks the specified variances: 4.00 for each A residual and 9.00 for each B residual.
Interpreting Residual and Marginal Covariance
The residual matrix is not necessarily the covariance of the observed repeated outcomes. If the conditional model includes random effects \(b_i\) with covariance \(G\), then, in a linear mixed model with design matrix \(Z_i\), the marginal covariance is
The first term represents variation and association induced by the random effects; the second is the remaining residual covariance. This decomposition is useful conceptually even when the model is expressed on a latent scale or does not have this exact linear-Gaussian form. In those settings, the observed-scale marginal covariance may require a separate calculation.
Worked Example: Distinguishing a Residual Covariance From a Marginal Covariance
Suppose one visit has a random intercept shared by two endpoints, with variance \(1.00\) squared units on each endpoint’s common scale. Suppose their conditional residual variances are each \(4.00\), and their conditional residual covariance is \(1.00\). Assume the random intercept enters each endpoint with coefficient 1.
The shared random intercept contributes \(1.00\) to the covariance between the endpoints. Adding the conditional residual covariance gives a marginal cross-endpoint covariance of \(1.00+1.00=2.00\) squared units (when both endpoint scales are the same). Each marginal variance is \(1.00+4.00=5.00\). The marginal correlation is therefore \(2.00/\sqrt{(5.00)(5.00)}=0.400\).
The residual correlation alone is \(1.00/\sqrt{(4.00)(4.00)}=0.250\). Reporting 0.400 as the residual correlation would incorrectly attribute the random-intercept contribution to the residual structure. The calculation assumes the random intercept is shared with coefficient 1 for both outcomes and is independent of the residual vector.
Irregular Visit Times and Model Choice
When visits are not equally spaced, lag can be defined by elapsed time rather than by the difference in visit numbers. A common temporal correlation function is continuous-time exponential decay, \(T_{jk}=\exp(-\kappa|t_j-t_k|)\), where \(t_j\) and \(t_k\) are times in a consistent unit and \(\kappa>0\) is a decay rate per unit time. The temporal correlation is 1 at zero separation and decreases as the time gap grows. Under a separable model, the cross-endpoint covariance at two distinct times remains \(\gamma\sigma_A\sigma_B\exp(-\kappa|t_j-t_k|)\).
Worked Example: Cross-Endpoint Covariance at Unequal Times
Suppose the same hypothetical endpoint scales and same-visit correlation apply, but measurements occur 3 days apart. Let \(\kappa=0.200\) per day. The temporal correlation is \(\exp[-(0.200\text{ per day})(3\text{ days})]=\exp(-0.600)\approx0.549\).
The covariance between A at the earlier time and B 3 days later is \((0.400)(2.00\text{ units})(3.00\text{ units})(0.549)\approx1.32\) unit-squared. The same-time cross-endpoint covariance remains \(2.40\) unit-squared because the time difference is zero. This calculation follows the exponential separable model; it does not establish that exponential decay is the correct structure for a particular data set.
Diagnostics, Alternatives, and Limitations
Covariance assumptions should be checked alongside the mean model. Residual summaries can be grouped by endpoint and by visit separation: within-endpoint residual pairs can reveal remaining serial association, while same-visit pairs across endpoints can reveal residual cross-endpoint association. For irregular schedules, grouping by elapsed-time separation is more informative than treating every adjacent pair as equally separated. Such checks are descriptive, especially with few subjects; noisy empirical correlations do not by themselves establish a better covariance model.
Warning signs include substantial residual correlation remaining at lags the model treats as independent, different apparent temporal patterns across endpoints, cross-endpoint associations that change markedly with visit separation, or estimated correlations close to their allowable boundaries. A fitted covariance matrix that is nearly singular can also indicate excessive flexibility or weak information in the design. For non-continuous outcomes, diagnostics must respect the modeled scale: raw differences between category labels are not generally meaningful residuals.
The principal alternatives express different assumptions:
| Structure | What it represents | Main trade-off |
|---|---|---|
| Independent conditional residuals | No residual association across endpoints or visits, after modeled effects | Simple, but can leave dependence unexplained |
| Separable endpoint-by-time covariance | Endpoint association and temporal association combine multiplicatively | Parsimonious, but imposes a shared temporal pattern |
| Unstructured covariance | Estimates each variance and covariance for the included measurements | Flexible, but parameters proliferate and estimation may be unstable |
A separable structure is attractive when its restrictions are scientifically plausible and the data cannot support a large set of free covariances. It can be inadequate when endpoints have different persistence, cross-endpoint associations depend on direction or lag, or the temporal pattern is not well represented by the chosen correlation function. Conversely, an unstructured matrix may overfit when visits are numerous or subjects are few. No covariance specification repairs a misspecified mean model, and covariance estimates cannot by themselves establish a biological mechanism.
Check Your Understanding
Answer each question before reading the brief solutions.
- In the separable model, which parameter describes same-endpoint serial correlation, and which describes same-visit cross-endpoint correlation?
- With \(\sigma_A=1.5\), \(\sigma_B=2.0\), and \(\gamma=0.300\), calculate the same-visit residual covariance between endpoints A and B.
- Under a separable model with \(\gamma=0.300\), \(\sigma_A=1.5\), \(\sigma_B=2.0\), and temporal correlation \(0.500\) across two visits, calculate the cross-endpoint covariance across those visits.
- Why is a residual covariance not automatically the marginal covariance of the observed outcomes when the model contains random effects?
- Name one limitation of an unstructured covariance matrix and one limitation of a separable covariance model.
Solutions. 1. The temporal parameter \(\phi\) (or the chosen time-decay parameterization) describes within-endpoint serial correlation; \(\gamma\) describes same-visit cross-endpoint correlation. 2. The covariance is \(\gamma\sigma_A\sigma_B=(0.300)(1.5)(2.0)=0.900\) in the product of the endpoint units. 3. It is \(\gamma\sigma_A\sigma_B(0.500)=(0.300)(1.5)(2.0)(0.500)=0.450\) in the product of the endpoint units. 4. Random effects contribute an additional covariance term; in a linear mixed model the marginal covariance is \(Z_iGZ_i^\mathsf{T}+R_i\), not just \(R_i\). 5. An unstructured matrix can require too many parameters for the available data; a separable model can impose an incorrect shared temporal pattern or multiplicative cross-lag structure.