Tutorials › Multivariate Statistics (Graduate) › Worked Comparison of Structured Covariance Fits

Home  /  Tutorials  /  Structured Covariance Models

Structured Covariance Models · Tutorial 902 of 975

Worked Comparison of Structured Covariance Fits

Fit and compare three covariance structures for repeated measurements, checking how likelihood, information criteria, and residual patterns support a model choice.

Advanced 12 min read

What You'll Learn

  • Specify compound-symmetry, AR(1) Toeplitz, and unstructured covariance models for repeated measurements
  • Derive a compound-symmetry maximum-likelihood fit from a sample covariance matrix
  • Compare fitted likelihoods, AIC, and BIC using a common data summary
  • Interpret covariance residuals and distinguish a fitted pattern from evidence of population fit
  • Recognize how parameter counts and likelihood gains affect model comparisons

Why Compare Structured Covariance Fits?

Repeated measurements on the same subject are generally dependent. A covariance model describes that dependence, and its structure can make estimation more stable and interpretation more direct. The choice is consequential: compound symmetry assumes one common correlation, AR(1) allows correlation to decay with measurement lag, and an unstructured model estimates every variance and covariance separately.

A useful comparison does more than report which model has the largest likelihood. It checks what each model estimates, accounts for the number of covariance parameters, and inspects the discrepancy between observed and fitted covariance patterns. The calculations below use a deliberately small, transparent example so that the comparison can be verified directly.

Setup: Suppose \(n\) independent subjects each have \(p=3\) measurements, and each subject's measurement vector has a known mean of zero. Let \(S=n^{-1}\sum_{i=1}^n X_iX_i^\mathsf{T}\) be the empirical covariance matrix. We compare Gaussian likelihoods with the same mean specification. For the worked comparison, the sufficient summary is $$ S=\begin{pmatrix} 1&0.50&0.25\\ 0.50&1&0.50\\ 0.25&0.50&1 \end{pmatrix}, \qquad n=20. $$ The matrix is positive definite: it is the \(3\times3\) AR(1) correlation matrix with parameter \(0.5\), whose determinant is \((1-0.5^2)^2=9/16>0\). Such a summary is feasible for 20 centered observation vectors; for example, three orthonormal centered vectors in \(\mathbb R^{20}\) can be transformed to have cross-product \(20S\).

The Three Candidate Models

For \(p=3\), a compound-symmetry (CS) covariance has one common variance and one common off-diagonal covariance. It is often parameterized by a variance \(\sigma^2\) and correlation \(\rho\). The AR(1) Toeplitz model has a common variance and correlation \(\rho^h\) at lag \(h\). The unstructured (UN) model has no restrictions beyond positive definiteness.

Definitions: The three covariance models are $$ \Sigma_{\mathrm{CS}}=\sigma^2\{(1-\rho)I_3+\rho J_3\},\qquad (\Sigma_{\mathrm{AR}})_{jk}=\sigma^2\rho^{|j-k|},\qquad \Sigma_{\mathrm{UN}}=\text{any positive-definite }3\times3\text{ matrix}. $$ Here \(I_3\) is the identity and \(J_3\) is the all-ones matrix. CS requires \(\sigma^2>0\) and \(-1/2<\rho<1\); AR(1) requires \(\sigma^2>0\) and \(|\rho|<1\). Their covariance parameter counts are 2, 2, and 6, respectively.

The distinction between the first two models is visible in their lag structure. CS requires the lag-1 and lag-2 covariances to be equal. AR(1) instead makes the lag-2 covariance the square of the lag-1 correlation times the variance. UN allows both lag patterns, and all three variances, to be estimated freely.

Worked Example: Fitting Compound Symmetry

Worked Example: Fitting Compound Symmetry

Fit the CS model to the displayed \(S\). The three sample variances are all 1. The three distinct sample covariances are \(0.50,0.25,0.50\), whose average is $$ \frac{0.50+0.25+0.50}{3}=\frac{1.25}{3}=\frac{5}{12}\approx0.4167. $$ For this balanced example, the CS maximum-likelihood fit is the matrix with diagonal 1 and off-diagonal entries \(5/12\). Thus \(\hat{\sigma}^2=1\) and \(\hat{\rho}=5/12\). This estimate is admissible because \(-1/2<5/12<1\).

The fit can be checked using the eigenspaces of a CS matrix. The unit vector proportional to \((1,1,1)^\mathsf{T}\) is its common-measurement direction; the two-dimensional orthogonal contrast space has a common eigenvalue. For the empirical \(S\), the eigenvalue in the common direction is $$ \frac{\boldsymbol{1}^\mathsf{T}S\boldsymbol{1}}{3} =\frac{5.5}{3}=\frac{11}{6}, $$ and the average eigenvalue in the contrast space is $$ \frac{\operatorname{tr}(S)-11/6}{2} =\frac{3-11/6}{2}=\frac{7}{12}. $$ A CS matrix with these eigenvalues has diagonal \(\{11/6+2(7/12)\}/3=1\) and off-diagonal \(\{11/6-7/12\}/3=5/12\), as claimed.

Result: Under a Gaussian model with known zero mean, the CS maximum-likelihood fit is obtained by matching the empirical covariance in the common direction and in the average contrast direction, provided the resulting eigenvalues are positive. In this example those eigenvalues are \(11/6\) and \(7/12\), so the fit is positive definite and lies inside the CS parameter space.

To see why these directions give the likelihood maximizer, write the CS eigenvalues as \(\lambda_+\) for the one-dimensional common direction and \(\lambda_-\) for the two-dimensional contrast space. Up to a constant, the negative twice log-likelihood divided by \(n\) is the sum of \(\log\lambda\) terms and the corresponding empirical eigenvalue contributions divided by \(\lambda\). For an eigenspace of dimension \(m\) with empirical trace \(t\), its contribution is \(m\log\lambda+t/\lambda\). Its derivative is \(m/\lambda-t/\lambda^2\), which vanishes at \(\lambda=t/m\); the objective decreases before and increases after that value. Applying this separately to the two orthogonal spaces gives \(11/6\) and \(7/12\). Both are positive, so no boundary restriction changes the solution. \(\square\)

Likelihood and Information-Criterion Comparison

For a zero-mean Gaussian model, the negative twice log-likelihood at covariance \(\Sigma\) is $$ -2\ell(\Sigma)=np\log(2\pi)+n\{\log|\Sigma|+\operatorname{tr}(S\Sigma^{-1})\}. $$ The first term is the same for all three candidates here. Define \(Q(\Sigma)=n\{\log|\Sigma|+\operatorname{tr}(S\Sigma^{-1})\}\); smaller \(Q\) means a larger maximized likelihood.

Why the unstructured estimate is \(S\): When \(S\) is positive definite, the Gaussian maximum-likelihood estimate with known mean zero is \(\hat{\Sigma}_{\mathrm{UN}}=S\). To verify it, let \(A=S^{-1/2}\Sigma S^{-1/2}\) have positive eigenvalues \(\lambda_1,\lambda_2,\lambda_3\). Then $$ \log|\Sigma|+\operatorname{tr}(S\Sigma^{-1}) =\log|S|+\sum_{j=1}^3\left(\log\lambda_j+\frac{1}{\lambda_j}\right). $$ For every \(\lambda>0\), the function \(\log\lambda+1/\lambda\) has derivative \((\lambda-1)/\lambda^2\) and its minimum is 1 at \(\lambda=1\). The expression is therefore minimized when every \(\lambda_j=1\), or \(\Sigma=S\). This proves the stated estimate. \(\square\)

Worked Example: Likelihoods, AIC, and BIC

The AR(1) estimate is \(\hat{\sigma}^2=1,\hat{\rho}=0.5\), since its covariance matrix equals \(S\). The unstructured estimate is also \(S\). Their maximized likelihoods are therefore identical. For this matrix, $$ |S|=(1-0.5^2)^2=\frac{9}{16},\qquad \log|S|=2\log(3/4)\approx-0.575364. $$ The trace term is \(\operatorname{tr}(SS^{-1})=3\). Hence \(Q_{\mathrm{AR}}=Q_{\mathrm{UN}}=20(3-0.575364)\approx48.4927\).

For CS, \(\hat{\Sigma}_{\mathrm{CS}}\) has eigenvalues \(11/6,7/12,7/12\), so $$ |\hat{\Sigma}_{\mathrm{CS}}|=\frac{11}{6}\left(\frac{7}{12}\right)^2 =\frac{539}{864},\qquad \log|\hat{\Sigma}_{\mathrm{CS}}|\approx-0.471857. $$ Its trace term is 3. Indeed, its common-direction contribution is \((11/6)/(11/6)=1\), while the contrast-space contribution is \((7/6)/(7/12)=2\). Thus \(Q_{\mathrm{CS}}=20(3-0.471857)\approx50.5629\), about 2.0701 larger than the AR(1) value.

AIC is \(-2\hat{\ell}+2k\), and BIC is \(-2\hat{\ell}+k\log n\), where \(k\) counts estimated parameters. Omitting the common constant \(np\log(2\pi)\), the comparison is:

ModelCovariance parameters \(k\)\(Q\)AIC without common constantBIC without common constant
CS250.562954.562956.5543
AR(1)248.492752.492754.4842
UN648.492760.492766.4671

AR(1) has the smallest AIC and BIC in this comparison. Compared with AR(1), CS loses about 2.0701 units of fit without changing the parameter count. UN gains no likelihood here and pays a penalty of 8 AIC units and \(4\log(20)\approx11.9829\) BIC units for its four additional covariance parameters.

Worked Example: Reading Covariance Residuals

Worked Example: Reading Covariance Residuals

A simple covariance residual matrix is \(R=S-\hat{\Sigma}\). Under CS, the diagonal residuals are zero, while the distinct off-diagonal residuals are $$ R_{12}=0.50-\frac{5}{12}=\frac{1}{12},\quad R_{13}=0.25-\frac{5}{12}=-\frac{1}{6},\quad R_{23}=0.50-\frac{5}{12}=\frac{1}{12}. $$ Thus the observed lag-1 covariances are both above the CS fit, and the observed lag-2 covariance is below it. This is a structured pattern, not three unrelated signs: the data suggest that covariance differs by lag, which CS cannot represent.

Under AR(1), \(\hat{\Sigma}_{\mathrm{AR}}=S\), so \(R\) is the zero matrix. That exact match is expected because the chosen empirical summary itself has AR(1) form. It does not establish that the population covariance is exactly AR(1); it only says that this sample summary has no discrepancy from the fitted AR(1) covariance.

In a real analysis, inspect residual covariance or correlation patterns by measurement pair and lag, and examine subject-level residual behavior as appropriate. Account for the fact that the same data estimated the covariance parameters: residual entries are not independent validation observations. A visual absence of a pattern is useful diagnostic evidence, not a proof of correct specification.

Practical Interpretation and Common Pitfall

A model comparison is clearest when the candidate models use the same observations, response mean structure, and likelihood convention. The example uses a known zero mean specifically so that the displayed \(S\) is the covariance maximum-likelihood summary. If a mean is estimated instead, fit that mean under each covariance model consistently; do not silently combine a restricted-likelihood value for one fit with a full-likelihood value for another. Information-criterion parameter counts must include all parameters estimated under the chosen likelihood convention.

  • Mistake: “UN must be best because it has the largest likelihood.” The unstructured model contains more free parameters and can achieve at least as large a likelihood. Here it ties AR(1), yet its AIC and BIC are worse because it uses four more covariance parameters.
  • Correction: Compare likelihood gains against the penalty for complexity, then check whether the fitted covariance pattern is scientifically interpretable and diagnostically adequate.
  • Mistake: treating a zero residual matrix as proof of a true AR(1) population. The residuals here are zero because the sample covariance was constructed to have that form.
  • Correction: Use residual patterns as a model check, not as a population-level guarantee, and interpret them with sampling variability in mind.
Key takeaway: A defensible structured covariance comparison combines three pieces of evidence: the fitted parameter values, likelihood-based criteria with consistent parameter counts, and residual patterns interpreted in context. In the worked summary, AR(1) captures the observed lag structure parsimoniously; CS misses that pattern, while UN adds parameters without improving fit.

Check Your Understanding

Use the calculations and definitions above to answer these questions.

  1. For the worked covariance summary, why is the CS correlation estimate \(5/12\) rather than the lag-1 sample correlation \(0.50\)?
  2. Verify that the CS eigenvalues \(11/6,7/12,7/12\) imply diagonal entries 1 and off-diagonal entries \(5/12\).
  3. Why do AR(1) and UN have the same maximized likelihood in this example?
  4. How much larger is the UN BIC than the AR(1) BIC after the common likelihood constant is omitted, and what causes the difference?
  5. What lag pattern appears in the CS covariance residuals, and why is it evidence against the CS restriction in this sample?

Previous

Next

Back to Tutorials
Clinical Biostats