One Dataset, Three Different Covariance Questions
A covariance matrix describes both the scale of each variable and how variables vary together. Consequently, “the covariance matrix has a particular structure” can mean several different things. Testing a fully specified matrix asks whether both its variances and covariances match fixed values. Testing sphericity asks whether the matrix is a scalar multiple of the identity. Testing diagonality allows different variances but requires zero covariances. These hypotheses constrain different features, so their tests can reach different conclusions on the same data.
We will analyze one multivariate Gaussian sample under all three null hypotheses. The calculations use the Gaussian likelihood with an unknown mean and the likelihood-ratio statistic. The chi-square reference distributions below are asymptotic; they are not exact finite-sample calibrations.
The Likelihood-Ratio Calculation
For a covariance restriction \(H\), let \(\hat\Sigma_H\) be its maximum-likelihood estimate under that restriction. The likelihood-ratio statistic is \(\Lambda=L_H/L_{\mathrm{full}}\), where each likelihood is maximized over the mean as well as over the covariance parameters allowed by that hypothesis.
To verify the result, the log-likelihood, apart from a constant, is $$ \ell(\mu,\Sigma)=-\frac{n}{2}\log|\Sigma| -\frac{1}{2}\sum_{i=1}^n(X_i-\mu)'\Sigma^{-1}(X_i-\mu). $$ For any fixed positive-definite \(\Sigma\), the identity $$ \sum_{i=1}^n(X_i-\mu)(X_i-\mu)' =\sum_{i=1}^n(X_i-\bar X)(X_i-\bar X)' +n(\bar X-\mu)(\bar X-\mu)' $$ shows that the quadratic term is minimized at \(\mu=\bar X\). At that mean, the log-likelihood divided by \(-n/2\), again up to a constant, is \(\log|\Sigma|+\operatorname{tr}(\Sigma^{-1}S)\).
The unrestricted covariance estimate is \(S\). To see it is the global maximizer, let \(\lambda_1,\ldots,\lambda_p\) be the positive eigenvalues of \(S^{-1/2}\Sigma S^{-1/2}\). Then $$ \log|\Sigma|+\operatorname{tr}(\Sigma^{-1}S) =\log|S|+\sum_{j=1}^p\left(\log\lambda_j+\frac{1}{\lambda_j}\right). $$ For \(g(\lambda)=\log\lambda+1/\lambda\), \(g'(\lambda)=(\lambda-1)/\lambda^2\), so \(g\) has its global minimum at \(1\). Thus the expression is minimized when every \(\lambda_j=1\), which gives \(\Sigma=S\). Subtracting the restricted maximized log-likelihood from the unrestricted one now gives the stated likelihood-ratio formula. \(\square\)
Worked Example: A Sample Covariance Matrix
Worked Example: Setting Up the Shared Dataset
Suppose \(n=10\) observations of \(p=3\) variables have sample mean \(\bar X=(0,0,0)'\) and maximum-likelihood sample covariance $$ S=\begin{pmatrix}4&1&0\\1&1&0.25\\0&0.25&0.25\end{pmatrix}. $$ These summaries are possible for a ten-observation sample. For example, take three orthonormal, mean-zero vectors \(q_1,q_2,q_3\) in \(\mathbb R^{10}\), place them as columns of \(Q\), and choose a matrix \(L\) with \(LL'=S\). The ten rows of \(X=\sqrt{10}\,Q L'\) have mean zero and satisfy \(X'X/10=S\).
The covariance is positive definite: its leading principal minors are \(4\), \(4(1)-1^2=3\), and $$ |S|=4(0.25-0.25^2)-0.25=0.5, $$ all of which are positive. Its trace is \(4+1+0.25=5.25\), and its diagonal product is \(4(1)(0.25)=1\). These are the quantities needed for the three tests.
Test 1: A Fully Specified Covariance
First test the point null \(H_0:\Sigma=I_3\), where \(I_3\) is the \(3\times3\) identity matrix. Under this null, the fitted covariance is fixed at \(I_3\). This is a more specific claim than merely requiring zero covariances: it also specifies every marginal variance to be one.
Worked Example: Testing the Identity Covariance
For \(\hat\Sigma_0=I_3\), the determinant is \(1\), and $$ \operatorname{tr}(I_3^{-1}S)=\operatorname{tr}(S)=5.25. $$ Substitution into the likelihood-ratio formula gives $$ -2\log\Lambda =10\{0-\log(0.5)+5.25-3\} =10(\log 2+2.25) \approx 29.432. $$
The unrestricted covariance has six distinct entries, while this null fixes all six. Under the null and standard regularity conditions, Wilks’ theorem gives an asymptotic \(\chi^2_6\) reference distribution. Its approximate upper-tail probability is about \(0.00005\), so at the 5% level this test rejects \(H_0:\Sigma=I_3\). This conclusion does not isolate the reason for disagreement: the sample variances differ from one, and the sample also has nonzero off-diagonal entries.
Test 2: Sphericity
Sphericity is the null \(H_0:\Sigma=\sigma^2 I_p\) for some unknown \(\sigma^2>0\). Unlike the specified-covariance null, it does not fix the common variance in advance. Under this restriction, the fitted covariance is $$ \hat\Sigma_0=\hat\sigma^2 I_p,\qquad \hat\sigma^2=\frac{\operatorname{tr}(S)}{p}. $$ Indeed, the covariance-dependent objective to minimize is \(p\log\sigma^2+\operatorname{tr}(S)/\sigma^2\). Its derivative is \(p/\sigma^2-\operatorname{tr}(S)/(\sigma^2)^2\), which vanishes at the stated value; the objective decreases before that value and increases after it.
At this estimate, \(\operatorname{tr}(\hat\Sigma_0^{-1}S)=p\), so the trace terms in the general likelihood-ratio formula cancel. Therefore the sphericity statistic is $$ -2\log\Lambda =n\left\{p\log\left(\frac{\operatorname{tr}(S)}{p}\right)-\log|S|\right\}. $$
Worked Example: Testing Sphericity
For the shared dataset, \(\hat\sigma^2=5.25/3=1.75\), so the restricted estimate is \(1.75I_3\). Its determinant is \(1.75^3=5.359375\). Thus $$ -2\log\Lambda =10\{3\log(1.75)-\log(0.5)\} \approx 10(2.37199) =23.720. $$
The unrestricted covariance has six distinct entries and the spherical family has one parameter, so the difference in dimensions is \(6-1=5\). The asymptotic reference distribution is \(\chi^2_5\); the approximate upper-tail probability is about \(0.00025\). The sphericity null is rejected at the 5% level. The sample’s unequal diagonal entries alone are inconsistent with a common variance, and the nonzero sample covariance between variables 1 and 2 is also inconsistent with sphericity.
Test 3: Diagonal Covariance
The diagonal null \(H_0:\Sigma\text{ is diagonal}\) permits each variable to have its own variance but requires all covariances between different variables to be zero. The restricted maximum-likelihood estimate is \(\hat\Sigma_0=\operatorname{diag}(s_{11},\ldots,s_{pp})\). To obtain each diagonal entry, maximize the likelihood over that coordinate’s variance; its estimate is the corresponding diagonal entry of \(S\). At this fitted matrix, \(\operatorname{tr}(\hat\Sigma_0^{-1}S)=p\), giving $$ -2\log\Lambda =n\left\{\sum_{j=1}^p\log(s_{jj})-\log|S|\right\}. $$
Worked Example: Testing a Diagonal Covariance
For this dataset, the fitted diagonal covariance is \(\operatorname{diag}(4,1,0.25)\), with determinant \(1\). Therefore $$ -2\log\Lambda =10\{\log(4)+\log(1)+\log(0.25)-\log(0.5)\} =10\log 2 \approx 6.931. $$
A symmetric \(3\times3\) covariance matrix has six distinct entries, and a diagonal covariance has three free variances. The difference is three, the number of off-diagonal entries being constrained to zero. The asymptotic reference distribution is \(\chi^2_3\), with an approximate upper-tail probability of \(0.074\). At the 5% level, this test does not reject the diagonal-covariance null.
Comparing the Conclusions
The three asymptotic test results can be summarized at the 5% level as follows. The likelihood-ratio statistics use the same observations, but each statistic measures departure from a different null and has its own degrees of freedom.
| Null hypothesis | Restriction being tested | Statistic | Reference degrees of freedom | Conclusion at 5% |
|---|---|---|---|---|
| \(\Sigma=I_3\) | All variances and covariances fixed | 29.432 | 6 | Reject |
| \(\Sigma=\sigma^2I_3\) | Equal variances and zero covariances | 23.720 | 5 | Reject |
| \(\Sigma\) diagonal | Zero covariances; variances unrestricted | 6.931 | 3 | Do not reject |
These findings are compatible. The diagonal test asks whether there is enough evidence against zero covariances, without treating the unequal sample variances as a violation. The sphericity test additionally requires equal variances, and the identity test additionally fixes their common value at one. The strong evidence against those more restrictive structures does not imply that the diagonal restriction must also be rejected.
Under regularity conditions, Wilks’ theorem states that \(-2\log\Lambda\) converges in distribution to a chi-square variable whose degrees of freedom equal the difference between the dimensions of the unrestricted and restricted parameter spaces. Here the mean is unrestricted in all models, so it contributes no difference in dimension. The covariance dimensions are \(p(p+1)/2\) unrestricted, \(1\) for sphericity, \(p\) for diagonality, and \(0\) for a fully specified covariance. This gives the degrees of freedom used above.
The theorem is an asymptotic result: the observations must be independent and identically distributed under the stated Gaussian model, the true parameter must satisfy the null, and the parameterization must be regular with the null parameter in the interior of its parameter space. With only ten observations, the chi-square p-values are illustrative rather than a guarantee of accurate finite-sample calibration. The numerical conclusions should therefore be understood as conclusions from the stated asymptotic tests.
Common Pitfall: Treating the Three Nulls as Equivalent
A common mistake is to reason that if a specified covariance is rejected, every structural covariance null must be rejected as well. That is incorrect: rejecting \(\Sigma=I_3\) may reflect incorrect variances even when zero covariances remain plausible. Another mistake is to use the sphericity statistic for the diagonal null. Sphericity also imposes equal variances, so it answers a stronger question. Match the restricted maximum-likelihood estimate and the degrees of freedom to the precise null being tested.
A further computational pitfall is to insert an unbiased sample covariance with divisor \(n-1\) into formulas derived here and still multiply by \(n\). If \(S_{n-1}\) uses divisor \(n-1\), first convert to the likelihood estimate \(S=(n-1)S_{n-1}/n\). Otherwise the determinant and trace terms do not correspond to the likelihood being compared.
Check Your Understanding
Use the definitions and calculations above to answer the following questions.
- Why does testing a fully specified \(3\times3\) covariance have six asymptotic degrees of freedom when the mean is unknown?
- What covariance estimate is fitted under sphericity, and why is it not generally the same as the identity matrix?
- Which entries of a covariance matrix are constrained by the diagonal null, and how many such constraints are there when \(p=3\)?
- How can the sphericity null be rejected while the diagonal null is not rejected?
- Why should the chi-square p-values for the ten-observation example be interpreted cautiously?