Likelihood Comparisons for Covariance Structures

Why Compare Covariance Structures by Likelihood?

In model-based clustering, a covariance structure determines how a fitted Gaussian component can spread, stretch, and orient. A restrictive structure—such as one covariance shared across components—uses fewer parameters. A more flexible structure may fit the data better, but that improvement alone does not establish that the flexibility is worthwhile. Likelihood comparisons make the improvement measurable; parameter counts help describe what was spent to obtain it.

The comparison must be made between models fitted to the same observations under the same likelihood convention. In a Gaussian mixture with \(K\) components, the observed-data likelihood is the product of mixture densities, with each density summing over the unobserved component memberships. If memberships are treated as known, the corresponding calculation is a classified likelihood instead. These are different likelihoods, so their values should not be compared as though they were interchangeable.

Definition: Two models are nested when every probability distribution allowed by the smaller model is also allowed by the larger model. For a covariance comparison, the models must use the same observations and likelihood, and must impose the same restrictions on other parameters—such as component means, mixing proportions, and the number of components—unless those differences are explicitly part of the comparison. The larger model relaxes one or more covariance restrictions.

Write \(\ell_R\) for the maximized log-likelihood under a restricted covariance model and \(\ell_F\) for the maximized log-likelihood under a less restricted model. The likelihood ratio in favor of the less restricted model is \(L_F/L_R=\exp(\ell_F-\ell_R)\). A ratio greater than one records improved fit, but is not by itself a penalty-adjusted decision.

What Nesting Guarantees—and What It Does Not

Theorem: Suppose the restricted parameter set \(\Theta_R\) is contained in the larger parameter set \(\Theta_F\), and both maximized likelihoods exist and are finite. Then \(\ell_F\geq \ell_R\), or equivalently \(L_F\geq L_R\).

Proof. For every \(\theta\in\Theta_R\), inclusion gives \(\theta\in\Theta_F\). Therefore the supremum of \(L(\theta)\) over \(\Theta_F\) is at least the supremum over \(\Theta_R\):

\sup_{\theta\in\Theta_F} L(\theta) \geq \sup_{\theta\in\Theta_R} L(\theta).

Taking logarithms preserves the inequality because the logarithm is increasing on positive numbers. If the maxima exist, the suprema are the stated maximized likelihoods. \(\square\)

This is a set-inclusion fact, not evidence that the larger model is preferable. It also does not provide a reference distribution for a likelihood-ratio test. In mixture models, latent memberships, label symmetry, and parameters on boundaries can make familiar regular-model approximations inapplicable. Establishing the appropriate inferential calibration is a separate question.

A Gaussian Likelihood Result for Covariance Comparisons

A useful calculation is available when the component memberships and means are fixed or known. Suppose \(x_1,\ldots,x_n\) are independent \(p\)-dimensional observations from \(N_p(\mu,\Sigma)\), with known \(\mu\), and define the scatter matrix \(S=\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^\mathsf{T}\). Assume \(S\) is positive definite. Apart from terms not depending on \(\Sigma\), the log-likelihood is

\ell(\Sigma) = -\frac{n}{2}\left\{\log|\Sigma|+ \operatorname{tr}(S\Sigma^{-1}/n)\right\}+\text{constant}.

Theorem: Under these conditions, the unrestricted positive-definite covariance maximum-likelihood estimate is \(\hat{\Sigma}=S/n\).

Proof. Let \(T=S/n\), which is positive definite by assumption, and write \(\Sigma=T^{1/2}BT^{1/2}\), where \(B\) is positive definite. Then \(\log|\Sigma|=\log|T|+\log|B|\), and \(\operatorname{tr}(T\Sigma^{-1})=\operatorname{tr}(B^{-1})\). If \(b_1,\ldots,b_p\) are the positive eigenvalues of \(B\), the part of the expression to be minimized is

\log|\Sigma|+\operatorname{tr}(T\Sigma^{-1}) = \log|T|+\sum_{j=1}^p\left(\log b_j+\frac{1}{b_j}\right).

For \(b>0\), the derivative of \(g(b)=\log b+1/b\) is \((b-1)/b^2\). Thus \(g\) decreases up to \(b=1\), increases after \(b=1\), and has its unique minimum there. The expression is minimized when every eigenvalue of \(B\) is one, so \(B=I\) and \(\Sigma=T=S/n\). Since the log-likelihood is the negative of this expression times \(n/2\), this gives the maximum. \(\square\)

When a covariance is constrained, its maximum-likelihood estimate must instead be found within that constrained family. In the examples below, the constraints are simple enough to give the estimates directly. The likelihood comparisons use the same Gaussian density and differ only in the permitted covariance structure.

Worked Covariance Comparisons

Worked Example: One Covariance Shared Across Two Groups

Consider two groups of \(n_1=n_2=10\) bivariate observations with known mean zero in each group. Their scatter matrices are \(S_1=10I\) and \(S_2=40I\), where \(I\) is the two-dimensional identity matrix. Compare separate unrestricted covariance matrices with one covariance shared across both groups.

With separate covariances, the estimates are \(\hat{\Sigma}_1=S_1/10=I\) and \(\hat{\Sigma}_2=S_2/10=4I\). Under the shared restriction, the pooled estimate is \(\hat{\Sigma}=(S_1+S_2)/20=2.5I\). The separate model has \(2\times 3=6\) covariance parameters; the shared model has 3. Thus the separate model adds 3 covariance parameters.

For a Gaussian group with known mean, the covariance-dependent log-likelihood is \(-n_g\log|\Sigma_g|/2-\operatorname{tr}(S_g\Sigma_g^{-1})/2\). At each estimate, the trace contribution is \(-p n_g/2\) in either model, so those terms cancel in the difference. Since \(|I|=1\), \(|4I|=16\), and \(|2.5I|=6.25\), the log-likelihood gain is

\ell_F-\ell_R = 10\log(6.25)-5\log(1)-5\log(16) = 10\log\left(\frac{6.25}{4}\right) = 10\log(1.5625) \approx 4.4629.

Therefore the separate-covariance likelihood is \(\exp(4.4629)\), or equivalently \(1.5625^{10}\approx 86.7\), times the shared-covariance likelihood. The arithmetic mean of the two covariance estimates is not being compared with their geometric mean by accident: the determinant terms in the Gaussian likelihood produce this particular ratio. The large likelihood ratio describes a substantial fit gain for this constructed data set; it does not erase the cost of the three extra parameters.

Worked Example: Full Versus Diagonal Covariance

Suppose \(n=8\) bivariate observations have known mean zero and scatter matrix \(S=\begin{pmatrix}16&8\\8&16\end{pmatrix}\). Compare an unrestricted covariance with a diagonal covariance. The full estimate is \(\hat{\Sigma}_F=S/8=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), whose determinant is \(3\). Under the diagonal restriction, the diagonal variances are the sample second moments \(16/8=2\) and \(16/8=2\), so \(\hat{\Sigma}_R=\operatorname{diag}(2,2)\), with determinant \(4\). Allowing the off-diagonal entry adds one covariance parameter.

At either estimate, the trace term \(\operatorname{tr}(S\hat{\Sigma}^{-1})\) equals \(np=16\). For the full estimate, \(\hat{\Sigma}_F=S/n\), so the trace is 16. For the diagonal estimate, its inverse is \(\operatorname{diag}(1/2,1/2)\), giving \(\operatorname{tr}(S\hat{\Sigma}_R^{-1})=8+8=16\). The trace terms cancel. Hence

\ell_F-\ell_R = \frac{8}{2}\log\left(\frac{|\hat{\Sigma}_R|}{|\hat{\Sigma}_F|}\right) = 4\log(4/3) \approx 1.1507, \qquad \frac{L_F}{L_R} = (4/3)^4 = \frac{256}{81} \approx 3.1605.

The full covariance fits better, as nesting requires. Here that gain comes from estimating one covariance that the diagonal model fixes at zero: the off-diagonal covariance. The example shows how the determinant-based likelihood comparison can isolate the fit contribution of a particular covariance relaxation.

Worked Example: Separate Versus Common Variances and an AIC Tradeoff

Now consider two groups of ten univariate observations each. Treat group membership as known, and estimate a separate mean for each group under both models. Suppose the within-group sums of squared deviations from those means are \(S_1=8\) and \(S_2=32\). The separate variance estimates are \(8/10=0.8\) and \(32/10=3.2\). A common variance estimate is \((8+32)/20=2\). The separate model adds one variance parameter.

The maximized normal log-likelihoods share the same constants and the same total residual contribution: each model has a residual contribution of \(-20/2=-10\). The separate model’s log-determinant contribution is \(-5\log(0.8)-5\log(3.2)\); the common model’s is \(-10\log(2)\). Therefore

\ell_F-\ell_R = 10\log(2)-5\log(0.8)-5\log(3.2) = 10\log\left(\frac{2}{\sqrt{0.8(3.2)}}\right) = 10\log(1.25) \approx 2.2314.

The likelihood ratio is \(1.25^{10}\approx 9.3132\). To illustrate how a parameter penalty can change the comparison, the standard AIC difference, separate minus common, is \(-2(\ell_F-\ell_R)+2(1)\approx-2.4629\), so AIC favors the separate-variance model here. The likelihood gain alone says only that the separate model fits better; the AIC result additionally charges it for its one extra parameter. This calculation is for the stated classified likelihood and should not be silently substituted for an observed-data mixture comparison.

From Covariance Gain to Model Judgment

For a \(p\)-dimensional unrestricted covariance matrix, the number of distinct covariance parameters is \(q=p(p+1)/2\): there are \(p\) variances and \(p(p-1)/2\) distinct covariances. If a constraint replaces \(q_R\) free covariance parameters by \(q_F\), its added covariance dimension is \(q_F-q_R\). For example, replacing one shared unrestricted covariance across \(K\) groups by \(K\) separate unrestricted covariances adds \((K-1)p(p+1)/2\) parameters, provided no other parameter restrictions change.

This count helps explain the flexibility gained, but it does not determine whether that gain is worth paying for. A likelihood ratio grows with sample size and can be large even when the practical improvement is modest. Conversely, a small sample may not provide much evidence for a real covariance difference. Information criteria such as AIC and BIC, predictive assessment, and substantive interpretability address different parts of this judgment; they are not replaced by simply choosing the model with the largest likelihood.

In a Gaussian mixture, an additional technical condition matters: the maximized likelihood must be finite for the comparison to be meaningful. An unconstrained mixture can have a likelihood that becomes unbounded when a component covariance collapses around an observation. Implementations therefore commonly impose constraints or otherwise ensure a well-defined fit. Also check that both fits use the same observations, the same treatment of missing data, and the same likelihood convention. A numerical difference caused by changing any of these is not a covariance-structure comparison.

Common Mistakes

Calling a likelihood increase proof of a better model. A nested larger model cannot fit worse at its optimum, but it has more flexibility. Compare its gain with the added parameter cost using an appropriate selection or validation procedure.
Comparing likelihoods from different data or conventions. Observed-data mixture likelihoods sum over latent memberships; classified likelihoods condition on assignments. Keep the likelihood definition fixed throughout a comparison.
Counting every covariance entry twice. A covariance matrix is symmetric. Its unrestricted parameter count is \(p(p+1)/2\), not \(p^2\).
Assuming the usual likelihood-ratio reference distribution automatically applies. Nesting establishes the direction of the maximized likelihood inequality, not the sampling distribution of a test statistic. Mixture boundaries and latent labels require care.
Reporting an exponentiated log-likelihood difference inaccurately. First compute \(\ell_F-\ell_R\); then exponentiate it, or simplify the likelihood ratio algebraically. For example, \(10\log(1.5625)\) exponentiates to \(1.5625^{10}\approx86.7\), not to a ratio obtained by rounding the logarithm too early.

Key takeaway: For nested covariance structures fitted to the same likelihood, the less constrained model has at least as large a maximized likelihood. The useful comparison is the size of that gain alongside the number and meaning of the added covariance parameters—not the likelihood gain in isolation.

Check Your Understanding

For each question, distinguish the mathematical guarantee from the model-selection judgment.

Why can a nested covariance model not have a lower maximized likelihood than its restricted counterpart?
How many free parameters does an unrestricted covariance matrix have in dimension \(p=4\)?
In a shared-versus-separate covariance comparison with \(K=3\) groups and dimension \(p=2\), how many covariance parameters are added?
Why must an observed-data mixture likelihood not be compared directly with a classified likelihood as if they were the same objective?
What does a likelihood ratio greater than one establish, and what does it not establish by itself?