Likelihood-Ratio Tests of Nested Covariance Models

Why Test Covariance Constraints?

A Gaussian clustering model can give each component its own covariance matrix, or impose a more economical structure: for example, a common covariance across components, diagonal covariances, or equal component volumes. A likelihood-ratio test asks whether relaxing a specified covariance constraint improves fit enough to justify the additional parameters, relative to a particular reference distribution.

The comparison must be defined precisely. Here the number of components and the rest of the model specification are held fixed. The question is whether the covariance structure can be restricted, not whether the chosen number of clusters is correct. In clustering, that distinction matters: a covariance test is conditional on the model's cluster count and other assumptions.

Definition: Let $M_0$ be a constrained model and $M_1$ a larger model that contains it, so every distribution allowed by $M_0$ is also allowed by $M_1$. Write $\ell_0$ and $\ell_1$ for the maximized log-likelihoods under the two models, using the same observations and likelihood. The likelihood-ratio statistic is $$ \Lambda = 2(\ell_1-\ell_0). $$ The null hypothesis is that the data-generating distribution belongs to $M_0$; the alternative allows the additional covariance freedom in $M_1$.

For observations $x_1,\ldots,x_n$ from a $K$-component Gaussian mixture, the observed-data likelihood has the form

L(\Theta)=\prod_{i=1}^n\left\{\sum_{g=1}^K \pi_g\,\phi_p(x_i;\mu_g,\Sigma_g)\right\},

where $\pi_g$ are mixing proportions, $\mu_g$ are component means, and $\Sigma_g$ are component covariance matrices. The observed-data log-likelihood is the log of this likelihood. Unless a different likelihood is explicitly justified, the test compares maxima of this observed-data likelihood, not a likelihood formed by treating estimated cluster assignments as known labels.

The Statistic and Its Degrees of Freedom

Because $M_0$ is contained in $M_1$, maximizing over $M_1$ cannot give a lower log-likelihood than maximizing over $M_0$. Therefore $\ell_1\geq\ell_0$ and $\Lambda\geq0$. This nonnegativity is an algebraic consequence of nesting; it does not by itself imply that $\Lambda$ has a chi-square distribution.

Under regularity conditions, the reference distribution is approximately chi-square with degrees of freedom equal to the difference in the models' local parameter dimensions. If the only difference is covariance constraints, this is the number of independent covariance parameters released by the alternative. A symmetric $p$-dimensional covariance matrix has $p(p+1)/2$ free entries, but the relevant difference is between the full dimensions of the two specified models, not automatically the number of entries in one matrix.

Theorem (Wilks' likelihood-ratio theorem, nested-model form): Suppose the null model is a smooth, locally identifiable submodel of dimension $d_0$ inside a regular alternative model of dimension $d_1$, with $d_0<d_1$. Assume the true parameter is an interior point of the null model; the observations are independent and identically distributed under a correctly specified model; the Fisher information is finite and nonsingular in local coordinates; and the maximum-likelihood estimators are consistent and have the usual local asymptotic normality expansion. Then, under the null, $$ \Lambda \xrightarrow{d} \chi^2_{d_1-d_0}. $$

The chi-square result is asymptotic: it describes the limiting null distribution as $n$ grows. It does not say that every covariance comparison has this reference, nor that a small-sample test is exact. In a regular finite mixture, the hypotheses must also be identifiable locally, apart from harmless permutations of component labels. Model singularities or parameters on boundaries can invalidate the theorem's conditions.

Why the Chi-Square Reference Arises

The key is that, near a regular true parameter, the log-likelihood is approximately a quadratic function. The unrestricted model can improve the fit over the null model only in directions that the null constraints exclude. Under the null, the standardized score in those directions is asymptotically normal; its squared length has a chi-square distribution.

Derivation: Let $\theta_0$ be the true parameter in the null model. In local coordinates, let $T$ be the $d_1$-dimensional tangent space of the alternative and $T_0$ the $d_0$-dimensional tangent space of the null, with $T_0\subset T$. Write $h=\sqrt n(\theta-\theta_0)$, and let $I$ be the nonsingular Fisher information matrix at $\theta_0$. Local asymptotic normality gives, uniformly over bounded local $h$, $$ \ell_n(\theta_0+h/\sqrt n)-\ell_n(\theta_0) = h^\mathsf{T}Z_n-\tfrac12 h^\mathsf{T}Ih+o_p(1), \qquad Z_n\xrightarrow{d}N(0,I). $$ After the linear change of coordinates $u=I^{1/2}h$, the quadratic part becomes $u^\mathsf{T}W_n-\tfrac12u^\mathsf{T}u$, where $W_n=I^{-1/2}Z_n\xrightarrow{d}N(0,I_{d_1})$. Maximizing this expression over a linear space $S$ gives $\tfrac12\|\operatorname{Proj}_S W_n\|^2$: the maximizer is the orthogonal projection of $W_n$ onto $S$. The transformed null tangent space $S_0$ is contained in the transformed alternative tangent space $S$. Thus twice the difference between the alternative and null maxima converges to $$ \|\operatorname{Proj}_S W\|^2-\|\operatorname{Proj}_{S_0}W\|^2 =\|\operatorname{Proj}_{S\cap S_0^\perp}W\|^2, \qquad W\sim N(0,I_{d_1}). $$ The space $S\cap S_0^\perp$ has dimension $d_1-d_0$. Orthogonal coordinates of a standard normal vector are independent standard normals, so this squared projected length is the sum of $d_1-d_0$ independent squared standard normals, which is $\chi^2_{d_1-d_0}$. The assumed local likelihood expansion and consistency ensure that the model maxima are governed asymptotically by these local quadratic maxima. Hence $\Lambda\xrightarrow{d}\chi^2_{d_1-d_0}$. $\square$

Worked Covariance Comparisons

Worked Example: Shared Versus Component-Specific Covariances

Consider a fixed two-component Gaussian mixture in $p=2$ dimensions. The null model requires $\Sigma_1=\Sigma_2$; the alternative estimates a separate unrestricted covariance matrix for each component. The mixing proportions and means are free in both models. Suppose the fitted observed-data log-likelihoods are $\ell_0=-118.4$ and $\ell_1=-114.9$.

A symmetric $2\times2$ covariance matrix has $2(2+1)/2=3$ parameters. The null has 3 covariance parameters, while the alternative has $2\times3=6$. Since all other parameters are common to the two model specifications, the dimension difference is $6-3=3$.

The likelihood-ratio statistic is $$ \Lambda=2\{-114.9-(-118.4)\}=2(3.5)=7.0. $$ Under the regular chi-square approximation, the reference is $\chi^2_3$, giving an upper-tail probability of approximately $0.072$. This is not strong evidence against a shared covariance at a conventional 5% level, under the stated model and regularity assumptions.

Worked Example: Diagonal Versus Unrestricted Covariances

Now let $K=3$ and $p=3$. Under the null, each component covariance is diagonal; under the alternative, each is unrestricted. Assume the same means and mixing-proportion specification in both models, and suppose $\ell_0=-302.6$ and $\ell_1=-297.4$.

A diagonal $3\times3$ covariance has 3 free variances. An unrestricted symmetric $3\times3$ covariance has $3(4)/2=6$ free entries, so the alternative releases 3 off-diagonal parameters per component. Across three components, the dimension difference is $3\times3=9$.

Thus $$ \Lambda=2\{-297.4-(-302.6)\}=2(5.2)=10.4. $$ If the regularity assumptions hold, compare this with $\chi^2_9$; its upper-tail probability is approximately $0.32$. The statistic does not indicate a large improvement from allowing the off-diagonal covariances, relative to this reference. It does not establish that the diagonal model is true.

Worked Example: Equal Determinants Versus Separate Volumes

For two components in two dimensions, compare unrestricted covariance matrices with a null constraint $\det(\Sigma_1)=\det(\Sigma_2)$. This is an equal-volume restriction: the two covariance ellipses have the same determinant, while their shapes and orientations may differ. Suppose the two maximized log-likelihoods are $\ell_0=-90.7$ under equal determinants and $\ell_1=-87.7$ without that constraint.

The unrestricted covariance model has $2\times3=6$ covariance parameters. Equality of the two determinants is one independent scalar constraint at positive-definite covariance matrices, so the null has local dimension 5 and the difference is 1.

The statistic is $$ \Lambda=2\{-87.7-(-90.7)\}=2(3.0)=6.0. $$ The regular reference is $\chi^2_1$, whose upper-tail probability at 6.0 is approximately $0.0143$. Subject to the assumptions, this provides evidence against the equal-volume constraint for this fixed two-component model. The test concerns the covariance restriction, not whether two clusters are the right representation of the population.

When Standard Calibration Fails

Gaussian mixtures require particular care because their likelihoods can be nonregular. The chi-square degrees-of-freedom rule is not justified merely because the covariance models look nested or because a software package reports a likelihood ratio. It requires the local, identifiable, interior-point setting in the theorem.

Boundary constraints. If a null hypothesis forces a parameter to the boundary of its allowed space, the local parameter space may be a cone rather than a linear space. The limit can then be a mixture of chi-square distributions, or another nonstandard distribution.
Unidentified parameters. If a component has zero mixing proportion, its mean and covariance are not identified by the distribution. A regular dimension difference and ordinary Wilks calibration do not follow.
Coincident components. If two mixture components have identical parameters, different parameter values can describe the same distribution locally. The information matrix may be singular, violating local identifiability.
Covariance collapse. In an unconstrained Gaussian mixture, a component covariance can approach singularity around an observation, making the likelihood unbounded. In that case a finite, well-behaved maximum may not exist without restrictions or regularization.
Small samples or weak separation. Even when the model is formally regular, the asymptotic approximation can be poor in finite samples, especially when components overlap strongly or covariance estimates are unstable.

When the regularity conditions are doubtful, do not present a standard chi-square p-value as guaranteed. A parametric bootstrap under the constrained null can provide a model-based calibration when the null fit and simulation procedure are well-defined: simulate data under the fitted null, refit both models to each simulated data set, and compare the observed statistic with the simulated statistics. This approach still depends on the null model and on reliable optimization; it does not repair an unbounded likelihood or make an unidentified model regular.

Interpretation and a Common Pitfall

A covariance-model likelihood-ratio test asks whether the extra covariance flexibility in a specified alternative yields a sufficiently large gain in maximized fit, judged against a reference distribution. Its conclusion is conditional on the component count, the Gaussian mixture formulation, and the other shared modeling choices. It does not directly measure classification accuracy or prove that the fitted components correspond to distinct populations.

Common mistake: “The alternative has more covariance parameters and a higher log-likelihood, so the likelihood-ratio test must be significant.” The higher likelihood is guaranteed by nesting, and it is not evidence by itself. Calculate $2(\ell_1-\ell_0)$, use the correct dimension difference only when the regularity conditions apply, and assess the statistic against its justified reference distribution. If the mixture is on a boundary, singular, or otherwise nonregular, the ordinary chi-square calibration may be wrong.

A useful report names both covariance hypotheses, states that the comparison holds the cluster count fixed, gives the likelihood-ratio statistic and degrees of freedom, and notes whether the chi-square approximation is defensible. That makes clear both what the test addresses and what its result cannot establish.

Check Your Understanding

Use the hypotheses and regularity conditions in this tutorial to answer the following.

Two nested covariance models have maximized log-likelihoods $-150.2$ and $-146.8$, with the latter from the larger model. Calculate the likelihood-ratio statistic.
For two components in four dimensions, how many covariance parameters are released when moving from a shared unrestricted covariance to separate unrestricted covariances?
Why does a larger maximized log-likelihood under the alternative not, by itself, imply a statistically significant improvement?
Name two mixture-model features that can invalidate the ordinary chi-square reference.
What does a covariance-model likelihood-ratio test say about the number of clusters, and what does it leave unanswered?