Likelihood-Ratio Tests for Nested Gaussian Models

Comparing a Gaussian Model With a Restricted Version

A Gaussian model may allow a mean vector and covariance matrix to vary freely, while a scientifically motivated model imposes structure: a mean is fixed, covariance is diagonal, or all marginal variances are equal. A likelihood-ratio test compares the best fit permitted by the restriction with the best fit in the larger model. The ratio records how much fit is lost when the restriction is imposed; its large-sample null distribution gives a reference for judging that loss.

Definition: Let \(\Theta_0\subseteq\Theta\) be a restricted parameter space nested inside an unrestricted parameter space. For observed data \(x\), define \(L_0=\sup_{\theta\in\Theta_0}L(\theta)\) and \(L_1=\sup_{\theta\in\Theta}L(\theta)\). When \(L_1>0\), the likelihood ratio is \(\Lambda=L_0/L_1\), and the likelihood-ratio statistic is \(T=-2\log\Lambda=2\{\ell_1-\ell_0\}\), where \(\ell_j=\log L_j\). If \(L_0=0\), then \(\Lambda=0\) and \(T=+\infty\). In regular Gaussian models with positive, finite maximized likelihoods, \(0<\Lambda\leq1\) and \(T\) is finite.

The inequality follows directly from nesting: every parameter value allowed under the restricted model is also allowed under the unrestricted model, so \(L_0\leq L_1\). Thus the restricted model cannot have a larger maximized likelihood. The statistic \(T\) is nonnegative when finite, and larger values indicate a greater loss of fit under the restriction. Nesting alone does not guarantee \(L_0>0\); that positivity is ensured in the regular Gaussian examples below, not by the definition itself.

Parameter Counts and the Null Reference Distribution

Suppose the unrestricted model has \(d\) free parameters and the restricted model is a smooth submodel of dimension \(d_0\), with the true null parameter in its interior. The effective number of restrictions is \(q=d-d_0\). Equivalently, \(q\) is the codimension of the restricted model. Count independent restrictions, not merely the number of equations used to write them: redundant equations do not reduce the dimension a second time.

Theorem (Wilks' theorem): Consider a regular, identifiable, finite-dimensional model based on independent and identically distributed observations. Suppose the true parameter \(\theta_0\) is an interior point of a smooth restricted model of dimension \(d_0\), itself contained in a smooth unrestricted model of dimension \(d\). Assume differentiability and the usual conditions for a nonsingular Fisher information matrix, a consistent root-\(n\) maximum-likelihood estimator, and a uniform local quadratic expansion of the log-likelihood. With \(q=d-d_0\) independent restrictions, under the null hypothesis \(T\xrightarrow{d}\chi^2_q\) as \(n\to\infty\).

The result is asymptotic, not a claim that \(T\) has exactly a chi-square distribution for a finite sample. Its justification comes from the local shape of the log-likelihood near the true parameter. Write the per-sample Fisher information at \(\theta_0\) as \(I\), assumed positive definite. For local displacements \(h/\sqrt n\), the regular quadratic expansion is

\ell_n(\theta_0+h/\sqrt n)-\ell_n(\theta_0) =h^\mathsf{T}Z_n-\frac12h^\mathsf{T}Ih+o_p(1), \qquad Z_n\xrightarrow{d}N_d(0,I).

Here \(Z_n\) is the log-likelihood score at \(\theta_0\), divided by \(\sqrt n\). The expansion is uniform for bounded \(h\); the assumed root-\(n\) behavior of the unrestricted and restricted estimators permits the maximizations to be localized to such bounded displacements. To compare the two maxima, use coordinates \(v=I^{1/2}h\). In these coordinates the quadratic part is \(v^\mathsf{T}W_n-\frac12\|v\|^2\), where \(W_n=I^{-1/2}Z_n\xrightarrow{d}N_d(0,I_d)\). The restricted model's local tangent space becomes a linear subspace \(H\) of dimension \(d_0\).

For any linear subspace \(A\), the maximum of \(v^\mathsf{T}W_n-\frac12\|v\|^2\) over \(v\in A\) is \(\frac12\|P_AW_n\|^2\), where \(P_A\) is orthogonal projection onto \(A\). This follows by writing the objective as \(\frac12\|P_AW_n\|^2-\frac12\|v-P_AW_n\|^2\) for \(v\in A\). Therefore twice the difference between the unrestricted and restricted maxima is

\|W_n\|^2-\|P_HW_n\|^2 =\|P_{H^\perp}W_n\|^2 \xrightarrow{d}\chi^2_q.

The last equality in distribution holds because \(H^\perp\) has dimension \(q\), and the projection of a standard multivariate normal vector onto any fixed \(q\)-dimensional subspace has \(q\) independent standard normal coordinates. Their squared sum has a \(\chi^2_q\) distribution. The local quadratic approximation and the assumed regularity conditions transfer this limiting result to \(T\). This proves Wilks' calibration. \(\square\)

Worked Gaussian Comparisons

For an \(N_p(\mu,\Sigma)\) sample, the unrestricted Gaussian model has \(p\) mean parameters and \(p(p+1)/2\) covariance parameters. The Gaussian log-likelihood, up to its constant term, is

\ell(\mu,\Sigma) =-\frac n2\log|\Sigma| -\frac12\sum_{i=1}^n(x_i-\mu)^\mathsf{T}\Sigma^{-1}(x_i-\mu).

The Gaussian maximum-likelihood estimates used below are the unrestricted mean \(\bar{x}\) and covariance \(S=n^{-1}\sum_i(x_i-\bar{x})(x_i-\bar{x})^\mathsf{T}\), when the required covariance MLE exists. In particular, these calculations assume positive-definite scatter matrices. For any two nested models with attained, positive maxima, subtraction of their maximized log-likelihoods gives \(T=2(\ell_1-\ell_0)\).

Worked Example: Testing a Fixed Mean With Unknown Covariance

Compare the unrestricted model, with \(\mu\) and positive-definite \(\Sigma\) unknown, to the restriction \(\mu=\mu_0\), leaving \(\Sigma\) unrestricted. For \(p=2\), suppose \(n=20\), \(S=\begin{pmatrix}4&1\\1&9\end{pmatrix}\), and \(\bar{x}-\mu_0=(1,2)^\mathsf{T}\).

Let \(d=\bar{x}-\mu_0\). Under the restricted mean, the covariance MLE is \(S_0=n^{-1}\sum_i(x_i-\mu_0)(x_i-\mu_0)^\mathsf{T}=S+dd^\mathsf{T}\). In this case, \(dd^\mathsf{T}=\begin{pmatrix}1&2\\2&4\end{pmatrix}\), so \(S_0=\begin{pmatrix}5&3\\3&13\end{pmatrix}\).

At either covariance MLE, the quadratic term in the log-likelihood equals \(np\). The maximized likelihood ratio is consequently \(\Lambda=(|S|/|S_0|)^{n/2}\). The determinants are \(|S|=4(9)-1=35\) and \(|S_0|=5(13)-9=56\). Thus \(T=n\log(|S_0|/|S|)=20\log(56/35)=20\log(1.6)\approx9.4001\).

The unrestricted model has \(2+3=5\) parameters and the restricted model has \(3\), so \(q=2\). Under the null, the large-sample reference is \(\chi^2_2\); its upper-tail probability at \(9.4001\) is approximately \(\exp(-9.4001/2)=0.0091\). This is an asymptotic calibration, not an exact finite-sample p-value.

Worked Example: Testing a Diagonal Covariance

Suppose the mean is unrestricted in both models, but the restricted model requires the covariance to be diagonal. Take \(p=2\), \(n=30\), and the unrestricted covariance estimate \(S=\begin{pmatrix}4&1\\1&9\end{pmatrix}\). Under the diagonal restriction, the covariance estimate is \(\operatorname{diag}(4,9)\): the fitted marginal variances are unchanged, while the covariance is set to zero.

The trace term in the Gaussian log-likelihood is \(np\) at either covariance maximum. Thus the ratio of maximized likelihoods is the \(n/2\) power of the full covariance determinant divided by the restricted determinant. Here the determinants are \(35\) and \(4(9)=36\), respectively, giving \(T=n\log(36/35)=30\log(36/35)\approx30(0.0281709)=0.8451\).

A symmetric \(2\times2\) covariance has three free parameters, whereas a diagonal one has two. The unrestricted and restricted means contribute the same two parameters and cancel in the difference, so \(q=1\). Wilks' theorem gives the large-sample null reference \(\chi^2_1\).

Worked Example: Testing Spherical Covariance

Now compare an unrestricted covariance matrix with the spherical restriction \(\Sigma=\sigma^2 I_3\), keeping the mean unrestricted in both models. Let \(p=3\), \(n=30\), and suppose \(S\) has eigenvalues \(1,2,3\). Its determinant is \(6\) and its trace is \(6\).

Under the spherical restriction, the covariance MLE is \(\hat{\sigma}^2=\operatorname{tr}(S)/3=2\). The restricted determinant is \(|2I_3|=8\). Because the fitted quadratic term is \(np\) in each model, the likelihood-ratio statistic is \(T=30\log(8/6)=30\log(4/3)\approx30(0.2876821)=8.6305\).

The unrestricted covariance has \(3(4)/2=6\) parameters; a spherical covariance has one. The mean parameters are common to both models, so the parameter-count difference is \(q=6-1=5\). The asymptotic null calibration is \(\chi^2_5\).

Using the Calibration Carefully

For a test with \(q\) restrictions, an asymptotic level-\(\alpha\) rule rejects for large \(T\): specifically, reject when \(T\) exceeds the \(1-\alpha\) quantile of \(\chi^2_q\). Equivalently, a large-sample p-value is the upper-tail probability \(P(\chi^2_q\geq T_{\mathrm{obs}})\). This procedure is useful because it turns a comparison of fitted Gaussian likelihoods into a common scale that accounts for how many independent restrictions were imposed.

The regularity conditions matter. The null value must be an interior point of a smooth model, and the information must be nonsingular. If a parameter is on a boundary, the model is not identifiable, or a covariance estimate is singular, the ordinary chi-square result may not apply. In those cases, one cannot justify the usual calibration merely by counting parameters; a separate distributional argument is needed.

Common Pitfall

Using the wrong likelihood-ratio direction. Nesting gives \(L_0\leq L_1\), so use \(\Lambda=L_0/L_1\) and \(T=-2\log\Lambda\). Reversing the ratio makes the statistic nonpositive.
Counting constraints instead of independent restrictions. The degrees of freedom are the difference in model dimensions. Redundant constraints do not increase \(q\).
Treating Wilks' theorem as an exact finite-sample law. The chi-square reference is a large-sample approximation under stated regularity conditions.
Assuming nesting forces a positive restricted likelihood. It only ensures \(L_0\leq L_1\). If \(L_1>0\) but \(L_0=0\), then \(T=+\infty\); the usual finite statistic requires positive maximized likelihoods.

Key takeaway: Form the statistic from the restricted and unrestricted maxima, \(T=2(\ell_1-\ell_0)\). For a smooth, regular null model, its large-sample degrees of freedom are the difference in parameter dimensions, and its null reference distribution is \(\chi^2_q\).

Check Your Understanding

Use the definitions and regularity conditions above to check each step of a likelihood-ratio comparison.

If the unrestricted and restricted models have dimensions 9 and 6, respectively, what is the Wilks degrees of freedom, assuming the restrictions are regular?
Why does nesting imply \(T\geq0\) when the maxima are positive and finite?
In the diagonal-covariance example, why do the mean parameters not contribute to the parameter-count difference?
What does the spherical covariance restriction require, and how many covariance parameters does it leave when \(p=3\)?
Name two conditions under which the ordinary chi-square calibration may fail.