Wilks Approximation and Gaussian Likelihood Curvature

Likelihood-Ratio Statistics and Local Curvature

A likelihood-ratio test compares the best fit available under a null hypothesis with the best fit in the full model. Its statistic measures the loss in fit caused by the restriction. In a regular model, a sufficiently large sample makes the log-likelihood look nearly quadratic around the true parameter. That local curvature is what turns the loss in fit into a chi-square statistic.

The approximation is useful because its degrees of freedom have a clear interpretation: they are the number of locally independent restrictions imposed by the null. But this conclusion is not automatic. It relies on an interior true parameter, identifiable and nonsingular local curvature, and a smooth null restriction. Boundaries, unidentified parameters, or singular curvature can change the limiting distribution.

Definition: Let $\ell_n(\theta)$ be the log-likelihood from $n$ independent observations, with parameter $\theta\in\mathbb{R}^d$. For a null hypothesis $H_0$, define the likelihood-ratio statistic $$ W_n=2\left\{\sup_{\theta\in\Theta}\ell_n(\theta)-\sup_{\theta\in\Theta_0}\ell_n(\theta)\right\}. $$ The factor of 2 is conventional. Under regular conditions, $W_n$ is approximately chi-square under the null, with degrees of freedom equal to the number of independent restrictions.

For example, a null specified by $g(\theta)=0$, where $g$ maps into $\mathbb{R}^q$, imposes $q$ locally independent restrictions when its Jacobian has rank $q$. This is a local statement: the relevant count is the rank of the restrictions, not necessarily the number of symbols used to write them.

Gaussian Log-Likelihood Curvature

For one observation $X\sim N_p(\mu(\theta),\Sigma(\theta))$, with positive-definite covariance matrix $\Sigma(\theta)$, the log-likelihood contribution is

\ell(\theta;X)=-\frac{p}{2}\log(2\pi)-\frac12\log|\Sigma(\theta)| -\frac12\{X-\mu(\theta)\}'\Sigma(\theta)^{-1}\{X-\mu(\theta)\}.

The negative Hessian of the log-likelihood describes its local curvature. Its expectation, evaluated at the true parameter, is the Fisher information. In a regular Gaussian model, the information per observation has entries

I_{ab}(\theta)= \frac{\partial\mu'}{\partial\theta_a}\Sigma^{-1} \frac{\partial\mu}{\partial\theta_b} +\frac12\operatorname{tr}\left( \Sigma^{-1}\frac{\partial\Sigma}{\partial\theta_a} \Sigma^{-1}\frac{\partial\Sigma}{\partial\theta_b} \right).

The first term is information from changes in the mean; the second is information from changes in covariance. To see the origin of this expression, write $r=X-\mu$, $S_a=\partial\Sigma/\partial\theta_a$, and $\mu_a=\partial\mu/\partial\theta_a$. Differentiating the Gaussian log-density gives the score component

U_a=\mu_a'\Sigma^{-1}r+ \frac12\left\{r'\Sigma^{-1}S_a\Sigma^{-1}r- \operatorname{tr}(\Sigma^{-1}S_a)\right\}.

For a centered Gaussian vector, odd centered moments vanish, and the covariance of two centered quadratic forms is $\operatorname{Cov}(r'Ar,r'Br)=2\operatorname{tr}(A\Sigma B\Sigma)$ for symmetric $A,B$. Applying these identities to $E[U_aU_b]$ gives the displayed information formula: the mean terms give its first term, the quadratic-form terms give its second, and the cross terms are zero. Thus, in a regular model, the log-likelihood has approximate curvature $nI(\theta_0)$ near the true value $\theta_0$. When nuisance parameters are present, the curvature relevant to a parameter of interest is the curvature remaining after those nuisance directions are allowed to adjust.

From a Quadratic Expansion to Wilks' Approximation

Here is a precise form of the local argument. Suppose the true parameter $\theta_0$ is an interior point of an identifiable, finite-dimensional model, and the null is $g(\theta)=0$, with $g(\theta_0)=0$. Assume $g$ is continuously differentiable near $\theta_0$, and its Jacobian $G=\partial g(\theta_0)/\partial\theta'$ has rank $q$. Assume also that the per-observation Fisher information $I=I(\theta_0)$ is positive definite.

In addition, require the usual regularity that justifies the local likelihood expansion and localizes both likelihood maximizers: the model is sufficiently smooth, the score satisfies a central limit theorem, the observed curvature converges to $I$, and the unrestricted and restricted maximizers are consistent and root-$n$ local. Under these conditions, for bounded $h$,

\ell_n(\theta_0+h/\sqrt n)-\ell_n(\theta_0) =h'\Delta_n-\frac12h'Ih+o_p(1), \qquad \Delta_n=\frac{1}{\sqrt n}\frac{\partial\ell_n(\theta_0)}{\partial\theta} \xrightarrow{d}N_d(0,I).

Theorem (Wilks approximation): Under the stated regularity conditions, when $\theta_0$ satisfies the null hypothesis, $$ W_n\xrightarrow{d}\chi^2_q, $$ where $q$ is the rank of the Jacobian of the null restrictions at $\theta_0$.

The quadratic expansion gives a complete geometric derivation. Locally, the unrestricted parameter can move in any direction $h\in\mathbb{R}^d$. Under the null, differentiability gives $g(\theta_0+h/\sqrt n)=Gh/\sqrt n+o(n^{-1/2})$, so the allowed local directions satisfy $Gh=0$ to first order. Define $Q(h)=h'\Delta_n-\tfrac12h'Ih$. The unrestricted maximum occurs at $h=I^{-1}\Delta_n$, by setting the derivative $\Delta_n-Ih$ to zero.

For the restricted maximum, impose $Gh=0$ with a Lagrange multiplier $\lambda$. The stationarity equation is $Ih-\Delta_n+G'\lambda=0$, so $h=I^{-1}(\Delta_n-G'\lambda)$. Substituting into the constraint gives $\lambda=(GI^{-1}G')^{-1}GI^{-1}\Delta_n$. The inverse exists because $I$ is positive definite and $G$ has rank $q$. The difference between the unrestricted and restricted maxima is therefore

\max_h Q(h)-\max_{Gh=0}Q(h) =\frac12 z_n'(GI^{-1}G')^{-1}z_n, \qquad z_n=GI^{-1}\Delta_n.

Since $2$ times this difference is the likelihood-ratio statistic, the local expansion yields $W_n=z_n'(GI^{-1}G')^{-1}z_n+o_p(1)$. The limit of $z_n$ is centered normal with covariance $GI^{-1}G'$. Whitening this $q$-dimensional normal vector makes the quadratic form a sum of squares of $q$ independent standard normal variables, which has a $\chi^2_q$ distribution. This proves the theorem. $\square$

Worked Examples

Worked Example: Curvature in a One-Restriction Gaussian Model

Suppose $X_1,\ldots,X_{25}$ are independent $N(\mu,4)$ observations, with known variance $4$. Test the restriction $H_0:\mu=0$. For this model, $\ell(\mu) = C-\tfrac{1}{8}\sum_{i=1}^{25}(X_i-\mu)^2$, where $C$ does not depend on $\mu$, and the unrestricted estimate is $\hat\mu=\bar X$.

The log-likelihood difference between $\hat\mu$ and $0$ is \[ \ell(\hat\mu)-\ell(0) =\frac18\left\{\sum_iX_i^2-\sum_i(X_i-\bar X)^2\right\} =\frac{25\bar X^2}{8}. \] The last equality follows by expanding the second sum: it equals $\sum_iX_i^2-25\bar X^2$. Hence $W=25\bar X^2/4$. If the observed sample mean is $0.4$, then $W=25(0.4)^2/4=1$. There is one restriction, so the usual large-sample reference distribution is $\chi^2_1$. In this particular known-variance Gaussian model, the statistic also has that distribution exactly under the null; that exact fact is special and is not required for the general approximation.

Worked Example: Two Restrictions and Two Degrees of Freedom

Let $X_i\sim N_3(\beta,I_3)$ independently for $i=1,\ldots,12$, with unknown mean vector $\beta$ and known covariance $I_3$. Consider the null restrictions $\beta_1=0$ and $\beta_2=0$, leaving $\beta_3$ unrestricted. Suppose the observed sample mean is $\bar X=(0.5,-0.25,0.2)'$.

The unrestricted maximum is at $\hat\beta=\bar X$. Under the null, the best fitting value is $\tilde\beta=(0,0,0.2)'$, since the third coordinate is free. The Gaussian log-likelihood is a constant minus $\tfrac{12}{2}\|\bar X-\beta\|^2$, apart from a term not depending on $\beta$. Therefore \[ W=12\left\{\|\bar X-\tilde\beta\|^2-\|\bar X-\hat\beta\|^2\right\} =12\{(0.5)^2+(-0.25)^2\}=3.75. \] The unrestricted residual term is zero in this expression because $\hat\beta=\bar X$. The Jacobian of the two restrictions has rank $2$, so the Wilks reference distribution is $\chi^2_2$, not $\chi^2_3$: the third coordinate was not restricted.

Worked Example: Nuisance Adjustment and Efficient Curvature

For each observation, suppose $X_i\sim N_2((\theta+\eta,\eta)',I_2)$, with parameters of interest $\theta$ and nuisance parameter $\eta$. Test $H_0:\theta=0$. The derivatives of the mean with respect to $(\theta,\eta)$ are $(1,0)'$ and $(1,1)'$. Thus the information per observation, and the total information for $n=10$, are

I_1=\begin{pmatrix}1&1\\1&2\end{pmatrix}, \qquad I_{10}=10I_1.

If $\eta$ were fixed, the $\theta$-curvature would be $10$. But $\eta$ is free to adjust. The curvature for $\theta$ after that adjustment is the Schur complement $10(1-1\cdot 2^{-1}\cdot 1)=5$; equivalently, the profile information is $10(I_{1,\theta\theta}-I_{1,\theta\eta}I_{1,\eta\eta}^{-1}I_{1,\eta\theta})=5$.

Suppose the observed sample mean is $(0.3,-0.1)'$. Without the restriction, the fitted mean equals the sample mean, giving $\hat\eta=-0.1$ and $\hat\theta=0.4$. Under $H_0$, the fitted mean must be $(\eta,\eta)'$; minimizing $(0.3-\eta)^2+(-0.1-\eta)^2$ gives $\tilde\eta=(0.3-0.1)/2=0.1$. The squared distance from the sample mean to this restricted fitted mean is $(0.3-0.1)^2+(-0.1-0.1)^2=0.08$. Consequently $W=10(0.08)=0.8$, and the null has one independent restriction, so its Wilks reference distribution is $\chi^2_1$. The nuisance adjustment changes the relevant curvature; it does not add another degree of freedom to the test.

When the Approximation Matters—and When It Can Fail

The chi-square calibration converts a likelihood loss into a reference scale that depends on the restriction dimension rather than on the units of the parameter. Its local geometric meaning is also useful: the likelihood rises quadratically away from its maximum, and the null removes $q$ locally available directions. If nuisance parameters are present, those directions must be allowed to re-optimize, which is why the relevant curvature is adjusted for nuisance variation.

A common mistake is to count every parameter mentioned in the null as a degree of freedom, or to count every parameter in the model. Neither count is generally correct. The degrees of freedom are the rank of the null Jacobian at the true parameter. In the second example, three means were estimated, but only two independent restrictions were imposed.

Another pitfall is to treat the approximation as an exact finite-sample law or to use it when regularity fails. If the true parameter is on a boundary, if a parameter is unidentified under the null, or if the information matrix is singular, the local quadratic argument above does not apply as stated. A different limiting distribution or a separate calibration may be needed. In applications, check both the restriction rank and the conditions that make the likelihood locally quadratic with nonsingular curvature.

Key takeaway: Under regularity, twice the log-likelihood gain from releasing $q$ independent smooth restrictions converges to $\chi^2_q$. Gaussian likelihood curvature supplies the quadratic approximation; nuisance parameters must be permitted to adjust within that geometry.

Check Your Understanding

Work through these questions by identifying the local restrictions and the curvature involved.

In a regular four-parameter model, a null hypothesis has a Jacobian of rank two. What is the Wilks degrees of freedom, and why is it not determined by the model dimension?
Why does a nuisance parameter affect the likelihood curvature relevant to a parameter of interest, even though it does not necessarily increase the null degrees of freedom?
In the local quadratic argument, what distributional property of the score leads to the chi-square limit?
Name two conditions under which the usual Wilks approximation may fail.
For a smooth null $g(\theta)=0$, what feature of its Jacobian determines the number of independent local restrictions?