Wilks' Theorem for Likelihood-Ratio Statistics

From a Likelihood Gain to a Chi-Square Limit

A likelihood-ratio statistic measures how much better the best-fitting model can explain the data than the best-fitting model subject to a null hypothesis. Wilks' theorem explains why, in a regular large-sample problem, twice this log-likelihood gain has an approximately chi-square distribution when the null is true. Its degrees of freedom are determined by the number of independent smooth restrictions imposed by the null.

The key idea is local. Near the true parameter, a regular log-likelihood is approximately quadratic. Maximizing that quadratic without restrictions and maximizing it over the local form of the null hypothesis produce two different gains. Their difference is the squared length of a projection onto the directions excluded by the null.

Definition: Let $X_1,\ldots,X_n$ be observations with likelihood $L(\theta)$, where $\theta\in\Theta\subset\mathbb{R}^p$. For a null parameter set $\Theta_0\subset\Theta$, the likelihood-ratio statistic is $$ \Lambda_n=\frac{\sup_{\theta\in\Theta_0}L(\theta)}{\sup_{\theta\in\Theta}L(\theta)},\qquad T_n=-2\log\Lambda_n =2\left\{\sup_{\theta\in\Theta}\ell(\theta)-\sup_{\theta\in\Theta_0}\ell(\theta)\right\}. $$ The statistic is nonnegative because the unrestricted model includes the null model.

Wilks' Theorem for Smooth Restrictions

Here is a standard regular-model version of Wilks' theorem. Suppose the observations are independent and identically distributed from a correctly specified model with true parameter $\theta_0$. Assume $\theta_0$ is an interior point of the open parameter space $\Theta\subset\mathbb{R}^p$. Write the null hypothesis locally as $g(\theta)=0$, where $g:\Theta\to\mathbb{R}^q$ is continuously differentiable near $\theta_0$, $g(\theta_0)=0$, and its $q\times p$ derivative matrix $A=\partial g(\theta_0)/\partial\theta^\mathsf{T}$ has full row rank. Thus the null imposes $q$ independent smooth restrictions at the true parameter.

For clarity, the remaining assumptions below state the local likelihood and maximization properties the proof needs. Let $S_n(\theta_0)=\partial\ell(\theta_0)/\partial\theta$ be the score and let $I$ be the positive-definite Fisher information matrix for one observation. Assume $n^{-1/2}S_n(\theta_0)\xrightarrow{d}N_p(0,I)$, and that, uniformly for $h$ in every bounded subset of $\mathbb{R}^p$,

\ell(\theta_0+h/\sqrt n)-\ell(\theta_0) =h^\mathsf{T}\frac{S_n(\theta_0)}{\sqrt n} -\frac12h^\mathsf{T}Ih+o_p(1).

Also assume that unrestricted and restricted likelihood maximizers exist with probability tending to one, are consistent for $\theta_0$ under the null, and are $O_p(n^{-1/2})$ from it. In local coordinates, assume the feasible null sets converge to the tangent set $\{h:Ah=0\}$, so that maximizing the local quadratic approximation over the null has the same limiting supremum as maximizing over that tangent set. These are standard localization conditions in regular likelihood theory; they rule out distant competing maxima that the local expansion cannot describe.

Theorem (Wilks' theorem): Under the conditions just stated, if the null hypothesis is true at $\theta_0$, then $$ T_n\xrightarrow{d}\chi^2_q. $$

Proof by Local Quadratic Maximization

Set $Z_n=S_n(\theta_0)/\sqrt n$. The local log-likelihood gain, up to a term that is $o_p(1)$ uniformly on bounded sets, is $Q_n(h)=2h^\mathsf{T}Z_n-h^\mathsf{T}Ih$ when written on the twice-log-likelihood scale. The localization assumptions allow the unrestricted and restricted likelihood gains to be evaluated using this quadratic form and its limiting feasible sets.

First maximize over all $h\in\mathbb{R}^p$. Completing the square gives

Q_n(h) =Z_n^\mathsf{T}I^{-1}Z_n -(h-I^{-1}Z_n)^\mathsf{T}I(h-I^{-1}Z_n).

Since $I$ is positive definite, the subtracted term is nonnegative and is zero at $h=I^{-1}Z_n$. Therefore the unrestricted maximum is $Z_n^\mathsf{T}I^{-1}Z_n$.

For the restricted maximum, change coordinates to $u=I^{1/2}h$ and $z_n=I^{-1/2}Z_n$, where $I^{1/2}$ is the positive-definite square root. The local null constraint becomes $Cu=0$, with $C=AI^{-1/2}$, and

Q_n(h)=2z_n^\mathsf{T}u-u^\mathsf{T}u =\|z_n\|^2-\|u-z_n\|^2.

Thus maximizing over the restricted set $S=\{u:Cu=0\}$ is equivalent to finding the point of $S$ closest to $z_n$. The maximizer is the orthogonal projection $P_Sz_n$, and the restricted maximum is $\|P_Sz_n\|^2$. Consequently, the difference between unrestricted and restricted maxima is

\|z_n\|^2-\|P_Sz_n\|^2 =\|P_{S^\perp}z_n\|^2.

The full row rank of $A$ implies that $C$ has rank $q$, so $S^\perp$ has dimension $q$. Also, the score limit gives $z_n\xrightarrow{d}N_p(0,I_p)$. In an orthonormal basis whose first $q$ directions span $S^\perp$, the projection $P_{S^\perp}z_n$ therefore converges in distribution to $q$ independent standard normal coordinates. Its squared length converges to the sum of their squares, which is $\chi^2_q$. The local likelihood approximation and localization assumptions transfer this limit to $T_n$, proving the theorem. $\square$

Worked Likelihood-Ratio Calculations

Worked Example: Equality of Two Normal Means

Suppose $X_{ij}\sim N(\theta_j,1)$, independently for $i=1,\ldots,n$ and $j=1,2$. Test $H_0:\theta_1=\theta_2$ against the unrestricted model. There is one smooth restriction, so Wilks' theorem predicts a $\chi^2_1$ limit.

Write $\bar X_1,\bar X_2$ for the sample means. The parts of the log-likelihood that depend on the means are $\ell(\theta)=-\frac n2\{(\bar X_1-\theta_1)^2+(\bar X_2-\theta_2)^2\}+\text{constant}$. The unrestricted estimates are $\hat\theta_j=\bar X_j$, making both squared terms zero. Under the null, the common estimate is $\tilde\theta=(\bar X_1+\bar X_2)/2$. The identity $(\bar X_1-\tilde\theta)^2+(\bar X_2-\tilde\theta)^2=(\bar X_1-\bar X_2)^2/2$ then gives

T_n=2\{\ell(\hat\theta)-\ell(\tilde\theta,\tilde\theta)\} =\frac n2(\bar X_1-\bar X_2)^2.

For $n=50$, suppose $\bar X_1=0.8$ and $\bar X_2=0.2$. Then $T_n=(50/2)(0.6)^2=25(0.36)=9$. Under the null, $\bar X_1-\bar X_2\sim N(0,2/n)$, so $T_n\sim\chi^2_1$ exactly in this model. This exact distribution is a special feature of the normal setup; the theorem itself only promises a limiting distribution in general.

Worked Example: A Bernoulli Success Probability

Let $X_1,\ldots,X_{100}$ be Bernoulli observations with success probability $p$, and test $H_0:p=0.5$. If the observed number of successes is 60, the unrestricted maximum-likelihood estimate is $\hat p=0.6$, while the null fixes $p=0.5$. The log-likelihood, apart from a constant, is $\ell(p)=60\log p+40\log(1-p)$. Thus

\begin{aligned} T_n &=2\{\ell(0.6)-\ell(0.5)\}\\ &=2\left\{60\log(1.2)+40\log(0.8)\right\}\\ &=2\{60(0.1823216)+40(-0.2231436)\}\\ &\approx2(2.01355)=4.0271. \end{aligned}

The null is a single interior restriction, so the large-sample reference distribution is $\chi^2_1$. The calculation shows how the log-likelihood gain is formed; the chi-square approximation is a statement about its distribution across repeated samples when $p=0.5$, not a claim that this observed statistic must itself be close to its mean.

Worked Example: Testing a Poisson Rate

Suppose $X_1,\ldots,X_{20}$ are independent Poisson observations with common rate $\lambda$. Test $H_0:\lambda=1$. The sufficient statistic is $T=\sum_iX_i$; take an observed total $T=30$. The unrestricted estimate is $\hat\lambda=T/n=1.5$, and the null estimate is $1$.

Ignoring terms that do not depend on $\lambda$, the log-likelihood is $T\log\lambda-n\lambda$. Substitution gives

\begin{aligned} T_n &=2\{30\log(1.5)-20(1.5)-[30\log(1)-20(1)]\}\\ &=2\{30\log(1.5)-10\}\\ &=2\{30(0.4054651)-10\}\\ &\approx2(2.16395)=4.3279. \end{aligned}

There is one interior restriction, so Wilks' theorem gives a $\chi^2_1$ limit when the null is true and the sample size grows. In this example $n=20$ is finite: the theorem motivates an approximation, not an exact finite-sample distribution.

Interpretation, Scope, and a Common Pitfall

The projection argument explains the degrees of freedom in the theorem. In the local quadratic approximation, the unrestricted fit can move in every parameter direction, whereas the null fit can move only along the tangent set $Ah=0$. The likelihood gain comes from the $q$ directions normal to that set. Information $I$ changes the geometry used to measure the displacement, but after transforming to information-scaled coordinates, the limiting gain is a sum of $q$ squared standard normal variables.

A common mistake is to treat the chi-square calibration as automatic whenever a statistic is written as $-2\log\Lambda_n$. It is not. Wilks' theorem requires a true null, a regular interior parameter point, a smooth null restriction, nonsingular information, and a well-behaved likelihood with the stated local approximation and localization. For example, if the null parameter lies on the boundary of the parameter space, the local feasible set need not be a linear space and the usual chi-square limit can fail. Nor does the theorem claim an exact chi-square distribution at a fixed sample size, except in special models such as the normal example above.

Key takeaway: Under regularity conditions, twice the log-likelihood gain from releasing $q$ independent smooth null restrictions converges under the null to $\chi^2_q$. The result follows by maximizing the local quadratic log-likelihood over the full space and the null tangent space.

Check Your Understanding

Use the theorem and its proof to answer these questions.

What is the likelihood-ratio statistic in terms of the unrestricted and restricted maximized log-likelihoods?
In the local quadratic proof, why is maximizing over the null equivalent to projecting onto a tangent set?
What feature of the restriction determines the degrees of freedom in the stated theorem?
Why does the normal-mean example have an exact chi-square distribution even though Wilks' theorem is generally asymptotic?
Give one regularity condition whose failure can make the usual chi-square calibration unreliable.