Wilks' Theorem for Nested GLM Comparisons

Why a Likelihood-Ratio Statistic Has a Reference Distribution

For nested models, the larger model can fit at least as well as the smaller one. The likelihood-ratio statistic measures the improvement in fit, but its size alone does not say whether that improvement is surprising under the smaller model. Wilks’ theorem supplies a reference distribution: under regularity conditions and the null hypothesis, the statistic is asymptotically chi-square. The degrees of freedom are determined by how many additional independent parameters the larger model estimates.

This is an asymptotic result, not a claim that a likelihood-ratio statistic has exactly a chi-square distribution in every finite sample. Its justification depends on a regular, identifiable model, an interior null parameter, and enough information in the data for the likelihood to have a stable local quadratic approximation.

Statistic, Hypothesis, and Degrees of Freedom

Definition: Let a full model have parameter $\theta$ in a $p$-dimensional parameter space, and let a nested null model restrict $\theta$ to a smooth $q$-dimensional submodel, where $q<p$. Using the same likelihood for both models, define the likelihood-ratio statistic $$ T=2\left\{\sup_{\theta\ \mathrm{in\ full\ model}}\ell(\theta)-\sup_{\theta\ \mathrm{in\ null\ model}}\ell(\theta)\right\}. $$ The dimension difference is $d=p-q$. Under the regularity conditions of Wilks’ theorem and when the null model is true, $T\xrightarrow{d}\chi^2_d$.

The parameter dimensions count all free parameters in each model, including nuisance parameters. If both models contain the same nuisance parameters, those parameters contribute to both dimensions and hence cancel in the difference. In a GLM comparison, the relevant dimensions are the ranks of the identifiable parameter spaces, not simply the number of coefficient labels written in a formula. Redundant parameters do not add dimensions.

The theorem applies to a correctly specified, regular likelihood problem. One standard formulation assumes independent observations and a finite-dimensional parameter $\theta$, with the following conditions in a neighbourhood of the true null value $\theta_0$:

The full parameter space is locally open in $\mathbb{R}^p$, and $\theta_0$ is an interior point. The null model is a smooth embedded submodel of dimension $q$, also containing $\theta_0$ as an interior point of that submodel.
The model is identifiable, and the Fisher information at $\theta_0$ exists and is finite and positive definite.
The log-likelihood is sufficiently differentiable near $\theta_0$, with regularity allowing its score and Hessian to have the usual limits and allowing a uniform local quadratic expansion.
The full and null maximum-likelihood estimators are consistent and localize at the usual $n^{-1/2}$ rate; the relevant likelihood maxima exist with probability tending to one.

A convenient way to state the central technical requirement is that, for bounded local displacements $h$, the log-likelihood has the expansion $$ \ell(\theta_0+h/\sqrt{n})-\ell(\theta_0) =h^{\mathsf T}\Delta_n-\frac{1}{2}h^{\mathsf T}Ih+o_p(1), $$ uniformly over bounded sets of $h$. Here $I$ is the limiting information matrix and $\Delta_n$ converges in distribution to a mean-zero normal vector with covariance $I$. Standard regular GLMs can satisfy these conditions, but the design, response distribution, and parameter values must actually permit them. The theorem does not automatically cover every GLM fit or every comparison.

Why the Limit Is Chi-Square

The key idea is that, close to the true parameter, maximizing the log-likelihood is approximately the same as maximizing a quadratic function. Under the null, the full model can improve the fit in $p-q$ local directions that the null model does not allow. After measuring those directions in information-scaled coordinates, the improvements are squares of independent standard normal variables.

Theorem (Wilks’ theorem): Suppose the full model is a regular $p$-dimensional parametric model and the true parameter lies in the interior of a smooth $q$-dimensional nested null submodel. Under the likelihood regularity and localization conditions stated above, the likelihood-ratio statistic $T$ converges in distribution under the null to a chi-square random variable with $p-q$ degrees of freedom.

Proof. Use a smooth local reparameterization and an information-scaled change of coordinates. These put the limiting information matrix in identity form and make the tangent space of the null submodel the first $q$ coordinate directions. Write $d=p-q$. In these coordinates the local likelihood expansion is

\ell(h)-\ell(0)=h^{\mathsf T}Z_n-\frac{1}{2}h^{\mathsf T}h+o_p(1), \qquad Z_n\xrightarrow{d}N_p(0,I_p).

Here $h$ denotes the local parameter displacement on the $n^{-1/2}$ scale, and $I_p$ is the $p$-dimensional identity matrix. The smooth null submodel has, on this scale, the tangent constraint $h_{q+1}=\cdots=h_p=0$. Its curvature does not change the limiting quadratic optimization: at bounded local scales, the rescaled submodel converges to its tangent space.

For a fixed vector $z$, the quadratic function $h^{\mathsf T}z-\frac12 h^{\mathsf T}h$ can be written as $$ \frac12 z^{\mathsf T}z-\frac12(h-z)^{\mathsf T}(h-z). $$ Its unrestricted maximum is therefore $\frac12\sum_{j=1}^{p}z_j^2$, attained at $h=z$. Under the tangent constraint, its maximum is $\frac12\sum_{j=1}^{q}z_j^2$, attained by setting the first $q$ coordinates of $h$ equal to those of $z$. Subtracting the constrained maximum from the unrestricted maximum and multiplying by 2 gives $$ T=\sum_{j=q+1}^{p}Z_{n,j}^2+o_p(1). $$ The normal limit has independent standard normal coordinates, so the sum of the $p-q$ squared coordinates converges in distribution to $\chi^2_{p-q}$. The assumed localization and uniform expansion justify using the local quadratic maxima for the original likelihood maxima. Thus $T\xrightarrow{d}\chi^2_{p-q}$. $\square$

The proof also explains the degrees of freedom: they count the local directions available to the full model but excluded by the null. They are not the sample size, the residual degrees of freedom, or the number of observations whose fitted values change.

Worked Examples

Worked Example: Three Binomial Groups

Suppose three independent groups each contain $20$ Bernoulli observations. The observed success counts are $4$, $10$, and $16$. Compare a null model with one common success probability $p$ to a model with a separate probability $p_g$ for each group. This is a nested logistic-regression comparison in which group is represented by categorical indicators.

Under the null, the maximum-likelihood estimate is the pooled proportion $ \hat p=(4+10+16)/60=0.5 $. Under the larger model, the estimates are $ \hat p_1=4/20=0.2 $, $ \hat p_2=10/20=0.5 $, and $ \hat p_3=16/20=0.8 $. The null has dimension $1$, the larger model has dimension $3$, and the dimension difference is $2$.

The binomial combinatorial terms cancel in the log-likelihood difference. The first group contributes $4\log(0.2/0.5)+16\log(0.8/0.5)$. Numerically, $4\log(0.4)=-3.66516$ and $16\log(1.6)=7.52006$, so its contribution is $3.85490$. The second group contributes zero because its fitted probability is $0.5$ under both models. The third group contributes $16\log(1.6)+4\log(0.4)=3.85490$, the same gain as the first group. Thus

\ell_{\mathrm{full}}-\ell_{\mathrm{null}} =3.85490+0+3.85490=7.70980, \qquad T=2(7.70980)=15.41960.

Under the regular null, the asymptotic reference distribution is $\chi^2_2$, not $\chi^2_3$: there are two additional parameters. The chi-square tail probability is approximately $0.00045$. This is an asymptotic calibration; the arithmetic calculation of $T$ itself does not establish that the finite-sample distribution is exactly chi-square.

Worked Example: Two Poisson Group Means

Let independent counts $Y_1$ and $Y_2$ have Poisson distributions. Compare the null model $Y_1,Y_2\sim\operatorname{Poisson}(\mu)$, with a common mean, to the model with separate means $\mu_1,\mu_2$. Suppose the observed counts are $12$ and $20$, with equal exposures. The null estimate is $\hat\mu=(12+20)/2=16$; the separate estimates are $12$ and $20$. The dimensions are $1$ and $2$, so $d=1$.

Ignoring terms that are the same in both log-likelihoods, the difference is $12\log(12/16)+20\log(20/16)$. The two terms are $12\log(0.75)=-3.45218$ and $20\log(1.25)=4.46287$. Therefore

\ell_{\mathrm{full}}-\ell_{\mathrm{null}} =-3.45218+4.46287=1.01069, \qquad T=2(1.01069)=2.02138.

Wilks’ reference distribution is $\chi^2_1$. The conclusion is about the limiting null distribution for repeated data under the stated model, not an exact distributional identity for these two observed counts.

Worked Example: Gaussian Mean Models With Unknown Variance

Suppose $n=40$ independent observations are normally distributed with a common unknown variance. A smaller mean model has two free mean parameters; a nested larger mean model has four. Suppose the observed residual sums of squares are $RSS_0=80$ and $RSS_1=64$, respectively. Such values are compatible with nested mean spaces: the added two directions can account for a reduction of $16$ in the residual sum of squares, leaving $64$ in the larger model.

For a model with residual sum of squares $RSS$, maximizing the normal likelihood over the variance gives $\hat\sigma^2=RSS/n$. Substitution into the log-likelihood gives, up to constants common to the models, $ \ell_{\mathrm{profile}}=-(n/2)\log(RSS/n) $. Consequently,

T=2(\ell_1-\ell_0) =n\log\left(\frac{RSS_0}{RSS_1}\right) =40\log(80/64) =40\log(1.25) \approx 8.926.

The unknown variance is estimated in both models, so it adds one parameter to each dimension and does not change their difference: $d=4-2=2$. Wilks’ theorem therefore gives the asymptotic reference distribution $\chi^2_2$. This example uses the likelihood-ratio statistic only; it does not assert an exact finite-sample chi-square calibration.

Using the Limit and Avoiding Overclaim

If $c_\alpha$ is the upper-tail critical value satisfying $P(\chi^2_d>c_\alpha)=\alpha$, Wilks’ theorem implies that a test rejecting when $T>c_\alpha$ has asymptotic size $\alpha$, under its assumptions. This follows directly from convergence in distribution: the chi-square distribution has a continuous distribution function at $c_\alpha$, so

P_{\theta_0}(T>c_\alpha) \longrightarrow P(\chi^2_d>c_\alpha)=\alpha.

This is an asymptotic-level statement. It does not say that the finite-sample rejection probability equals $\alpha$. It explains why the chi-square reference distribution is used when the sample and model provide a reasonable approximation to the regular likelihood limit.

A common mistake is to say, “The models are nested, so their likelihood-ratio statistic is chi-square with the difference in parameter counts.” Nesting supplies the comparison, but it is not enough for Wilks’ theorem. The null parameter must be interior, the model must be identifiable and regular, and the likelihood must have the required large-sample behavior. For example, a parameter constrained to a boundary or an unidentified parameter can invalidate the ordinary chi-square limit. A nominal parameter count can also give the wrong degrees of freedom when parameters are redundant. The correction is to check the regularity conditions and use the dimension of each identifiable model before invoking the theorem.

Key takeaway: In a regular nested likelihood comparison, the null likelihood-ratio statistic converges to $\chi^2_d$, where $d$ is the difference in identifiable parameter dimensions. The chi-square calibration is an asymptotic consequence of a local quadratic likelihood, not a universal finite-sample rule.

Check Your Understanding

Use the theorem and its assumptions to answer these questions.

A full model has six identifiable parameters and its nested null model has four. Under regularity and the null, what is the limiting reference distribution?
Why do common nuisance parameters generally not change the degrees of freedom for a nested comparison?
In the binomial example, why does the middle group contribute zero to the log-likelihood difference, and why do the first and third groups contribute equally?
What does convergence to a chi-square distribution imply about a test using a chi-square critical value, and what does it not imply about a finite sample?
Name two conditions whose failure can make the ordinary Wilks chi-square calibration unreliable.