Likelihood-Ratio Tests and Wilks' Theorem

Why Compare Maximized Likelihoods?

A likelihood-ratio test asks how much fit is lost when a model is required to satisfy a null hypothesis. The comparison is between the best fit available under the null and the best fit available without that restriction. This makes the test applicable to many likelihood models, including models with nuisance parameters, provided the null and alternative are nested and the relevant regularity conditions hold.

The central result is Wilks’ theorem: under the null and suitable regularity conditions, twice the log-likelihood improvement from removing the restrictions has an asymptotic chi-square distribution. The degrees of freedom are the number of independent restrictions, not necessarily the number of parameters in either model.

Definition of the Likelihood-Ratio Statistic

Let $Y_1,\ldots,Y_n$ be data with likelihood $L_n(\theta)$ and log likelihood $\ell_n(\theta)=\log L_n(\theta)$, where $\theta$ belongs to a parameter space $\Theta$. Consider a null parameter space $\Theta_0$ contained in the alternative parameter space $\Theta$. The null may specify some parameter values while leaving other parameters unrestricted.

Definition: The likelihood ratio for testing $H_0:\theta\in\Theta_0$ against $H_a:\theta\in\Theta\setminus\Theta_0$ is $$ \Lambda_n= \frac{\sup_{\theta\in\Theta_0}L_n(\theta)} {\sup_{\theta\in\Theta}L_n(\theta)}. $$ The likelihood-ratio statistic is $$ LR_n=-2\log\Lambda_n =2\left[ \sup_{\theta\in\Theta}\ell_n(\theta) -\sup_{\theta\in\Theta_0}\ell_n(\theta) \right]. $$

Because $\Theta_0\subseteq\Theta$, the unrestricted likelihood maximum cannot be smaller than the null-constrained maximum. Consequently $0\leq\Lambda_n\leq1$ and $LR_n\geq0$. A large value means that imposing the null restriction causes a substantial decrease in the maximized log likelihood. The statistic itself is not a p-value; it is compared with a reference distribution.

When the maxima are attained at the unrestricted maximum-likelihood estimator $\hat{\theta}$ and the restricted maximum-likelihood estimator $\tilde{\theta}$, respectively, the statistic is $2[\ell_n(\hat{\theta})-\ell_n(\tilde{\theta})]$. These estimators need not have the same values for nuisance parameters: each maximization chooses the nuisance parameter values that fit best within its own parameter space.

Worked Example: Equal Probabilities in a Multinomial Model

Suppose, hypothetically, that 40 independent trials produce three categories with counts $y_1=16$, $y_2=10$, and $y_3=14$. Test the null hypothesis that the three category probabilities are equal. The model is multinomial with probabilities $\pi_1,\pi_2,\pi_3>0$, where $\pi_1+\pi_2+\pi_3=1$. The null sets each probability to $1/3$; the alternative leaves the probabilities unrestricted subject to their sum being one.

The likelihood, omitting a factor that is the same under both hypotheses, is proportional to $\prod_{j=1}^3\pi_j^{y_j}$. Under the unrestricted model, the maximum-likelihood estimates are $y_j/40$. Under the null, the probabilities are fixed at $1/3$. Thus

LR_{40} =2\sum_{j=1}^3 y_j\log\left(\frac{y_j}{40/3}\right) =2\left[ 16\log(1.2)+10\log(0.75)+14\log(1.05) \right] \approx 1.447.

There are two independent restrictions: the unrestricted probability vector has two free components, whereas the null specifies both. The null probabilities are interior, and the multinomial model is identifiable with positive information there. Wilks’ theorem therefore gives an approximate $\chi^2_2$ reference distribution. The upper-tail p-value is $P(\chi^2_2\geq1.447)\approx0.485$. At conventional significance levels this is not evidence against equal probabilities; it does not establish that the probabilities are equal.

Wilks’ Theorem and Its Degrees of Freedom

A common form of the null hypothesis is $H_0:g(\theta)=0$, where $g$ is a vector of $q$ smooth restrictions. The number relevant to Wilks’ theorem is the rank of the derivative matrix of $g$ at the true null parameter. If that rank is $q$, the restrictions are locally independent. Counting equations alone can be misleading: two written restrictions that impose the same local constraint do not count as two independent restrictions.

Theorem (Wilks): Suppose the data arise from a correctly specified, identifiable, regular likelihood model; the true parameter is an interior point; the log likelihood is sufficiently smooth; and the Fisher information is finite and nonsingular. Suppose also that the null is a smooth restriction of rank $q$, and that the restricted and unrestricted maximum-likelihood estimators behave regularly. Under $H_0$, $$ LR_n \xrightarrow{d} \chi^2_q. $$ Here $\chi^2_q$ denotes the distribution of the sum of squares of $q$ independent standard normal random variables. The approximation is asymptotic: it describes the null distribution as the sample size grows, and need not be exact in a finite sample.

The result applies to composite null hypotheses as well as to hypotheses that fix the entire parameter. For instance, if $\theta=(\psi,\lambda)$, where $\lambda$ is a nuisance parameter, and the null imposes $q$ independent restrictions on $\psi$, both likelihood maxima optimize over the allowed values of $\lambda$. Under regularity conditions, the degrees of freedom remain $q$, not the number of nuisance parameters.

Why the Null Limit Is Chi-Square

The key is the local quadratic shape of a regular log likelihood near the true parameter $\theta_0$. Write the local parameter as $\theta=\theta_0+h/\sqrt{n}$, and let $Z_n$ be the score at $\theta_0$, divided by $\sqrt{n}$. The standard likelihood expansion under the null is

\ell_n(\theta_0+h/\sqrt{n})-\ell_n(\theta_0) = h^{\mathsf T}Z_n-\frac{1}{2}h^{\mathsf T}Ih+o_p(1), \qquad Z_n\xrightarrow{d}N(0,I),

where $I$ is the per-observation Fisher information at $\theta_0$. In this local approximation, the unrestricted model maximizes the quadratic expression over all $h$. The null maximizes it only over directions tangent to the null restriction. The likelihood-ratio statistic is twice the difference between those two maxima.

To see the distribution of that difference, transform coordinates using $v=I^{1/2}h$ and $a_n=I^{-1/2}Z_n$. Then $a_n$ converges in distribution to a standard multivariate normal vector, and the quadratic expression becomes $v^{\mathsf T}a_n-\frac12v^{\mathsf T}v$. In the transformed coordinates, the local null directions form a subspace $T$ of dimension $d-q$, where $d$ is the full parameter dimension. If $P_T$ is projection onto that subspace, maximizing over all directions gives $\frac12\lVert a_n\rVert^2$, while maximizing over null directions gives $\frac12\lVert P_Ta_n\rVert^2$. Therefore twice the difference is approximately

\lVert a_n\rVert^2-\lVert P_Ta_n\rVert^2 = \lVert (I_d-P_T)a_n\rVert^2.

The matrix $I_d-P_T$ projects onto the $q$-dimensional space perpendicular to the null directions. A standard multivariate normal vector projected onto that space has $q$ independent standard normal coordinates. The squared length of those coordinates has a $\chi^2_q$ distribution. This explains both the chi-square form and why the degrees of freedom equal the number of independent restrictions.

With nuisance parameters, this geometric argument still applies after accounting for the nuisance directions. Equivalently, the information for the parameters restricted by the null is adjusted for nuisance estimation, as in the earlier material on efficient information and nuisance adjustment. The dimension removed by the null remains $q$.

Worked Example: Testing Equality of Two Poisson Rates

Suppose, hypothetically, that two independent processes are observed for one unit of exposure each, producing counts $Y_1=18$ and $Y_2=8$. Assume $Y_i\sim\operatorname{Poisson}(\lambda_i)$, with independent counts and positive rates. Test $H_0:\lambda_1=\lambda_2$ against unequal rates. The unrestricted model has two rate parameters; the null leaves one common rate, so there is one independent restriction. The null rate is interior, and this Poisson model has positive information at positive rates.

Under the alternative, the maximum-likelihood estimates are $\hat{\lambda}_1=18$ and $\hat{\lambda}_2=8$. Under the null, the common-rate estimate is the total count divided by the total exposure: $\tilde{\lambda}=26/2=13$. The log-likelihood terms involving factorials cancel in the likelihood ratio. Substitution gives

LR =2\left[ 18\log\left(\frac{18}{13}\right) +8\log\left(\frac{8}{13}\right) \right] \approx 3.947.

Wilks’ theorem gives an approximate $\chi^2_1$ reference distribution. Its upper-tail p-value is approximately $0.0470$. At a 5% significance level, this is just below $0.05$, so the result provides evidence against equal rates under the stated model and asymptotic calibration. It does not show that the difference in rates is large or practically important; the estimated rates are 18 and 8 events per unit of exposure.

Worked Example: A Known-Variance Normal Mean

For a hypothetical sample of $n=25$ independent observations from $N(\mu,1)$, suppose the sample mean is $\bar{x}=0.4$. Test $H_0:\mu=0$ against the unrestricted alternative. The variance is known to be one, and the observations are assumed to follow the specified normal model. The unrestricted estimate is $\hat{\mu}=\bar{x}$, while under the null $\mu$ is fixed at zero. This is one independent restriction, and the null value is interior in the unrestricted parameter space.

Ignoring constants that cancel, the log likelihood is $-\frac12\sum_{i=1}^{25}(x_i-\mu)^2$. The difference between the unrestricted and null log likelihoods is $\frac12n\bar{x}^{\,2}$, so

LR = n\bar{x}^{\,2} =25(0.4)^2 =4.000.

The Wilks reference distribution is $\chi^2_1$, giving an upper-tail p-value of approximately $0.0455$. Thus, at the 5% level, these hypothetical data provide evidence against $\mu=0$ under the normal model. In this particular known-variance normal model the chi-square calibration is exact under the null, but Wilks’ theorem supplies the more general asymptotic justification used across regular likelihood models.

Using the Test and Avoiding Common Errors

For an upper-tail test at significance level $\alpha$, reject $H_0$ when $LR_n$ exceeds the $1-\alpha$ quantile of $\chi^2_q$, or report the corresponding upper-tail p-value. The p-value is the probability, assuming the null and the reference conditions, of obtaining a likelihood-ratio statistic at least as large as the observed one. It is not the probability that the null hypothesis is true.

The chi-square approximation requires care. It may fail when the true parameter is on a boundary, the model is not identifiable under the null, information is singular, or the restriction is not smooth with the required rank. In such settings the limiting distribution can differ from $\chi^2_q$. Small samples can also make the asymptotic approximation inaccurate even when the regularity conditions hold; the likelihood-ratio statistic alone does not guarantee a reliable finite-sample p-value.

Mistake: counting parameters rather than restrictions. The degrees of freedom are not the number of parameters in the unrestricted model. They are the number of independent restrictions imposed by the null.
Mistake: reversing the likelihood ratio. With $\Lambda_n=L_{\text{null,max}}/L_{\text{unrestricted,max}}$, the ratio is at most one. The statistic is $-2\log\Lambda_n$, so a worse fit under the null yields a larger statistic.
Mistake: treating the approximation as exact in every sample. Wilks’ theorem gives a limiting null distribution under regularity conditions. Check the model and restrictions before using its chi-square reference distribution.
Mistake: interpreting a nonsignificant result as proof of the null. Failure to reject means the data do not provide sufficient evidence against the null at the chosen level; it does not establish that the null is true.

Key takeaway: For nested regular likelihood models, compare the restricted and unrestricted maxima using $LR_n=2(\ell_{\text{unrestricted,max}}-\ell_{\text{null,max}})$. Under the null, its asymptotic reference distribution is $\chi^2_q$, where $q$ is the rank of the independent restrictions.

Check Your Understanding

Answer each question using the definitions and conditions developed above.

Why must the likelihood ratio $\Lambda_n$ be no greater than one when the null model is nested within the alternative?
A model has five free parameters under the alternative, and the null imposes two locally independent restrictions. What is the Wilks reference distribution, assuming regularity conditions hold?
In a model with nuisance parameters, should the restricted likelihood maximum hold the nuisance parameters at their unrestricted estimates? Explain.
In the quadratic justification, why does projection onto the directions excluded by the null produce $q$ squared normal components?
Name two circumstances in which the usual $\chi^2_q$ calibration may not apply.