Likelihood-Ratio Tests for Nested Models

Comparing Nested Models

A likelihood-ratio test asks how much better the data can be explained when a model is allowed to vary freely than when it is constrained by the null hypothesis. If the improvement is small relative to what sampling variation can produce, the data provide little evidence against the constraint. If the improvement is large, the restricted model is difficult to reconcile with the observations.

The comparison is especially direct for nested models: every distribution allowed by the null model is also allowed by the alternative model. The alternative therefore has at least as high a maximized likelihood. The size of that increase, measured on the log-likelihood scale, is the likelihood-ratio statistic.

Definition: Let $\Theta$ be the parameter space for an unrestricted model and let $\Theta_0\subseteq\Theta$ be the parameter space under the null hypothesis. For data with likelihood $L(\theta)$ and log-likelihood $\ell(\theta)=\log L(\theta)$, define $$ \Lambda=\frac{\sup_{\theta\in\Theta_0}L(\theta)}{\sup_{\theta\in\Theta}L(\theta)} \qquad\text{and}\qquad T_{\mathrm{LR}}=-2\log\Lambda =2\left\{\sup_{\theta\in\Theta}\ell(\theta)-\sup_{\theta\in\Theta_0}\ell(\theta)\right\}. $$ When the suprema are attained, these are the unrestricted and restricted maximum likelihoods, respectively. Small $\Lambda$, equivalently large $T_{\mathrm{LR}}$, is evidence against the null hypothesis.

Both maximizations use the same observed data and the same likelihood, including any nuisance parameters in the model. The restricted maximization imposes the null hypothesis; the unrestricted maximization does not. Terms in the log likelihood that do not depend on parameters may be omitted from both fits, since they cancel in the difference.

Why the Statistic Is Nonnegative

For nested models, every parameter value allowed under the null is also available to the unrestricted fit. Maximizing over a larger set cannot lower the maximum. This gives a basic property that is also a useful check on calculations.

Result: If $\Theta_0\subseteq\Theta$, then $\sup_{\theta\in\Theta}\ell(\theta)\geq\sup_{\theta\in\Theta_0}\ell(\theta)$, so $T_{\mathrm{LR}}\geq0$ and $0\leq\Lambda\leq1$.

Proof. The set of log-likelihood values $\{\ell(\theta):\theta\in\Theta_0\}$ is a subset of $\{\ell(\theta):\theta\in\Theta\}$. Its supremum cannot exceed the supremum over the larger set. Thus the difference defining $T_{\mathrm{LR}}$ is nonnegative. Since $\Lambda=\exp(-T_{\mathrm{LR}}/2)$, it lies between zero and one. $\square$

For smooth, regular models, the large-sample reference distribution under the null is determined by how many independent restrictions distinguish the two models. If the unrestricted model has $k$ free parameters and the null model has $k_0$, the relevant difference is $q=k-k_0$. More generally, $q$ is the number of independent restrictions, or the codimension of the null parameter space in the unrestricted parameter space.

Large-sample calibration: Suppose the null parameter is an interior point of a regular, identifiable model; the model is differentiable with nonsingular Fisher information; the observations satisfy the usual independent sampling conditions; and the null restrictions define a smooth parameter space of codimension $q$. Under the null, the likelihood-ratio statistic has the asymptotic distribution $$ T_{\mathrm{LR}}\xrightarrow{d}\chi^2_q. $$ This is the likelihood-ratio form of Wilks' theorem. Its use requires regularity: the parameter should not be on a boundary, and the null should not cause loss of identifiability or other model singularities.

The chi-square reference distribution is a large-sample approximation, not generally an exact finite-sample distribution. For an observed statistic $t_{\mathrm{obs}}$, the approximate p-value is the upper-tail probability

p_{\mathrm{LR}}\approx P\!\left(\chi^2_q\geq t_{\mathrm{obs}}\right) =1-F_{\chi^2_q}(t_{\mathrm{obs}}).

A small p-value means that, if the null model were true, a statistic at least this large would be unusual under the stated approximation. It is not the probability that the null hypothesis is true.

Worked Likelihood-Ratio Tests

Worked Example: A Bernoulli Success Probability

Let $X_1,\ldots,X_{20}$ be independent Bernoulli observations with success probability $p$. Test $H_0:p=0.5$ against the unrestricted alternative $0<p<1$. The observed number of successes is $x=14$.

The likelihood, ignoring the binomial coefficient that is constant in $p$, is $L(p)=p^{14}(1-p)^6$. Under the unrestricted model, the maximum likelihood estimate is $\hat p=14/20=0.7$. Under the null, $p=0.5$. Therefore

T_{\mathrm{LR}} =2\left\{14\log(0.7)+6\log(0.3) -\left[14\log(0.5)+6\log(0.5)\right]\right\} =2\left\{14\log(1.4)+6\log(0.6)\right\}.

Using $\log(1.4)\approx0.336472$ and $\log(0.6)\approx-0.510826$, the log-likelihood difference is $14(0.336472)+6(-0.510826)\approx4.710608-3.064956=1.645652$. Hence $T_{\mathrm{LR}}\approx3.2913$. The unrestricted model has one free parameter and the null model has none, so $q=1$. The large-sample p-value is $P(\chi^2_1\geq3.2913)\approx0.070$. At the 5% level, this approximation does not reject $H_0$.

Worked Example: Testing a Normal Mean with Unknown Variance

Suppose $X_1,\ldots,X_{10}$ are independent $N(\mu,\sigma^2)$ observations, with both $\mu$ and $\sigma^2$ unknown. Test $H_0:\mu=0$ against the unrestricted alternative. The observed summaries are $\sum_i X_i=5$ and $\sum_i X_i^2=20$.

Under the unrestricted model, $\hat\mu=5/10=0.5$, and the residual sum of squares is $\mathrm{RSS}_1=\sum_i(X_i-\hat\mu)^2=20-5^2/10=17.5$. Under the null, the mean is fixed at zero, so $\mathrm{RSS}_0=\sum_iX_i^2=20$. In either fit, the variance is maximized at the corresponding residual sum of squares divided by $n$. Substituting these variance estimates into the normal log likelihood gives

T_{\mathrm{LR}} =n\log\left(\frac{\mathrm{RSS}_0}{\mathrm{RSS}_1}\right) =10\log\left(\frac{20}{17.5}\right) =10\log(8/7) \approx10(0.133531)=1.3353.

The unrestricted model has two parameters, $\mu$ and $\sigma^2$; the null model has one, $\sigma^2$. Thus $q=1$, not two: the variance is estimated in both models and is not a restriction being tested. The approximate p-value is $P(\chi^2_1\geq1.3353)\approx0.248$, which is not strong evidence against a zero mean.

Worked Example: Comparing Two Poisson Rates

Suppose independent counts $Y_1\sim\operatorname{Poisson}(\lambda_1)$ and $Y_2\sim\operatorname{Poisson}(\lambda_2)$ are observed over equal exposures of one unit. The counts are $y_1=4$ and $y_2=12$. Test the null hypothesis of a common rate, $H_0:\lambda_1=\lambda_2$, against separate unrestricted rates.

Under the unrestricted model, the maximum likelihood estimates are $\hat\lambda_1=4$ and $\hat\lambda_2=12$. Under the common-rate null, the estimate is the pooled rate $\hat\lambda=(4+12)/2=8$. The factorial terms in the Poisson likelihood cancel from the log-likelihood difference, giving

T_{\mathrm{LR}} =2\left\{4\log(4/8)+12\log(12/8)\right\} =2\left\{4\log(0.5)+12\log(1.5)\right\}.

Since $4\log(0.5)\approx-2.772589$ and $12\log(1.5)\approx4.865581$, the difference inside braces is approximately $2.092992$, and $T_{\mathrm{LR}}\approx4.1860$. The unrestricted model has two rates, while the null has one common rate, so $q=1$. The approximate p-value is $P(\chi^2_1\geq4.1860)\approx0.041$. The data provide evidence against equal rates at the 5% level under this large-sample calibration.

Degrees of Freedom and Practical Interpretation

The degrees of freedom count the independent restrictions imposed by the null, not the total number of parameters in the model. For example, testing two coefficients jointly to be zero imposes two restrictions, even if the model also contains an intercept and several nuisance parameters. If restrictions are redundant, count only the independent ones. A parameter estimated in both fits does not add a degree of freedom merely because it is present.

In practice, the analysis follows a short sequence:

Fit both models.
Maximize the same likelihood over the unrestricted and null parameter spaces.

Form the statistic.
Subtract the restricted maximum log likelihood from the unrestricted maximum and multiply by two.

Count restrictions.
Find the difference in the numbers of free parameters, or the codimension of the null space.

Calibrate and interpret.
Use the upper tail of $\chi^2_q$ as a large-sample p-value, while checking whether the regularity conditions are plausible.

Common Mistakes

Reversing the likelihood ratio. Defining the ratio as restricted likelihood divided by unrestricted likelihood ensures it is at most one. Equivalently, use $2(\ell_{\text{unrestricted}}-\ell_{\text{restricted}})$, which is nonnegative. Reversing the subtraction can produce a negative statistic.
Counting all model parameters as degrees of freedom. The chi-square degrees of freedom are the number of independent restrictions, not the unrestricted model’s parameter count.
Treating the calibration as exact. The chi-square p-value is an asymptotic approximation under regularity conditions. Small samples or boundary and nonidentifiability problems can make it inaccurate.
Comparing nonnested models with this reference automatically. The standard nested-model calibration stated here depends on the null model being contained in the alternative and on the regularity assumptions. It does not apply simply because two models have maximized likelihoods.

Key takeaway: For nested models, twice the unrestricted-minus-restricted maximum log likelihood is the likelihood-ratio statistic. Under regular null conditions, its large-sample reference distribution is chi-square with degrees of freedom equal to the number of independent restrictions.

Check Your Understanding

Answer each question by identifying the model comparison and the relevant calculation.

Why must the likelihood-ratio statistic be nonnegative when the null model is nested in the alternative?
A null model fixes two independent coefficients at zero, while all other parameters remain unrestricted. What are the asymptotic chi-square degrees of freedom?
In the normal-mean example, why does estimating the variance in both models not make the degrees of freedom equal to two?
If the unrestricted and restricted maximum log likelihoods are $-48.2$ and $-50.0$, respectively, calculate the likelihood-ratio statistic.
What does a likelihood-ratio p-value of 0.03 mean under the large-sample calibration, and what does it not mean?