Likelihood-Ratio Tests for Nested Generalized Models

Why Compare Nested Models by Their Likelihoods?

A generalized linear model can be made more restrictive by fixing one or more of its coefficients or by imposing smooth constraints on its parameters. The likelihood-ratio test asks whether the increase in maximized likelihood from relaxing those restrictions is large relative to what random variation would typically produce under the restricted model. Its central advantage is that it compares the fitted likelihoods directly, while its asymptotic reference distribution depends on how many independent restrictions were imposed.

The result below concerns a genuine likelihood for the observed data. It applies to regular nested models, including generalized linear models with a correctly specified probability distribution. It does not automatically apply to a quasi-likelihood that specifies only a mean and variance.

Definition and Hypotheses

Definition: Let $\mathcal{M}_0$ be a restricted model contained in a larger model $\mathcal{M}_1$, with parameter spaces of dimensions $p_0$ and $p_1$, where $p_0<p_1$. Write $\ell_0$ and $\ell_1$ for the log-likelihoods maximized over the respective parameter spaces. The likelihood-ratio statistic for testing the restricted model against the larger model is $$ T=2(\ell_1-\ell_0) =2\left\{\sup_{\theta\in\mathcal{M}_1}\ell(\theta)-\sup_{\theta\in\mathcal{M}_0}\ell(\theta)\right\}. $$ The null hypothesis is that the true parameter belongs to $\mathcal{M}_0$; the alternative allows it to lie elsewhere in $\mathcal{M}_1$.

Because $\mathcal{M}_0$ is contained in $\mathcal{M}_1$, maximizing over the larger space cannot give a smaller log-likelihood, so $T\geq 0$. When the null is expressed by $r$ independent coefficient restrictions, the two models usually differ in dimension by $r=p_1-p_0$. A large value of $T$ is evidence against the restrictions.

For example, in a generalized linear model with coefficient vector $\beta$, the null might be $C\beta=c$, where $C$ has rank $r$. The alternative allows all coefficients in the larger model to vary. The degrees of freedom are determined by the rank of the restrictions, not by the number of observations or by the number of coefficients that happen to be nonzero in the fitted model.

Worked Likelihood-Ratio Calculations

Worked Example: A Slope Restriction in Logistic Regression

Suppose two groups each contain 10 independent binary observations. In group 0 there are 2 successes, and in group 1 there are 7. Model the success probability by $\operatorname{logit}(p_i)=\beta_0+\beta_1x_i$, where $x_i$ is the group indicator. Test $H_0:\beta_1=0$ against the model with an unrestricted slope.

Under the null, both groups share one success probability. Its maximum-likelihood estimate is the pooled proportion, $\hat p_0=(2+7)/20=0.45$. Under the alternative, the two group probabilities are estimated separately: $\hat p_{00}=2/10=0.20$ and $\hat p_{01}=7/10=0.70$. The binomial combinatorial terms are the same under both fits and cancel in the likelihood ratio.

The statistic is therefore $$ T=2\left[ 2\log\left(\frac{0.20}{0.45}\right) +8\log\left(\frac{0.80}{0.55}\right) +7\log\left(\frac{0.70}{0.45}\right) +3\log\left(\frac{0.30}{0.55}\right) \right]. $$

The four contributions inside the brackets are approximately $-1.6219$, $2.9975$, $3.0928$, and $-1.8184$. Their sum is $2.6500$, giving $T\approx5.30$. There is one restriction, so the asymptotic reference is $\chi^2_1$, with an upper-tail probability of approximately $0.021$. This is evidence against equal probabilities, though the small group counts make the quality of an asymptotic approximation worth considering.

Worked Example: Testing a Poisson Regression Slope

For each of two groups, suppose 100 independent Poisson counts are observed, with exposure 1 per observation. The group totals are 200 and 800. Let the group-specific means follow a log-linear model, $\log(\lambda_i)=\beta_0+\beta_1x_i$, with $x_i=0$ or $1$. Test whether the group effect is zero.

Under the null, both groups have the same rate. The pooled maximum-likelihood estimate is $\hat\lambda_0=(200+800)/200=5$. In the alternative model, the estimates are the group means, $\hat\lambda_0=200/100=2$ and $\hat\lambda_1=800/100=8$. For a Poisson count $y$, the log-likelihood contribution, apart from notation for the rate, is $y\log(\lambda)-\lambda-\log(y!)$. The factorial terms cancel between fits.

There are 100 observations in each group, so the rate terms in the log-likelihood give $$ T=2\left[200\log\left(\frac{2}{5}\right)+800\log\left(\frac{8}{5}\right)-100(2+8)+200(5)\right]. $$ The linear terms cancel because $100(2+8)=200(5)=1000$. Since $\log(0.4)\approx-0.916291$ and $\log(1.6)\approx0.470004$, $$ T=2\left[200(-0.916291)+800(0.470004)\right]\approx385.49. $$

The null imposes one restriction, so the usual reference is $\chi^2_1$. The statistic is far into its upper tail, providing strong evidence against a common rate, subject to the model and regularity assumptions.

Worked Example: A Normal-Response Model With Known Variance

Consider a normal generalized linear model with identity link and known variance $\sigma^2=1$. There are 20 independent observations at each of $x=0,1,2$; the observed response at each design point is respectively $0,1,3$. Compare the intercept-only model with the model that includes a slope.

The intercept-only estimate is the grand mean, $\hat\mu_0=(0+1+3)/3=4/3$. For one observation at each design point, the residual sum of squares is $$ (0-4/3)^2+(1-4/3)^2+(3-4/3)^2 =\frac{16+1+25}{9}=\frac{14}{3}. $$ With 20 observations at each point, the restricted residual sum of squares is $20(14/3)=280/3$.

For the model with intercept and slope, the fitted line has slope $3/2$ and intercept $-1/6$. Its fitted values are $-1/6,4/3,17/6$; the three squared residuals sum to $$ (1/6)^2+(-1/3)^2+(1/6)^2=\frac{1}{6}. $$ Thus the unrestricted residual sum of squares is $20(1/6)=10/3$.

For independent normal responses with known variance, the log-likelihood, up to a common constant, is minus the residual sum of squares divided by $2\sigma^2$. Consequently, $$ T=\frac{280/3-10/3}{1}=90. $$ The slope restriction contributes one degree of freedom, so its asymptotic reference distribution is $\chi^2_1$. This example shows the likelihood calculation explicitly; the chi-square calibration still relies on the large-sample regularity conditions, rather than following exactly from the statistic’s formula.

Why the Chi-Square Degrees of Freedom Equal the Number of Restrictions

The asymptotic result is a form of Wilks’ theorem. A precise version for the present setting is as follows. Suppose the true parameter is an interior point of the null model, which is locally a smooth subspace of codimension $r$. Assume the model is identifiable and has a genuine likelihood; the log-likelihood is sufficiently differentiable near the true parameter; the normalized information converges to a finite positive-definite matrix; and the likelihood admits the usual local asymptotic quadratic expansion. Also assume the unrestricted and restricted maximum-likelihood estimators are consistent, are root-$n$ local under the null, and do not lie on parameter-space boundaries with nonvanishing probability. Then, under the null, $$ T\xrightarrow{d}\chi^2_r. $$

For independent, identically distributed observations, these conditions are commonly established using differentiability and integrability assumptions on the likelihood derivatives and nonsingular Fisher information. For fixed-design generalized linear models, corresponding conditions are imposed on the sequence of designs and information matrices. The key point is that the likelihood behaves locally like a nonsingular quadratic function; the result is not guaranteed merely because the model is called a generalized linear model.

Result: Under the stated regularity conditions, if the null model is a smooth, nested submodel of codimension $r$, then its likelihood-ratio statistic converges in distribution to $\chi^2_r$ when the null is true. For linear coefficient restrictions, $r$ is the rank of the restriction matrix.

The reason for the result can be seen by using local coordinates. Write the parameter as $\theta=(\alpha,\beta)$, where $\alpha$ has dimension $p-r$ and $\beta$ has dimension $r$, so the null is $\beta=0$. A smooth restriction of full rank can be put in this form locally. Let $\theta_0=(\alpha_0,0)$ be the true null value. With the parameter displacement scaled as $h/\sqrt n$, the local log-likelihood expansion is $$ \ell_n(\theta_0+h/\sqrt n)-\ell_n(\theta_0) =h^\mathsf{T}\Delta_n-\frac12h^\mathsf{T}Ih+o_p(1), $$ where $\Delta_n\xrightarrow{d}N(0,I)$, and $I$ is the limiting information matrix at $\theta_0$. Partition $h=(h_\alpha,h_\beta)$, $\Delta_n=(\Delta_\alpha,\Delta_\beta)$, and $I$ into matching blocks.

For a fixed $h_\beta$, maximizing the quadratic expression over the nuisance displacement $h_\alpha$ gives $$ \hat h_\alpha=I_{\alpha\alpha}^{-1} \left(\Delta_\alpha-I_{\alpha\beta}h_\beta\right). $$ Substituting this value into the quadratic leaves, apart from a term that does not depend on $h_\beta$, $$ h_\beta^\mathsf{T}U_n-\frac12h_\beta^\mathsf{T}I_{\mathrm{eff}}h_\beta, \quad U_n=\Delta_\beta-I_{\beta\alpha}I_{\alpha\alpha}^{-1}\Delta_\alpha, \quad I_{\mathrm{eff}}=I_{\beta\beta}-I_{\beta\alpha}I_{\alpha\alpha}^{-1}I_{\alpha\beta}. $$

The restricted fit fixes $h_\beta=0$, whereas the unrestricted fit maximizes over $h_\beta$. Since $I_{\mathrm{eff}}$ is positive definite, the unrestricted maximum occurs at $h_\beta=I_{\mathrm{eff}}^{-1}U_n$, and its increase over the restricted maximum is $\tfrac12U_n^\mathsf{T}I_{\mathrm{eff}}^{-1}U_n$. Therefore the twice-log-likelihood difference satisfies $$ T=U_n^\mathsf{T}I_{\mathrm{eff}}^{-1}U_n+o_p(1). $$

Because $\Delta_n$ has limiting covariance $I$, direct calculation gives $U_n\xrightarrow{d}N(0,I_{\mathrm{eff}})$. Thus $I_{\mathrm{eff}}^{-1/2}U_n$ converges to an $r$-dimensional standard normal vector. Its squared Euclidean length has a $\chi^2_r$ distribution, proving the asserted limit.

Proof conclusion: The likelihood gain is asymptotically the squared length of an $r$-dimensional standardized normal vector. Hence $T\xrightarrow{d}\chi^2_r$. $\square$

Using the Test and Avoiding Common Errors

For an observed statistic $t$, the asymptotic p-value is $P(\chi^2_r\geq t)$. The test is an upper-tail test because maximizing the larger model can only increase the likelihood. The null and alternative must be fitted to the same observations with compatible likelihoods; constants that are identical under both fits may be omitted, but model-dependent terms may not.

Common mistake: counting coefficients rather than restrictions. A student may compare a model with six coefficients to one with three and use six degrees of freedom. The correction is to use the difference in dimensions, which is three here, or more generally the rank of the independent restrictions. Another error is to treat every calculated likelihood ratio as having an exact chi-square distribution. The chi-square result is asymptotic and can be unreliable with small samples, weak identification, or estimates near boundaries.

A further limitation is important in generalized modeling: if only a quasi-likelihood has been specified, there may be no full probability likelihood to maximize. In that case the likelihood-ratio statistic and Wilks’ theorem do not follow from the quasi-likelihood alone. Boundary hypotheses, such as a variance component equal to zero, also need special care because the usual interior-point argument above does not apply.

Likelihood-ratio testing is useful because it gives a common comparison for nested models across response distributions and link functions, while making the restrictions explicit. Its interpretation rests on two separate checks: whether the model comparison is genuinely nested, and whether the regularity conditions make the chi-square calibration credible.

Check Your Understanding

Answer each question using the likelihood-ratio definition and its regularity conditions.

A model with eight free parameters is compared with a nested model having five. What is the usual chi-square degrees of freedom, and what determines it?
Why must the unrestricted log-likelihood be at least as large as the restricted log-likelihood?
In the local likelihood argument, what role does the efficient information matrix $I_{\mathrm{eff}}$ play?
What feature of a boundary null hypothesis can prevent the standard Wilks chi-square result from applying?
Why is a likelihood-ratio test not automatically justified when an analysis specifies only a quasi-likelihood?