Likelihood-Ratio Tests for Linear Restrictions

Why Compare Restricted and Unrestricted Fits?

A linear restriction on regression coefficients specifies a smaller model inside a larger one. For example, testing whether a treatment coefficient is zero asks whether the data support the full model more strongly than the model in which that coefficient is fixed at zero. A likelihood-ratio test answers this by comparing the best fit available under the null with the best fit available without the restriction.

The comparison uses the maximized log-likelihoods, not the values of coefficients alone. The unrestricted model can fit at least as well because it includes every parameter value allowed by the restricted model. The size of the improvement is then calibrated against a chi-square distribution when the usual large-sample regularity conditions hold.

Definition and Test Statistic

Definition: Let \(\beta\in\mathbb{R}^p\) denote the GLM coefficient vector, and let \(C\) be a fixed \(q\times p\) matrix of rank \(q\). A system of \(q\) independent linear restrictions is \(H_0:C\beta=d\), where \(d\in\mathbb{R}^q\) is fixed. The unrestricted model allows \(\beta\) to vary freely; the restricted model allows only values satisfying \(C\beta=d\). Write \(\ell_U\) and \(\ell_R\) for the maximized log-likelihoods in these models, including any nuisance parameters that are estimated in both. The likelihood-ratio statistic is \(G^2=2(\ell_U-\ell_R)\).

Equivalently, if \(L_U\) and \(L_R\) are the maximized likelihoods, \(G^2=-2\log(L_R/L_U)\). The likelihood ratio in this expression compares the two maximized likelihoods. If a model’s likelihood contains terms common to both fits, those terms may cancel in the difference, but only after the same likelihood has been used for both.

When the null is true, the model is correctly specified and regular, the parameter is identifiable and interior, and the sample size is sufficiently large, Wilks’ theorem gives an approximate \(\chi^2_q\) reference distribution for \(G^2\). The degrees of freedom are \(q=\operatorname{rank}(C)\), not necessarily the number of written equations if some restrictions are redundant. This is a large-sample calibration; in small samples the reference distribution need not be accurate.

Worked Binomial Comparison

Worked Example: Testing a Logistic Group Effect

Suppose two independent groups each contain 20 people. The first group has 8 successes and the second has 14. In a logistic regression with an intercept and a binary group indicator, test whether the group coefficient is zero. Under the null, both groups have the same success probability; under the alternative, each group can have its own probability.

Unrestricted fit. The maximum-likelihood success probabilities are the observed proportions: \(\hat p_1=8/20=0.40\) and \(\hat p_2=14/20=0.70\). The unrestricted log-likelihood, apart from binomial-coefficient terms common to both models, is \(8\log(0.40)+12\log(0.60)+14\log(0.70)+6\log(0.30)\).

Restricted fit. Under a common success probability, the pooled estimate is \(\hat p_R=(8+14)/40=0.55\). The corresponding log-likelihood, omitting the same common terms, is \(22\log(0.55)+18\log(0.45)\).

Subtracting the restricted log-likelihood from the unrestricted one and multiplying by two gives

\begin{aligned} G^2 &=2\left[ 8\log\left(\frac{0.40}{0.55}\right) +12\log\left(\frac{0.60}{0.45}\right) +14\log\left(\frac{0.70}{0.55}\right) +6\log\left(\frac{0.30}{0.45}\right) \right]\\ &=2\left[-2.5476+3.4522+3.3763-2.4328\right]\\ &\approx 3.6961. \end{aligned}

There is one independent restriction, so the reference distribution is \(\chi^2_1\). The upper-tail probability is approximately \(0.0545\). Thus, at a 5% significance level, this test does not reject the equal-probability null. The result is close to the threshold, so it is better to report the statistic and p-value than to describe the evidence as either decisive or absent.

Worked Poisson Comparison

Worked Example: Testing Equality of Two Poisson Rates

Suppose two groups are observed for equal exposure of 30 units each, with 10 events in the first group and 20 in the second. Use a Poisson regression with an intercept and a binary group indicator to test whether the group coefficient is zero. This restriction says that the two event rates are equal.

Unrestricted fit. The maximum-likelihood rate estimates are \(10/30=1/3\) and \(20/30=2/3\). For a Poisson count \(y\) with mean \(\mu\), the log-likelihood is \(y\log(\mu)-\mu-\log(y!)\). The factorial terms are identical between the fits for each observation, so they cancel in the log-likelihood difference. The exposure terms also sum to the same total fitted mean, \(10+20=30\), in both models.

Restricted fit. With equal rates, the pooled estimate is \(30/60=1/2\), giving fitted means of 15 events in each group. Therefore

\begin{aligned} G^2 &=2\left[ 10\log\left(\frac{1/3}{1/2}\right) +20\log\left(\frac{2/3}{1/2}\right) \right]\\ &=2\left[10\log\left(\frac23\right)+20\log\left(\frac43\right)\right]\\ &=2\left[10(-0.405465)+20(0.287682)\right]\\ &=2\left[-4.054650+5.753640\right]\\ &\approx 3.3980. \end{aligned}

The null imposes one independent restriction, so \(G^2\) is compared with \(\chi^2_1\); its upper-tail probability is about \(0.065\). At the 5% level, the test does not reject equal rates. This calculation illustrates why each logarithm must be multiplied by its observed count: the contribution \(10\log(2/3)\), for example, is about \(-4.055\), not \(-0.405\).

Why the Chi-Square Reference Appears

The chi-square calibration follows from the local quadratic shape of the log-likelihood near the true parameter. The derivation below states the conditions explicitly. Let the full parameter be \(\theta\in\mathbb{R}^p\), so it can include regression coefficients and any regular nuisance parameters being estimated. Suppose the true value \(\theta_0\) satisfies \(C\theta_0=d\), where \(C\) has rank \(q\).

Regularity conditions for the derivation: The model is correctly specified and identifiable; \(\theta_0\) is an interior point; the log-likelihood is sufficiently smooth; the per-observation Fisher information \(I\) at \(\theta_0\) is finite and positive definite; and the local likelihood admits the uniform quadratic expansion below on bounded sets. The unrestricted and restricted maximum-likelihood estimators are consistent and root-\(n\) bounded. The sample size tends to infinity with fixed \(p\) and \(q\).

Under these conditions, for bounded \(h\in\mathbb{R}^p\), the local log-likelihood expansion is

\ell_n(\theta_0+h/\sqrt n)-\ell_n(\theta_0) =h^\mathsf{T}\Delta_n-\frac12h^\mathsf{T}Ih+o_p(1), \qquad \Delta_n\xrightarrow{d}N(0,I).

Here \(I\) is the limiting information per observation and \(\Delta_n\) is the normalized score. Under the null, the local parameter \(h\) must satisfy \(Ch=0\). We can therefore compare the maximum of the quadratic expression over all \(h\) with its maximum over that linear subspace.

Whiten the quadratic.
Set \(u=I^{1/2}h\) and \(z=I^{-1/2}\Delta_n\). Then \(z\xrightarrow{d}N(0,I_p)\), and the quadratic becomes \(u^\mathsf{T}z-\tfrac12\|u\|^2\).

Identify the null subspace.
The restriction \(Ch=0\) becomes \(CI^{-1/2}u=0\). Its solution space \(S\) has dimension \(p-q\), since \(C\) has rank \(q\).

Compare the two quadratic maxima.
Over all \(u\), the maximum is \(\tfrac12\|z\|^2\). Over \(S\), it is \(\tfrac12\|P_Sz\|^2\), where \(P_S\) is orthogonal projection onto \(S\).

The factor of two in \(G^2\) cancels the factor \(\tfrac12\) in those maxima. The difference is \(\|z\|^2-\|P_Sz\|^2=\|P_{S^\perp}z\|^2\). The orthogonal complement \(S^\perp\) has dimension \(q\). Since \(z\) converges to a standard multivariate normal vector, its projection onto a fixed \(q\)-dimensional subspace has \(q\) independent standard normal coordinates. The squared length of that projection has a \(\chi^2_q\) distribution. The uniform expansion and root-\(n\) boundedness ensure that the local quadratic comparison gives the asymptotic likelihood-ratio statistic. Hence, under the null, \(G^2\xrightarrow{d}\chi^2_q\). \(\square\)

Worked Gaussian Comparison

Worked Example: Testing a Gaussian Regression Slope

Consider three independent observations from \(Y_i\sim N(\beta_0+\beta_1x_i,1)\), with fixed covariates \(x=(-1,0,1)\) and observed responses \(y=(1,2,4)\). The variance is known to be 1. Test \(H_0:\beta_1=0\), allowing the intercept to be estimated in both models.

Unrestricted fit. The least-squares slope is \(\hat\beta_1=\sum_i(x_i-\bar x)(y_i-\bar y)/\sum_i(x_i-\bar x)^2\). Here \(\bar x=0\), \(\bar y=7/3\), the numerator is \((-1)(1)+0(2)+1(4)=3\), and the denominator is \(1+0+1=2\). Thus \(\hat\beta_1=3/2\) and \(\hat\beta_0=7/3\). The fitted values are \(5/6,7/3,23/6\), so the residuals are \(1/6,-1/3,1/6\), and the residual sum of squares is \(1/36+1/9+1/36=1/6\).

Restricted fit. With slope zero, the fitted value for every observation is the sample mean \(7/3\). The residuals are \(-4/3,-1/3,5/3\), and the residual sum of squares is \(16/9+1/9+25/9=42/9=14/3\).

For known variance 1, the Gaussian log-likelihood is a common constant minus half the residual sum of squares. Therefore

G^2 =2(\ell_U-\ell_R) =\operatorname{SSE}_R-\operatorname{SSE}_U =\frac{14}{3}-\frac{1}{6} =\frac{27}{6} =4.5.

There is one restriction, so the large-sample reference is \(\chi^2_1\), with an upper-tail probability of about \(0.034\). In this known-variance normal model with fixed design, the statistic also has an exact \(\chi^2_1\) distribution under the null: the fitted slope is normal with variance \(1/\sum_i(x_i-\bar x)^2=1/2\), and its squared standardized value is \((3/2)^2/(1/2)=4.5\).

Interpreting and Reporting the Test

For an observed statistic \(g\), the large-sample p-value is \(P(\chi^2_q\geq g)\). A small value indicates that the unrestricted fit improves the log-likelihood more than would usually be expected from the \(q\) additional free directions when the restrictions hold. It does not measure the size or practical importance of the coefficient effects. Report the restrictions, \(G^2\), the degrees of freedom, and the p-value, and interpret the result in the context of the model.

Result: The unrestricted fit cannot have a lower maximized likelihood. The restricted parameter set is a subset of the unrestricted parameter set. Therefore, every parameter value available to the restricted fit is also available to the unrestricted fit. Taking maxima gives \(L_U\geq L_R\), and hence \(G^2=2\log(L_U/L_R)\geq0\). A negative computed value signals a fitting, optimization, or likelihood-comparison error. \(\square\)

This nonnegativity is a useful diagnostic, not a guarantee that the optimization has succeeded. Both models must use the same observations and the same likelihood convention. For example, comparing one fit’s full likelihood with another fit’s likelihood after dropping data-dependent terms can give a meaningless difference.

Common Mistakes

Using the wrong subtraction order. The statistic is twice the unrestricted maximum log-likelihood minus the restricted maximum log-likelihood. Reversing the order produces a nonpositive quantity, contrary to the nested-model comparison.
Counting written equations instead of independent restrictions. The degrees of freedom are the rank of \(C\). Repeated or linearly dependent rows do not add restrictions.
Forgetting counts in grouped likelihood terms. In a binomial or Poisson log-likelihood, each logarithm is multiplied by its observed count. Keep the count attached when evaluating each term.
Treating the reference distribution as automatically exact. In general GLMs the chi-square result is asymptotic and relies on regularity conditions. Small samples, boundary parameters, or nonidentifiability may invalidate the usual calibration.

Key takeaway: Fit the model both with and without the linear restrictions, calculate \(G^2=2(\ell_U-\ell_R)\), and compare it with \(\chi^2_q\), where \(q\) is the number of independent restrictions. The approximation is justified by regular large-sample likelihood theory, not merely by the fact that two models are nested.

Check Your Understanding

Use the definitions and derivation above to answer these questions.

A null hypothesis imposes four equations on the coefficient vector, but one equation is a linear combination of the other three. What degrees of freedom should the likelihood-ratio test use?
Why must the unrestricted maximized likelihood be at least as large as the restricted maximized likelihood?
In the two-group binomial example, what probability is used under the restricted model, and how is it estimated?
What is the likelihood-ratio statistic for the Poisson example before rounding, and how many degrees of freedom does its reference distribution have?
Name two regularity conditions needed for the usual asymptotic chi-square interpretation.