Deriving the Likelihood-Ratio Test

Why Compare Maximized Likelihoods?

A null hypothesis often says that a collection of model parameters obeys a restriction, while the alternative lets them vary freely. The likelihood-ratio test asks how much better the data can be explained when that restriction is removed. Its statistic is built from two fits to the same observations: the best fit allowed by the null and the best fit allowed by the larger model.

The comparison is especially useful in generalized linear models (GLMs), where the models being compared may differ by one or several coefficients. The statistic itself is a direct likelihood comparison; its familiar chi-square reference distribution is a large-sample result, not an automatic property of every finite-sample likelihood ratio.

Definition and Direction of the Comparison

Definition: Let $L(\theta)$ be the likelihood under a model with parameter $\theta\in\Theta$. Suppose the null hypothesis restricts the parameter to a subset $\Theta_0\subseteq\Theta$. The likelihood-ratio statistic is $$ \Lambda=\frac{\sup_{\theta\in\Theta_0}L(\theta)}{\sup_{\theta\in\Theta}L(\theta)}, $$ and the deviance form of the likelihood-ratio statistic is $$ G^2=-2\log\Lambda =2\left\{\sup_{\theta\in\Theta}\ell(\theta) -\sup_{\theta\in\Theta_0}\ell(\theta)\right\}. $$

Because $\Theta_0$ is contained in $\Theta$, the unrestricted maximum likelihood cannot be smaller than the null-constrained maximum. Thus $0\leq\Lambda\leq1$ and $G^2\geq0$. Large values indicate that imposing the null restriction causes a substantial loss in maximized log likelihood. The statistic uses the joint likelihood of the observations; constants common to both fits cancel in the difference.

For a GLM comparison, the null model must be nested in the alternative: every distribution allowed by the null must also be allowed by the larger model. If the null fixes $r$ independent parameter restrictions and the regularity conditions below hold, $G^2$ is asymptotically calibrated against a chi-square distribution with $r$ degrees of freedom. The count $r$ here is the number of locally independent restrictions, not necessarily the number of coefficients displayed in a particular parameterization.

Worked Likelihood Comparisons

Worked Example: Comparing Two Binomial GLMs

Two groups each contain 10 independent patients. Group A has 8 responses and group B has 4. Compare a null model with a common response probability $p$ to an alternative model with separate probabilities $p_A$ and $p_B$. The null is the special case $p_A=p_B$.

Under the alternative, the maximum likelihood estimates are $\hat p_A=8/10=0.8$ and $\hat p_B=4/10=0.4$. Under the null, the common estimate is the pooled proportion $\hat p=12/20=0.6$. Omitting binomial coefficients, which are identical in the two fits, the alternative log likelihood is $$ \ell_1=8\log(0.8)+2\log(0.2)+4\log(0.4)+6\log(0.6) \approx -11.7341. $$ The null log likelihood is $$ \ell_0=12\log(0.6)+8\log(0.4)\approx -13.4602. $$ Therefore $$ G^2=2(\ell_1-\ell_0) =2\{-11.7341-(-13.4602)\}\approx 3.4522. $$ There is one restriction, equality of the two probabilities, so the regular large-sample reference is $\chi^2_1$. Its 5% upper critical value is approximately $3.841$; this statistic does not exceed that value. The reference is approximate, and the small group sizes deserve attention when assessing its accuracy.

Worked Example: A Poisson Intercept Under a Specified Null

Suppose $Y_1,\ldots,Y_4$ are independent Poisson observations with a common mean $\lambda$, and the observed counts are $1,2,3,2$. Test $H_0:\lambda=2$ against the alternative that $\lambda$ is unrestricted and positive.

The unrestricted maximum likelihood estimate is the sample mean, $\hat\lambda=(1+2+3+2)/4=2$. This would give no likelihood improvement, so it cannot illustrate a nonzero statistic. Instead, consider the specified null $H_0:\lambda=3$, keeping the same data and unrestricted alternative. The null is a single fixed value, and the unrestricted estimate remains $\hat\lambda=2$.

The Poisson log likelihood is $\ell(\lambda)=\sum_i[y_i\log\lambda-\lambda-\log(y_i!)]$. The factorial terms cancel in the difference. Hence $$ \ell(2)-\ell(3) =8\log(2/3)-4(2-3) =8\log(2/3)+4 \approx 0.7563, $$ and $$ G^2=2\{\ell(2)-\ell(3)\}\approx 1.5126. $$ The null fixes one parameter, giving a $\chi^2_1$ asymptotic reference under regularity conditions. The calculation illustrates the likelihood comparison; with only four observations, the chi-square approximation may be poor, so it should not be mistaken for an exact finite-sample distribution.

Worked Example: Testing a Gaussian Mean

Let $Y_1,\ldots,Y_4$ be independent $N(\mu,4)$ observations, with known variance $4$. The data are $1,1,2,4$. Test $H_0:\mu=0$ against an unrestricted mean.

The sample mean is $\bar Y=2$, which maximizes the alternative likelihood. For a normal model with known variance $\sigma^2$, the log-likelihood difference between the maximum at $\bar Y$ and its value at a specified $\mu_0$ is $n(\bar Y-\mu_0)^2/(2\sigma^2)$. Here $\mu_0=0$, so $$ G^2=\frac{n(\bar Y-0)^2}{\sigma^2} =\frac{4(2)^2}{4}=4. $$ There is one restriction. The large-sample reference is $\chi^2_1$, whose 5% upper critical value is approximately $3.841$, so this statistic exceeds the critical value. In this particular normal model with known variance, the statistic under the null is exactly $\chi^2_1$, not merely asymptotically so: $\sqrt{n}\bar Y/\sigma$ is exactly standard normal under $H_0$, and its square has that chi-square distribution.

Why the Chi-Square Calibration Holds

The calibration follows from a local quadratic approximation to the log likelihood. The derivation below uses a regular, finite-dimensional model and a null that fixes $r$ coordinates while leaving nuisance coordinates unrestricted. A smooth null restriction of rank $r$ can be represented this way in suitable local coordinates.

Theorem: Let $\theta=(\theta_1^T,\theta_2^T)^T$ be a parameter of dimension $p$, where $\theta_1$ has dimension $r$. Suppose the true parameter $\theta_0$ is an interior point and satisfies $H_0:\theta_1=\theta_{10}$, with $\theta_2$ unrestricted. Assume the model is identifiable and regular, the Fisher information per observation converges to a positive-definite matrix $I$, the normalized score at $\theta_0$ converges in distribution to $N(0,I)$, and a uniform quadratic likelihood expansion holds in bounded root-$n$ neighborhoods. Assume also that the unrestricted and null-constrained maximum likelihood estimators exist, are consistent, and are root-$n$ tight. Then under $H_0$, $$ G^2\xrightarrow{d}\chi^2_r. $$

These assumptions summarize standard regular likelihood conditions: differentiability and adequate control of the likelihood derivatives, together with information growth sufficient for the score limit and the local expansion. They exclude, among other cases, a true parameter on a boundary or a singular information matrix. For independent observations, the normalized score can be written $\Delta_n=n^{-1/2}\,\partial\ell(\theta_0)/\partial\theta$; the assumptions give $\Delta_n\xrightarrow{d}N(0,I)$.

Proof by Local Quadratic Maximization

Write $h=\sqrt n(\theta-\theta_0)$. By the assumed uniform expansion, for bounded $h$, $$ \ell(\theta_0+h/\sqrt n)-\ell(\theta_0) =h^T\Delta_n-\frac12h^TIh+o_p(1). $$ Root-$n$ tightness of both estimators ensures their maximizing local coordinates are bounded in probability. Consequently the likelihood maxima, up to an $o_p(1)$ error in their difference, are obtained by maximizing the displayed quadratic form over all $h$ and over the null tangent set $h_1=0$.

Partition the score and information according to $(\theta_1,\theta_2)$: $$ \Delta_n=\begin{pmatrix}\Delta_{1n}\\\Delta_{2n}\end{pmatrix}, \qquad I=\begin{pmatrix}I_{11}&I_{12}\\I_{21}&I_{22}\end{pmatrix}. $$ Since $I$ is positive definite, $I_{22}$ is positive definite. The unrestricted quadratic has its maximum at $h=I^{-1}\Delta_n$, with maximum value $\frac12\Delta_n^TI^{-1}\Delta_n$. Under the null, $h_1=0$, and maximizing over $h_2$ gives $h_2=I_{22}^{-1}\Delta_{2n}$ and maximum value $\frac12\Delta_{2n}^TI_{22}^{-1}\Delta_{2n}$.

Define the Schur complement $J=I_{11}-I_{12}I_{22}^{-1}I_{21}$ and the adjusted score $U_n=\Delta_{1n}-I_{12}I_{22}^{-1}\Delta_{2n}$. Block-matrix completion of the square gives $$ \Delta_n^TI^{-1}\Delta_n-\Delta_{2n}^TI_{22}^{-1}\Delta_{2n} =U_n^TJ^{-1}U_n. $$ Thus twice the difference between the unrestricted and null-constrained quadratic maxima is $U_n^TJ^{-1}U_n$, up to $o_p(1)$.

Because $\Delta_n\xrightarrow{d}N(0,I)$, the linear transformation $U_n$ converges to a mean-zero normal vector with covariance $$ \operatorname{Var}(U)=I_{11}-I_{12}I_{22}^{-1}I_{21}=J. $$ Positive definiteness of $I$ implies positive definiteness of $J$. Therefore $J^{-1/2}U_n\xrightarrow{d}N(0,I_r)$, and its squared Euclidean length converges to $\chi^2_r$. The $o_p(1)$ remainder does not change this limiting distribution, proving the result. $\square$

Interpretation, Scope, and Common Pitfalls

The nuisance coordinates matter even though the null restrictions concern only $\theta_1$. The unrestricted and constrained fits both optimize over $\theta_2$; the adjusted score $U_n$ accounts for the information shared between the tested and nuisance parameters. Simply using the unadjusted score for $\theta_1$ would generally give the wrong variance and therefore the wrong calibration.

Reversing the ratio: using unrestricted likelihood divided by null likelihood and then multiplying its logarithm by $-2$ gives a nonpositive quantity. The standard statistic is $-2\log(L_0/L_{\mathrm{unrestricted}})$.
Comparing unlike fits: the two likelihoods must refer to the same observations and compatible models, with the null model nested in the alternative. A difference between unrelated maximized likelihoods is not this test.
Treating the approximation as exact: Wilks' theorem gives a limiting distribution under regularity conditions. It does not assert that every finite-sample GLM likelihood ratio is exactly chi-square.
Ignoring nonregularity: boundary parameters, nonidentifiability, or singular information can invalidate the stated calibration. A boundary case may have a different limiting distribution rather than $\chi^2_r$.

Key takeaway: The likelihood-ratio statistic is twice the unrestricted maximized log likelihood minus the null-constrained maximized log likelihood. Under a regular interior null with $r$ independent restrictions, its large-sample reference distribution is $\chi^2_r$.

Check Your Understanding

Work through these questions using the definitions and derivation above.

Why must the likelihood-ratio statistic be nonnegative for nested hypotheses?
In a regular model, what role does the nuisance-parameter block $I_{22}$ play in the adjusted score?
Suppose the null fixes two independent parameter coordinates and all regularity conditions hold. What is the large-sample reference distribution for $G^2$?
Why do the binomial coefficients cancel in the first worked example?
Name two conditions under which the chi-square calibration derived here may fail.