Regular Likelihood Tests Under Smooth Null Restrictions

Three Likelihood Tests for One Null Hypothesis

A regular parametric model often allows the same null hypothesis to be examined in three ways. The Wald test measures how far the unrestricted estimate lies from the null; the score test asks whether the likelihood would increase steeply if we moved away from the best fit under the null; and the likelihood-ratio test compares the best likelihoods under the unrestricted and restricted models. Under suitable regularity conditions, their statistics have the same asymptotic chi-square reference distribution under the null.

This shared reference distribution does not make the statistics identical in a finite sample. The tests use different estimates and different information, so their numerical values—and sometimes their practical behavior—can differ.

Definition: Let $X_1,\ldots,X_n$ be an independent, identically distributed sample from a parametric model with parameter $\theta\in\Theta\subset\mathbb{R}^p$. A smooth null hypothesis has the form $$ H_0:g(\theta)=0, $$ where $g:\Theta\to\mathbb{R}^q$ is continuously differentiable and its Jacobian matrix $G(\theta)=\partial g(\theta)/\partial\theta^\mathsf{T}$ has rank $q$ near the true null value. Thus the null imposes $q$ locally independent restrictions. The alternative allows $\theta$ to vary freely in $\Theta$.

Write the sample log-likelihood as $\ell_n(\theta)=\sum_{i=1}^n\log f_\theta(X_i)$, where $f_\theta$ is the density or mass function. The unrestricted maximum likelihood estimate maximizes $\ell_n(\theta)$ over $\Theta$; the restricted maximum likelihood estimate maximizes it over $\{\theta\in\Theta:g(\theta)=0\}$. Denote these by $\hat\theta$ and $\tilde\theta$, respectively. The restricted estimate is not generally obtained by simply substituting the null value into every component of the unrestricted estimate: nuisance parameters must still be fitted.

Score and Fisher information: The sample score is $U_n(\theta)=\partial\ell_n(\theta)/\partial\theta$. Under the usual differentiability and integrability conditions, the Fisher information for one observation is $I(\theta)=E_\theta[U_1(\theta)U_1(\theta)^\mathsf{T}]$, and the sample Fisher information is $I_n(\theta)=nI(\theta)$. When the information identity applies, $I(\theta)=-E_\theta[\partial^2\log f_\theta(X_1)/\partial\theta\partial\theta^\mathsf{T}]$. Information matrices used in a statistic must be nonsingular in the relevant directions.

The Three Statistics

The standard forms below use the sample information $I_n$. In applications, it may be replaced by a consistent observed-information estimate. The degrees of freedom are the number $q$ of independent restrictions, not necessarily the number of components in $\theta$.

Wald statistic: For a smooth restriction $g(\theta)=0$, a standard Wald statistic is $$ W_n=g(\hat\theta)^\mathsf{T} \left\{G(\hat\theta)I_n(\hat\theta)^{-1}G(\hat\theta)^\mathsf{T}\right\}^{-1} g(\hat\theta). $$ It evaluates the estimated restriction using the estimated covariance of $g(\hat\theta)$.

For the score statistic, it is useful to express the model in local coordinates $\theta=(\psi^\mathsf{T},\lambda^\mathsf{T})^\mathsf{T}$, with $\psi\in\mathbb{R}^q$ and nuisance parameter $\lambda\in\mathbb{R}^{p-q}$, so that the null is $\psi=\psi_0$. Fit $\lambda$ under this null, giving $\tilde\lambda$. Partition the sample information into blocks $I_{\psi\psi},I_{\psi\lambda},I_{\lambda\psi},I_{\lambda\lambda}$.

Score statistic: Evaluate the score for $\psi$ and the sample information at $(\psi_0,\tilde\lambda)$. The efficient information for $\psi$, after accounting for nuisance estimation, is the Schur complement $$ I_{\psi\psi\cdot\lambda} =I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}. $$ The score statistic is $$ S_n=U_\psi(\psi_0,\tilde\lambda)^\mathsf{T} I_{\psi\psi\cdot\lambda}(\psi_0,\tilde\lambda)^{-1} U_\psi(\psi_0,\tilde\lambda). $$

Likelihood-ratio statistic: Compare the maximized log-likelihoods with $$ R_n=2\{\ell_n(\hat\theta)-\ell_n(\tilde\theta)\}. $$ The unrestricted maximum cannot have a smaller log-likelihood than the restricted maximum, so $R_n\geq 0$.

Why the Chi-Square Reference Distribution Applies

The following results concern behavior under a true null value $\theta_0$ in the interior of the parameter space. Assume an identifiable, dominated model with common support; differentiability of the log-likelihood; finite, nonsingular Fisher information; and the moment and uniformity conditions needed for the score central limit theorem, likelihood expansion, and consistency and asymptotic normality of the maximum likelihood estimates. Also assume the restriction has constant Jacobian rank $q$, and the restricted maximum exists locally and is consistent. These are substantive regularity requirements: a boundary null, singular information, or nonidentifiability may invalidate the conclusions.

Theorem: Under these regularity conditions, if $H_0$ is true, then each of $W_n$, $S_n$, and $R_n$ converges in distribution to $\chi^2_q$. Consequently, a large-sample level-$\alpha$ test rejects when its statistic exceeds the $1-\alpha$ quantile of $\chi^2_q$.

For the Wald statistic, the result follows from the asymptotic normality of the unrestricted estimate and the delta method. Specifically, under the null, $$ \sqrt{n}\,g(\hat\theta)\xrightarrow{d}N\left(0,G(\theta_0)I(\theta_0)^{-1}G(\theta_0)^\mathsf{T}\right). $$ The covariance matrix is positive definite because $G$ has rank $q$ and $I$ is positive definite. Replacing the population quantities by consistent estimates does not change the limit. A centered normal vector, multiplied on both sides by the inverse of its covariance matrix, gives a chi-square variable with degrees of freedom equal to the vector dimension $q$. This proves the Wald result.

For the score test, the information adjustment for nuisance parameters is essential. The score component for $\psi$, after projecting out its linear association with the nuisance score, has covariance $I_{\psi\psi\cdot\lambda}$. Under the null, its normalized version converges to a $q$-dimensional standard normal vector; evaluating the nuisance parameter at its restricted estimate gives the corresponding feasible score statistic. Its squared length therefore converges to $\chi^2_q$.

The likelihood-ratio result follows from the local quadratic likelihood expansion. In local coordinates, write a nearby parameter as $\theta_0+h/\sqrt n$. For bounded $h$, regularity gives $$ \ell_n(\theta_0+h/\sqrt n)-\ell_n(\theta_0) =h^\mathsf{T}\Delta_n-\tfrac12 h^\mathsf{T}I(\theta_0)h+o_p(1), \qquad \Delta_n=\frac{U_n(\theta_0)}{\sqrt n}\xrightarrow{d}N(0,I(\theta_0)). $$ Maximizing this quadratic over all local directions and over the null tangent space gives a difference equal to one half the squared length of the $q$ normal components perpendicular to that tangent space. The factor 2 in $R_n$ cancels the one half. The resulting squared length converges to $\chi^2_q$. This establishes the likelihood-ratio result.

Worked Examples

Worked Example: Testing a Normal Mean with Unknown Variance

Let $X_1,\ldots,X_{10}$ be normal observations with unknown mean $\mu$ and variance $\sigma^2$, and test $H_0:\mu=0$. Suppose the observed values consist of five $1.5$'s and five $-0.5$'s. Their sample mean is $\bar X=0.5$, and their sum of squared deviations from the mean is $10$.

The unrestricted estimates are $\hat\mu=0.5$ and $\hat\sigma^2=10/10=1$. Under the null, the restricted estimate of the nuisance variance is $$ \tilde\sigma^2=\frac{\sum_i X_i^2}{10} =\frac{5(1.5^2)+5(0.5^2)}{10} =\frac{12.5}{10}=1.25. $$

For the Wald statistic, the estimated variance of $\hat\mu$ is $\hat\sigma^2/10=0.1$, so $$ W_{10}=\frac{(0.5-0)^2}{0.1}=2.5. $$ For the score statistic, the normal-model mean and variance scores are orthogonal at $\mu=0$. The mean score is $10\bar X/\tilde\sigma^2=4$, and its information is $10/\tilde\sigma^2=8$. Thus $S_{10}=4^2/8=2$. For the likelihood ratio, $$ R_{10}=10\log\left(\frac{\tilde\sigma^2}{\hat\sigma^2}\right) =10\log(1.25)\approx2.2314. $$ All three use an asymptotic $\chi^2_1$ reference, but their finite-sample values are not the same.

Worked Example: A Binomial Success Probability

Suppose $Y\sim\operatorname{Binomial}(20,p)$, and the observed number of successes is $y=14$. Test $H_0:p=0.5$ against the unrestricted alternative $0<p<1$. The unrestricted estimate is $\hat p=14/20=0.7$; the restricted estimate is $0.5$.

The Wald statistic uses estimated variance $\hat p(1-\hat p)/20=0.7(0.3)/20=0.0105$: $$ W_{20}=\frac{(0.7-0.5)^2}{0.0105}\approx3.8095. $$ The score at the null is $U(p)=y/p-(20-y)/(1-p)$, so $U(0.5)=28-12=16$. The null information is $20/[0.5(0.5)]=80$, giving $$ S_{20}=\frac{16^2}{80}=3.2. $$ The log-likelihood ratio is $$ R_{20}=2\left\{14\log\left(\frac{0.7}{0.5}\right)+6\log\left(\frac{0.3}{0.5}\right)\right\} =2\{14\log(1.4)+6\log(0.6)\}\approx3.2913. $$ For example, $14\log(1.4)\approx4.7106$ and $6\log(0.6)\approx-3.06495$, giving the displayed value. The large-sample reference is $\chi^2_1$.

Worked Example: Testing Equality of Two Poisson Rates

Let $Y_1$ and $Y_2$ be independent Poisson counts, each observed over exposure 10. Suppose $y_1=12$ and $y_2=8$. Test $H_0:\lambda_1=\lambda_2$. Under the unrestricted model, the rate estimates are $\hat\lambda_1=12/10=1.2$ and $\hat\lambda_2=8/10=0.8$. Under the null, the common restricted estimate is $\tilde\lambda=(12+8)/20=1$.

The restriction is $g(\lambda_1,\lambda_2)=\lambda_1-\lambda_2=0$. The estimated variance of the difference is $1.2/10+0.8/10=0.2$, so the Wald statistic is $W_{20}=0.4^2/0.2=0.8$. For the score statistic, use log-rate coordinates $\alpha=(\log\lambda_1+\log\lambda_2)/2$ and $\delta=(\log\lambda_1-\log\lambda_2)/2$, so equality is $\delta=0$. At the restricted fit, the score for $\delta$ is $y_1-y_2=4$; its information, after accounting for the common log-rate nuisance parameter, is $20$. Thus $S_{20}=4^2/20=0.8$.

The likelihood-ratio statistic is $$ R_{20}=2\{12\log(1.2)+8\log(0.8)\}\approx2(2.18785-1.78515)=0.8054. $$ The three values are close here, and each is compared with $\chi^2_1$ in a large-sample test.

Common Pitfall and Practical Interpretation

A common mistake is to use the unrestricted estimate in the score test, or to use the restricted estimate in the Wald test. That reverses the logic of the tests: the score test evaluates the slope at the best fit under the null, while the Wald test measures the unrestricted estimate's distance from the null. The likelihood-ratio test compares both maximized fits. Keep the estimation step and information matrix consistent with the statistic being calculated.

The chi-square calibration is an approximation, justified as sample size increases under the stated regularity conditions. It does not guarantee accurate small-sample tail probabilities. Nor does the shared limiting distribution imply that the three tests are interchangeable in every finite sample. The earlier material on Fisher information and nuisance parameters is particularly useful for understanding why the score test uses efficient information rather than just the information block for the tested parameter.

Key takeaway: For $q$ smooth, locally independent null restrictions in a regular model, Wald, score, and likelihood-ratio statistics are all asymptotically $\chi^2_q$ under the null. Their defining distinction is which fit supplies the estimate and information: unrestricted for Wald, restricted for score, and both for the likelihood ratio.

Check Your Understanding

Use the definitions and conditions in this tutorial to answer the following questions.

What does the rank condition on $G(\theta)$ ensure about the number of restrictions?
Which estimate is used to evaluate the score statistic, and why must nuisance parameters be fitted under the null?
In the binomial example, verify the score at $p=0.5$ from $U(p)=y/p-(n-y)/(1-p)$.
Why does the likelihood-ratio statistic have a factor of 2?
Name one situation in which the stated chi-square approximation may fail.