Wald, Score, and Likelihood-Ratio Limits

Three Ways to Measure Evidence Against a Smooth Null

For a regular model, a null hypothesis can be viewed as a restriction on the parameter rather than as a separate probability model. Wald, score, and likelihood-ratio tests measure departures from that restriction in different ways: Wald measures the distance from the unrestricted estimate to the null, the score test measures the slope of the likelihood at the restricted fit, and the likelihood-ratio test compares the best likelihoods under the unrestricted and restricted models. Under regularity conditions, all three have the same first-order chi-square limit under the null.

Here the observations $X_1,\ldots,X_n$ are independent and identically distributed, with parameter $\theta\in\Theta\subset\mathbb R^p$. Write $\ell_n(\theta)$ for the log-likelihood, $U_n(\theta)=\partial\ell_n(\theta)/\partial\theta$ for the score, and $I(\theta)$ for the Fisher information per observation. The total information is $I_n(\theta)=nI(\theta)$. The exponential-family setting supplies useful smoothness and information identities, but the limits below require the stated local regularity conditions as well.

Definition: A smooth null hypothesis is $H_0:g(\theta)=0$, where $g:\Theta\to\mathbb R^r$ is continuously differentiable near the true null value $\theta_0$, $g(\theta_0)=0$, and the $r\times p$ Jacobian $G(\theta_0)=\partial g(\theta_0)/\partial\theta'$ has rank $r$. Thus the null imposes $r$ locally independent restrictions.

For the standard limits, assume $\theta_0$ is an interior point, $I(\theta_0)$ is finite and positive definite, and the likelihood is sufficiently differentiable near $\theta_0$. Assume the score satisfies a central limit theorem, normalized observed information converges to $I(\theta_0)$, and a uniform local quadratic expansion of the log-likelihood holds. Also assume the unrestricted maximum likelihood estimator $\hat\theta$ and the restricted estimator $\tilde\theta$, which maximizes $\ell_n$ subject to $g(\theta)=0$, exist with probability tending to one, are consistent, and are root-$n$ consistent. These conditions rule out boundary and nonexistence complications and justify the local comparisons used below.

Definitions of the Three Statistics

Let $\hat\theta$ be the unrestricted maximum likelihood estimator and let $\tilde\theta$ be the restricted maximum likelihood estimator. The Wald statistic uses the estimated covariance of $g(\hat\theta)$. The score statistic evaluates the score at the restricted fit, where the restriction has not allowed the likelihood to reach its unrestricted maximum. The likelihood-ratio statistic compares the log-likelihoods at the two fits.

Definition: With information estimates consistent for $I(\theta_0)$, the three statistics can be written $$ W_n=g(\hat\theta)'\left\{G(\hat\theta)I_n(\hat\theta)^{-1}G(\hat\theta)'\right\}^{-1}g(\hat\theta),\qquad S_n=U_n(\tilde\theta)'I_n(\tilde\theta)^{-1}U_n(\tilde\theta),\qquad R_n=2\{\ell_n(\hat\theta)-\ell_n(\tilde\theta)\}. $$ Here the score statistic uses the full information matrix inverse; equivalently it measures the score in the $r$ directions normal to the null. Observed information may be used in place of expected information when it is consistent and nonsingular with probability tending to one.

The inverse in the Wald expression exists locally because $G(\theta_0)$ has full row rank and $I(\theta_0)$ is positive definite. The restricted score is generally not zero: at an interior constrained maximum it is normal to the restriction surface. Its quadratic form with the inverse information measures that normal component.

Why All Three Limits Are Chi-Square

Theorem: Under $H_0$ and the regularity conditions above, $$ W_n\xrightarrow{d}\chi^2_r,\qquad S_n\xrightarrow{d}\chi^2_r,\qquad R_n\xrightarrow{d}\chi^2_r. $$ Moreover, the three statistics are first-order equivalent: $W_n-S_n\xrightarrow{p}0$, $W_n-R_n\xrightarrow{p}0$, and $S_n-R_n\xrightarrow{p}0$.

A local-coordinate derivation gives both the degrees of freedom and the equivalence. In the rest of this derivation, set $I=I(\theta_0)$, $G=G(\theta_0)$, and define the standardized score $$ Z_n=I^{-1/2}\frac{U_n(\theta_0)}{\sqrt n}. $$ The score central limit theorem and information identity give $Z_n\xrightarrow{d}Z$, where $Z\sim N_p(0,I_p)$. For local parameters $\theta=\theta_0+I^{-1/2}h/\sqrt n$, the assumed uniform quadratic expansion is $$ \ell_n(\theta)-\ell_n(\theta_0)=h'Z_n-\frac12h'h+o_p(1) $$ uniformly for bounded $h$. Linearizing the constraint shows that its local form is $Ah=0$, where $A=GI^{-1/2}$. The matrix $A$ has rank $r$.

The unrestricted maximizer of the displayed quadratic is $h=Z_n$. Under the local constraint, it is the orthogonal projection of $Z_n$ onto $\ker(A)$: $$ \tilde h_n=P_TZ_n+o_p(1),\qquad P_T=I_p-A'(AA')^{-1}A. $$ The projection onto the orthogonal complement of this tangent space is $P_N=A'(AA')^{-1}A$. These conclusions follow by completing the square: $h'Z_n-\tfrac12h'h=\tfrac12Z_n'Z_n-\tfrac12\|h-Z_n\|^2$, so maximizing over $\ker(A)$ chooses the closest point in that space. The root-$n$ localization and uniform expansion ensure that these local quadratic maximizers describe the likelihood maximizers to $o_p(1)$.

For the likelihood ratio, the unrestricted maximum of the quadratic exceeds the constrained maximum by one half the squared length of the normal component. Consequently, $$ R_n=Z_n'P_NZ_n+o_p(1). $$ For the Wald statistic, $g(\hat\theta)=G(\hat\theta-\theta_0)+o_p(n^{-1/2})$. Substituting the unrestricted local estimate and the consistent information estimate gives $$ W_n=Z_n'A'(AA')^{-1}AZ_n+o_p(1)=Z_n'P_NZ_n+o_p(1). $$

For the score statistic, expand the score at the restricted estimate. Its standardized, information-whitened form is $Z_n-\tilde h_n+o_p(1)=P_NZ_n+o_p(1)$. Therefore its information-weighted squared length is $$ S_n=Z_n'P_NZ_n+o_p(1). $$ The matrix $P_N$ is a symmetric idempotent projection of rank $r$. If $Z\sim N_p(0,I_p)$, then $Z'P_NZ\sim\chi^2_r$: an orthogonal change of coordinates makes this sum the squares of $r$ independent standard normal variables. Applying convergence in distribution and the continuous mapping theorem proves the three limits. Since all three statistics equal the same random quadratic form up to terms converging in probability to zero, their pairwise differences also converge in probability to zero. $\square$

Worked Examples

Worked Example: Testing a Bernoulli Probability

Let $X_1,\ldots,X_{100}$ be Bernoulli observations, with 60 successes, and test $H_0:p=0.5$. The unrestricted estimate is $\hat p=0.6$. The information for one observation is $I(p)=1/\{p(1-p)\}$, and the null imposes one restriction, so the limiting reference distribution for each statistic is $\chi^2_1$.

For the Wald statistic, the estimated variance of $\hat p$ is $0.6(0.4)/100=0.0024$. Thus $$ W=\frac{(0.6-0.5)^2}{0.0024}=4.1667. $$ For the score statistic, use the null variance $0.5(0.5)/100=0.0025$: $$ S=\frac{(0.6-0.5)^2}{0.0025}=4. $$ For the likelihood ratio, the log-likelihood at the unrestricted estimate minus that at the null is $60\log(0.6/0.5)+40\log(0.4/0.5)$. Hence $$ R=2\{60\log(1.2)+40\log(0.8)\}=4.0271\ \text{(approximately)}. $$ All three are compared with $\chi^2_1$. Their different values at this sample size are compatible with first-order equivalence, which is an asymptotic statement, not an exact finite-sample identity.

Worked Example: A Linear Restriction on a Normal Mean Vector

Suppose $X_1,\ldots,X_{50}$ are independent $N_2(\theta,\Sigma)$, with known $$ \Sigma=\begin{pmatrix}1&0.3\\0.3&4\end{pmatrix}, $$ and the observed sample mean is $\bar X=(0.4,-0.1)'$. Test $H_0:\theta_1+\theta_2=0$. Let $c=(1,1)'$; the estimated contrast is $c'\bar X=0.3$, and $c'\Sigma c=1+4+2(0.3)=5.6$.

The Wald statistic is the squared estimated contrast divided by its variance: $$ W=\frac{(c'\bar X)^2}{c'\Sigma c/50} =\frac{0.09}{5.6/50} =\frac{4.5}{5.6} \approx0.8036. $$ Under the null, the restricted estimate is the likelihood-maximizing mean satisfying $c'\theta=0$. The score test for this one linear restriction uses the same contrast variance, so $S=0.8036$. The normal log-likelihood is quadratic with known covariance; maximizing it subject to the linear restriction reduces the log-likelihood by half this same squared standardized contrast. Therefore $R=0.8036$ as well. This exact agreement in this model is stronger than the general asymptotic equivalence.

Worked Example: Testing Equality of Two Poisson Rates

Consider independent Poisson counts from two groups, each observed for exposure 50. The counts are 60 and 40, so the unrestricted rate estimates are $\hat\lambda_1=60/50=1.2$ and $\hat\lambda_2=40/50=0.8$. Test the smooth one-dimensional restriction $H_0:\lambda_1-\lambda_2=0$. Under the null, the restricted estimate is the pooled rate $\tilde\lambda=(60+40)/(50+50)=1$.

The Wald variance of the rate difference is estimated as $1.2/50+0.8/50=0.04$, giving $$ W=\frac{(1.2-0.8)^2}{0.04}=4. $$ At the restricted fit, the score test uses the null variance $1/50+1/50=0.04$, and therefore $S=0.4^2/0.04=4$.

For the likelihood ratio, the exposure terms cancel between the unrestricted and restricted log-likelihoods. The remaining log-likelihood difference is $60\log(1.2)+40\log(0.8)$. Numerically, $60\log(1.2)\approx10.9392934$ and $40\log(0.8)\approx-8.9257421$, so $$ R=2\{60\log(1.2)+40\log(0.8)\}\approx2(2.0135514)=4.0271. $$ The statistics have one degree of freedom asymptotically. The arithmetic also illustrates why intermediate rounding should not determine the final reported value.

Interpretation, Scope, and a Common Pitfall

The degrees of freedom are the rank $r$ of the restriction Jacobian, not necessarily the number of parameter coordinates that appear in the written hypothesis. Redundant equations do not add independent restrictions. In regular exponential families, the smooth likelihood and information structure often make the local quadratic approximation especially natural; nevertheless, the theorem still depends on interiority, nonsingular information, and well-behaved estimators. If a maximum occurs at a boundary, the stated chi-square result need not apply.

Common mistake: concluding that Wald, score, and likelihood-ratio statistics must be numerically equal because they have the same chi-square limit. The proof shows only that each statistic differs from the same limiting quadratic form by a term that vanishes in probability. At a finite sample size the statistics use different evaluations of the likelihood and information, as the Bernoulli and Poisson examples show. A second related error is assigning degrees of freedom $p$, the full parameter dimension, rather than $r$, the number of locally independent restrictions.

The practical value of the result is that three different constructions can serve as asymptotically valid tests of the same smooth null, while their finite-sample behavior and computational convenience may differ. First-order equivalence is a local statement under the null; it does not promise exact equality or settle behavior under alternatives that move with sample size.

Check Your Understanding

Use the definitions and local projection argument to answer the following.

For a null hypothesis with a rank-three Jacobian, what is the asymptotic reference distribution of each statistic under the stated regularity conditions?
Why does the score statistic use the score at the restricted estimate rather than at the unrestricted estimate?
In the local quadratic derivation, what subspace does $P_N$ project onto, and why does its rank determine the chi-square degrees of freedom?
For the Bernoulli example, identify the variance used in the Wald statistic and the variance used in the score statistic.
Give one reason the three statistics can differ in a finite sample even though their pairwise differences converge in probability to zero.