First-Order Equivalence of Wald, Score, and Likelihood-Ratio Tests

Why Compare the Three Likelihood Tests?

Wald, score, and likelihood-ratio tests answer the same basic question in different ways. The Wald test measures how far the unrestricted estimate lies from the null value. The score test measures the likelihood’s slope at the null. The likelihood-ratio test compares the likelihood at the unrestricted estimate with its value under the null. Under regularity conditions, when the null is true, all three statistics have the same leading quadratic form as the sample size grows.

This is a statement about first-order asymptotic equivalence, not exact equality. It does not imply that the three statistics agree in a finite sample, or that their finite-sample p-values must be identical. We derive the equivalence for testing a fixed parameter value in a regular, finite-dimensional model. Write the parameter as $\theta\in\mathbb{R}^p$, and suppose the true value under the null is the interior point $\theta_0$.

Definition: Let $\ell_n(\theta)$ be the log-likelihood, $U_n(\theta)=\nabla\ell_n(\theta)$ its score vector, and $J_n(\theta)=-\nabla^2\ell_n(\theta)$ its observed information matrix. Let $I(\theta)$ be the Fisher information per observation, assumed positive definite at $\theta_0$. The expected information in a sample of size $n$ is $nI(\theta)$. The statistics considered here are $$ W_n=(\hat{\theta}-\theta_0)^\mathsf{T} nI(\hat{\theta})(\hat{\theta}-\theta_0),\qquad S_n=U_n(\theta_0)^\mathsf{T}\{nI(\theta_0)\}^{-1}U_n(\theta_0), $$ and $$ R_n=2\{\ell_n(\hat{\theta})-\ell_n(\theta_0)\}, $$ where $\hat{\theta}$ is the unrestricted maximum-likelihood estimator. First-order equivalence means that each pairwise difference converges to zero in probability under the null.

We use expected information in the displayed Wald and score statistics. The same first-order result holds if expected information is replaced by an observed-information estimate that is consistent, after division by $n$, for $I(\theta_0)$. Assume independent, identically distributed observations from a correctly specified model; an interior true parameter; the differentiability needed for the score and Hessian; a consistent, root-$n$ consistent interior MLE; and nonsingular Fisher information. Also assume the normalized observed information converges locally uniformly in probability: for each fixed $C$, $$ \sup_{\|\theta-\theta_0\|\leq C/\sqrt{n}} \left\|n^{-1}J_n(\theta)-I(\theta_0)\right\|\xrightarrow{p}0. $$ These conditions ensure that the information along the short segment from $\theta_0$ to $\hat{\theta}$ has the same leading limit.

The Score-Based Expansion of the MLE

The key step is to relate the MLE’s displacement from the null to the score at the null. For a vector parameter, one must not generally apply a scalar mean-value argument and claim there is a single intermediate point whose information matrix maps the displacement to the score difference. The valid argument uses the integral Taylor formula along the entire line segment.

Theorem: Under the regularity conditions above and under $H_0:\theta=\theta_0$, $$ \hat{\theta}-\theta_0=\{nI(\theta_0)\}^{-1}U_n(\theta_0)+o_p(n^{-1/2}). $$

Proof. Let $\Delta_n=\hat{\theta}-\theta_0$. Since the interior MLE satisfies $U_n(\hat{\theta})=0$, applying the integral Taylor formula to the score gives $$ 0=U_n(\theta_0)-\left\{\int_0^1 J_n(\theta_0+t\Delta_n)\,dt\right\}\Delta_n. $$ Set $\bar{J}_n=\int_0^1 J_n(\theta_0+t\Delta_n)\,dt$. Root-$n$ consistency gives $\Delta_n=O_p(n^{-1/2})$. The local uniform convergence condition therefore implies $\bar{J}_n/n\xrightarrow{p}I(\theta_0)$: for any chosen high-probability bound $\|\Delta_n\|\leq C/\sqrt n$, every point on the segment satisfies the stated uniform bound, and the probability of exceeding such a bound can be made arbitrarily small by increasing $C$. Since $I(\theta_0)$ is positive definite, $(\bar{J}_n/n)^{-1}\xrightarrow{p}I(\theta_0)^{-1}$. Thus $$ \Delta_n=\bar{J}_n^{-1}U_n(\theta_0) =\{nI(\theta_0)\}^{-1}U_n(\theta_0)+o_p(n^{-1/2}), $$ where $U_n(\theta_0)=O_p(\sqrt n)$ under the usual score central limit conditions. $\square$

Deriving the Common Quadratic Form

The MLE expansion shows that the Wald statistic uses, to first order, the score at the null. The score statistic already has that form. To connect the likelihood-ratio statistic to them, expand the log-likelihood from $\theta_0$ to $\hat{\theta}$, again using an integral remainder.

Theorem: Under the stated conditions and under the null, $$ W_n=S_n+o_p(1),\qquad R_n=S_n+o_p(1). $$ Consequently $W_n-R_n=o_p(1)$ as well.

Proof. The estimator expansion, together with $nI(\hat{\theta})/n\xrightarrow{p}I(\theta_0)$, gives $$ W_n=\Delta_n^\mathsf{T}nI(\theta_0)\Delta_n+o_p(1) =U_n(\theta_0)^\mathsf{T}\{nI(\theta_0)\}^{-1}U_n(\theta_0)+o_p(1) =S_n+o_p(1). $$ For the likelihood ratio, the integral Taylor formula gives $$ \ell_n(\hat{\theta})-\ell_n(\theta_0) =U_n(\theta_0)^\mathsf{T}\Delta_n -\int_0^1(1-t)\Delta_n^\mathsf{T}J_n(\theta_0+t\Delta_n)\Delta_n\,dt. $$ Local information convergence and $\Delta_n=O_p(n^{-1/2})$ make the integral equal to $\tfrac12\Delta_n^\mathsf{T}nI(\theta_0)\Delta_n+o_p(1)$. Define $A_n=\{nI(\theta_0)\}^{-1}U_n(\theta_0)$ and write $\Delta_n=A_n+r_n$, where $r_n=o_p(n^{-1/2})$. Since $U_n(\theta_0)=nI(\theta_0)A_n$, direct expansion yields $$ U_n(\theta_0)^\mathsf{T}\Delta_n-\tfrac12\Delta_n^\mathsf{T}nI(\theta_0)\Delta_n =\tfrac12 U_n(\theta_0)^\mathsf{T}\{nI(\theta_0)\}^{-1}U_n(\theta_0) -\tfrac12 r_n^\mathsf{T}nI(\theta_0)r_n. $$ The last term is $o_p(1)$, because $r_n=o_p(n^{-1/2})$. Multiplying the likelihood difference by two proves $R_n=S_n+o_p(1)$. $\square$

The common quadratic form also explains the familiar null calibration. Under the usual central limit theorem for the score, $U_n(\theta_0)/\sqrt n\xrightarrow{d}N(0,I(\theta_0))$. Its standardized squared length therefore converges in distribution to a chi-square random variable with $p$ degrees of freedom. The equivalence result is stronger than merely saying the three statistics have this same limiting distribution: their differences themselves vanish in probability.

Worked Examples

Worked Example: Exact Agreement for a Normal Mean

Suppose $X_1,\ldots,X_{25}$ are independent $N(\mu,4)$ observations, where the variance is known. Test $H_0:\mu=0$ when $\bar X=1.2$.

Ignoring terms not involving $\mu$, the log-likelihood is $\ell(\mu)=-\frac{1}{8}\sum_{i=1}^{25}(X_i-\mu)^2+\text{constant}$. Its MLE is $\hat{\mu}=\bar X=1.2$, its score at the null is $U_n(0)=\frac{25}{4}\bar X=7.5$, and its expected information is $25/4$. Therefore $$ S_n=\frac{7.5^2}{25/4}=9. $$ The Wald statistic is $$ W_n=\frac{25}{4}(1.2-0)^2=9. $$ Finally, completing the square gives $\ell(\hat{\mu})-\ell(0)=\frac{25}{8}\bar X^2$, so $$ R_n=2\{\ell(\hat{\mu})-\ell(0)\}=\frac{25}{4}(1.2)^2=9. $$ Here the log-likelihood is exactly quadratic in the parameter, and all three statistics agree exactly for every sample, not merely to first order.

Worked Example: Exponential Rate—Two Statistics Agree, One Does Not

Let $X_1,\ldots,X_4$ be independent exponential observations with rate $\lambda$, so each density is $f(x;\lambda)=\lambda e^{-\lambda x}$ for $x>0$. Test $H_0:\lambda=1$, given $\sum_i X_i=8/3$.

The log-likelihood is $\ell(\lambda)=4\log\lambda-\lambda(8/3)$, giving $\hat{\lambda}=4/(8/3)=3/2$. The score at the null is $U_n(1)=4-8/3=4/3$, and the expected information is $4/\lambda^2$. Thus $$ S_n=\frac{(4/3)^2}{4}= \frac49. $$ The Wald statistic, using expected information at $\hat{\lambda}$, is $$ W_n= \frac{4}{(3/2)^2}(3/2-1)^2 =\frac{16}{9}\cdot\frac14=\frac49. $$ The likelihood-ratio statistic is instead $$ R_n=2\left\{4\log(3/2)-\frac32\cdot\frac83+\frac83\right\} =2\left\{4\log(3/2)-\frac43\right\}\approx0.5771. $$ Thus the Wald and score values happen to agree in this example, but the likelihood-ratio value differs. The difference is compatible with first-order equivalence: the theorem does not assert equality at this sample size.

Worked Example: Finite-Sample Differences for a Poisson Rate

Suppose four independent observations have a Poisson distribution with mean $\lambda$, and their observed total is 6. Test $H_0:\lambda=1$. The log-likelihood, up to a constant, is $\ell(\lambda)=6\log\lambda-4\lambda$, so $\hat{\lambda}=6/4=3/2$.

The score at the null is $U_n(1)=6-4=2$, and the expected information is $4/\lambda$. Hence $$ S_n=\frac{2^2}{4}=1. $$ The Wald statistic is $$ W_n=\frac{4}{3/2}(3/2-1)^2 =\frac83\cdot\frac14=\frac23. $$ The likelihood-ratio statistic is $$ R_n=2\left\{6\log(3/2)-4(3/2-1)\right\} =12\log(3/2)-4\approx0.8656. $$ All three numbers are different: $W_n\approx0.6667$, $R_n\approx0.8656$, and $S_n=1$. This calculation illustrates why a finite-sample analysis should not substitute one statistic for another on the grounds that they are asymptotically equivalent.

What Equivalence Does—and Does Not—Tell Us

Under the regular true null, the three statistics share the same leading quadratic approximation. This is useful because it explains why distinct constructions can have the same large-sample null calibration. It also gives a practical diagnostic: in a well-behaved model with substantial information, the statistics may be close, while a large discrepancy can signal that the sample is not yet in a regime where the local quadratic approximation is accurate.

The result does not guarantee that the tests have identical finite-sample size, power, or p-values. Nor does it say that their values are close for every observed sample. Wald statistics use the estimate and its estimated information; score statistics use the null value; likelihood-ratio statistics compare maximized likelihoods. Those ingredients can behave differently when the likelihood is notably nonquadratic, the sample is small, or the estimate is far from the null. The theorem concerns their behavior in probability under a true, interior, regular null as $n$ increases.

Common Pitfall

Wrong reasoning: “Each statistic converges to the same chi-square distribution, so the three statistics must be equal, or nearly equal, in my sample.” A common limiting distribution does not imply equality of finite-sample values.
Correction: First-order equivalence means pairwise differences converge to zero in probability under the stated null conditions. It is an asymptotic statement; the exponential and Poisson calculations show that meaningful finite-sample differences can remain.
Wrong reasoning: For a vector score, use a scalar mean-value theorem to assert one intermediate information matrix exactly maps the score difference to the parameter displacement.
Correction: Use the integral Taylor formula along the segment between the null and the estimate. Under local information convergence, the averaged information has the required limit without assuming a generally invalid single-point vector identity.

Key takeaway: Under a regular true null, the MLE displacement is the inverse-information-weighted score to first order. Substituting that expansion into the Wald and likelihood-ratio statistics shows that all three reduce to the same score quadratic form, up to $o_p(1)$.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

What does first-order equivalence assert about $W_n-S_n$ and $R_n-S_n$ under the null?
Why is the integral Taylor formula needed when expanding a vector-valued score?
In the normal-mean example, which feature of the log-likelihood makes the three statistics agree exactly?
For the Poisson example, calculate why the score statistic is 1 while the Wald statistic is $2/3$.
Does convergence of all three statistics to the same chi-square distribution imply that their finite-sample p-values are identical? Explain.