Comparing Likelihood-Ratio, Score, and Wald Tests

Why Compare the Three Tests?

Likelihood-ratio, score, and Wald tests address the same broad question: does a parameter satisfy a specified restriction? Their constructions differ. The likelihood-ratio test compares the best fit under the restriction with the unrestricted best fit; the score test measures the slope away from the restricted fit; and the Wald test measures how far the unrestricted estimate lies from the restriction. Under regularity conditions, these approaches have the same first-order null calibration. At a fixed sample size, however, their statistics need not agree, and they can make different decisions.

The comparison below concerns a smooth restriction of fixed dimension in a regular parametric model. The Gaussian example makes the finite-sample distinction explicit without changing the underlying hypothesis.

Setup and Common Large-Sample Calibration

For the comparison theorem, assume the model is identifiable and differentiable in quadratic mean near \(\theta_0\), with finite, positive-definite Fisher information per observation. Assume the relevant maximum likelihood estimators are consistent and root-\(n\) regular, and that the likelihood admits the usual local quadratic expansion with a remainder negligible in probability for parameter changes of order \(n^{-1/2}\). The restriction must have the stated full rank, and the nuisance parameters must be identifiable and regular under the restriction. These conditions exclude, for example, boundary parameters and singular information.

A local coordinate system can represent the restriction as \(\psi=\psi_0\), with \(\psi\in\mathbb{R}^q\) and the remaining coordinates denoted by \(\lambda\). Profile the log-likelihood over \(\lambda\), and call the resulting function \(\ell_{p,n}(\psi)\). Let \(U_p\) be its score at \(\psi_0\), and let \(J\) be the efficient information per observation for \(\psi\), after accounting for \(\lambda\). In a partition of the information matrix, this is \(J=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\). It is positive definite under the assumptions above.

The local quadratic expansion of the profile log-likelihood is

for \(h=O_p(n^{-1/2})\). Maximizing the quadratic part gives \(\hat{\psi}-\psi_0=(nJ)^{-1}U_p+o_p(n^{-1/2})\). Substitution into the same expansion gives the likelihood-ratio statistic. The score statistic is the quadratic form in \(U_p\) scaled by the efficient information. The Wald statistic uses the squared estimated displacement scaled by its covariance. Consequently, all three have the same leading expression:

Under \(H_0\), the efficient score satisfies \(U_p/\sqrt{n}\xrightarrow{d}N(0,J)\). Therefore the leading quadratic form converges to \(\chi^2_q\), proving the theorem. The argument explains both the agreement and its limit: it uses a local approximation, so terms negligible as \(n\) grows can still affect a finite-sample statistic. \(\square\)

Worked Gaussian Comparison

Consider independent observations \(X_i\sim N(\mu,\sigma^2)\), with both \(\mu\) and \(\sigma^2\) unknown, and test \(H_0:\mu=0\). Define the maximum-likelihood variance estimates

Here \(S^2\) is the unrestricted estimate of \(\sigma^2\), while \(S_0\) is its restricted estimate when \(\mu=0\). For \(S^2>0\), the three statistics are

To obtain the score expression, the score for \(\mu\) at the restricted fit is \(n\bar X/S_0\). Its information is \(n/S_0\). The mean and variance scores are orthogonal at \(\mu=0\), so adjusting for the nuisance variance leaves the score statistic \(Q=(n\bar X/S_0)^2/(n/S_0)\). The displayed relations show exactly how the three tests differ in this model.

Decision Thresholds and Exact Null Behavior

In this Gaussian model, each statistic is an increasing function of \(W\). This makes their asymptotic-calibration decisions particularly easy to compare. With \(c=3.841\), the Wald test rejects when \(W>c\). Since \(LR=n\log(1+W/n)\), the likelihood-ratio test rejects when \(W>n\{\exp(c/n)-1\}\). Since \(Q=W/(1+W/n)\), the score test rejects, for \(n>c\), when \(W>nc/(n-c)\).

The differences can also be seen from the exact null distribution. Under \(H_0\), the usual one-sample statistic \(T=\bar X/(s/\sqrt{n})\), where \(s^2=(n-1)^{-1}\sum_i(X_i-\bar X)^2\), has a \(t_{n-1}\) distribution. Since \(S^2=(n-1)s^2/n\), direct substitution gives

Consequently, at a fixed \(n\), the Wald statistic does not have an exact \(\chi^2_1\) null distribution. The likelihood-ratio and score statistics are nonlinear transformations of this same \(W\), so their exact null distributions and their rejection probabilities at the common asymptotic cutoff differ as well. For \(n=10\), the three \(W\)-thresholds just derived correspond to different thresholds for \(T^2\), obtained by multiplying each by \(9/10\). Exact calibration, if required, must use the finite-sample distribution rather than assume the asymptotic cutoff is exact.

What the Comparison Does—and Does Not—Say

The common chi-square limit is useful because it gives a shared large-sample reference for tests of the same regular restriction. It does not establish that the tests are interchangeable in every finite sample, nor that the same rejection decision must follow from each one. The Gaussian example shows that even when all three statistics are simple functions of one quantity, their finite-sample thresholds under a common asymptotic calibration can differ.

A common mistake is to reason that equal asymptotic null distributions imply equal test statistics or equal finite-sample Type I error rates. The correction is that convergence in distribution controls behavior in the limit, not exact equality at a particular \(n\). In applications, also check that the regularity assumptions are plausible: boundary restrictions, weak identification, or singular information can invalidate the standard comparison. If a finite-sample calibration is available for the model, it may be preferable to applying the same asymptotic cutoff to all three statistics.