Why Wald, Likelihood-Ratio, and Score Tests Differ

Three Tests, Three Views of the Likelihood

Wald, likelihood-ratio, and score tests address the same null hypothesis, but they inspect the likelihood at different points. The Wald test measures how far the unrestricted estimate lies from the null. The likelihood-ratio test compares the best fit under the null with the unrestricted best fit. The score test asks whether the likelihood is still trying to move away from the null when evaluated at the restricted estimate.

Under regularity conditions, these tests have the same first-order large-sample null distribution, and they have the same first-order behavior under local alternatives. That does not make their finite-sample statistics interchangeable: each uses a different approximation to the likelihood’s shape, and those approximations can differ appreciably in a small sample or for a curved likelihood.

Write \(\hat{\theta}\) for the unrestricted maximum likelihood estimate and \(\tilde{\theta}\) for the estimate that maximizes the likelihood subject to \(H_0\). Let \(\widehat{\operatorname{Var}}(\hat{\theta})\) estimate the covariance matrix of \(\hat{\theta}\), and let \(R(\theta)\) be the \(q\)-by-\(p\) derivative matrix of \(r(\theta)\), where \(p\) is the dimension of \(\theta\). With consistent information and covariance estimates, the three statistics are:

Here \(U_{\mathrm{eff}}\) and \(I_{\mathrm{eff}}\) are the score and information for the tested parameter after adjustment for nuisance parameters. If the parameter is divided into a tested component \(\psi\) and a nuisance component \(\lambda\), the adjustment removes the part of the tested score predictable from the nuisance score. In block notation, the efficient score and information are \(U_\psi-I_{\psi\lambda}I_{\lambda\lambda}^{-1}U_\lambda\) and \(I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\), respectively. The score test evaluates these quantities at the restricted estimate; it does not require fitting the unrestricted model.

Why the Statistics Agree Locally

The shared limit comes from approximating the log likelihood near the null by a quadratic function. To see the mechanism, first consider the tested parameter \(\psi\) with nuisance parameter \(\lambda\). After locally accounting for nuisance estimation, write \(U_{\mathrm{eff}}\) for the efficient score at the null and \(J\) for the efficient information. The quadratic approximation to the profile log likelihood, in a displacement \(d=\psi-\psi_0\), is

The maximum of this quadratic occurs at \(d=J^{-1}U_{\mathrm{eff}}\). Substitution gives the approximate likelihood gain; the same displacement gives the approximate Wald statistic, while the score statistic uses the slope and curvature right at \(d=0\).

Thus the three statistics coincide for the same local quadratic approximation. In regular large samples, the relevant estimates are close to the null under \(H_0\), so higher-order departures from that approximation vanish to first order. Each statistic therefore converges in distribution to \(\chi^2_q\). For a local alternative \(\psi=\psi_0+h/\sqrt{n}\), each also has the same first-order noncentral chi-square limit, with noncentrality determined by the efficient information and \(h\). This is a statement about asymptotic approximation, not a guarantee of identical finite-sample values or power.

Finite-Sample Differences in a Bernoulli Model

For a Bernoulli model, let \(X_1,\ldots,X_n\) be independent with success probability \(p\), and test \(H_0:p=p_0\). The log likelihood, omitting terms not depending on \(p\), is \(x\log p+(n-x)\log(1-p)\), where \(x\) is the observed number of successes. The unrestricted estimate is \(\hat p=x/n\). The score statistic uses the slope and information at \(p_0\); the Wald statistic uses the estimated variance at \(\hat p\); and the likelihood-ratio statistic uses the log-likelihood difference. These formulas expose their different evaluation points.

Parameterization and the Wald Test

The likelihood-ratio statistic is unchanged by a one-to-one re-expression of the parameter: the maximized likelihood values do not change. The score statistic is also invariant under a smooth, nonsingular reparameterization when the score and information are transformed consistently. A Wald statistic, in contrast, measures distance using a local covariance approximation at the unrestricted estimate. A nonlinear change of coordinates can change that distance, even when it leaves the hypothesis and model unchanged. The difference disappears to first order in regular large samples, but can matter in finite samples.

Choosing and Interpreting the Tests

All three tests are asymptotically valid under the stated regularity conditions, but their practical features differ. The score test can be attractive when fitting the unrestricted model is difficult, because it needs only the restricted estimate and the score and information there. The likelihood-ratio test compares fitted likelihoods directly and is invariant to smooth reparameterization. The Wald test is convenient when the unrestricted estimate and its covariance matrix are already available, but it can be sensitive to parameterization and to an estimate lying far from the null relative to the local curvature.

None of these asymptotic comparisons guarantees accurate chi-square calibration in a particular small sample. The Bernoulli example shows that using the same reference distribution does not force the same p-value or decision. When results differ, report which statistic was used, its reference distribution and p-value, and the model and null being tested. Do not describe one test as universally correct solely because its statistic is larger or smaller; the adequacy of the approximation depends on the data and model.