A Worked Comparison of Wald, Score, and Likelihood-Ratio Tests

One Model, Three Tests

Wald, score, and likelihood-ratio tests can address the same hypothesis while using different information from the likelihood. A worked comparison makes those differences concrete: the Wald test uses an unrestricted estimate, the score test evaluates the slope of the log-likelihood at the null value, and the likelihood-ratio test compares the best fits under the null and alternative.

We use an exponential model because all three calculations can be written explicitly. The example also has an instructive feature: its Wald and score statistics are exactly equal, while its likelihood-ratio statistic is generally different. This equality is particular to the model and parameterization, not a general identity between the tests.

Model and Hypothesis

Model and test: Let \(X_1,\ldots,X_n\) be independent and identically distributed with density \[ f(x;\lambda)=\lambda e^{-\lambda x},\qquad x>0,\quad \lambda>0. \] Test \(H_0:\lambda=\lambda_0\) against \(H_1:\lambda\ne\lambda_0\), where \(\lambda_0\) is fixed and positive. Write \(S=\sum_{i=1}^n X_i\). The unrestricted maximum likelihood estimator is \(\hat{\lambda}=n/S\), provided \(S>0\).

The parameter space \((0,\infty)\) is open, so any fixed \(\lambda_0>0\) is an interior null value. The model is regular, and the null imposes one restriction. Under \(H_0\), the standard large-sample calibration for each of the three test statistics is a chi-square distribution with one degree of freedom. The resulting p-values are approximations, not finite-sample exact p-values.

The log-likelihood, score, and Fisher information are

\ell(\lambda)=n\log\lambda-\lambda S,\qquad U(\lambda)=\frac{n}{\lambda}-S,\qquad I_n(\lambda)=\frac{n}{\lambda^2}.

The second derivative is \(\ell''(\lambda)=-n/\lambda^2\), so the log-likelihood is strictly concave. Setting \(U(\lambda)=0\) gives the unrestricted maximum \(\hat{\lambda}=n/S\). Under the null, the restricted estimate is simply \(\lambda_0\).

The Three Statistics

For this scalar hypothesis, the Wald statistic uses the estimated variance of the unrestricted estimator, the score statistic uses Fisher information at the null, and the likelihood-ratio statistic compares the unrestricted and restricted log-likelihoods:

W=\frac{(\hat{\lambda}-\lambda_0)^2}{\hat{\lambda}^{\,2}/n},\qquad Q=\frac{U(\lambda_0)^2}{I_n(\lambda_0)},\qquad G^2=2\{\ell(\hat{\lambda})-\ell(\lambda_0)\}.

Here \(\hat{\lambda}^{\,2}/n\) estimates the variance of \(\hat{\lambda}\). For the score statistic, the numerator is evaluated at the restricted value, as required: \(U(\lambda_0)=n/\lambda_0-S\). The likelihood-ratio statistic compares the null fit with the best fit over the full parameter space.

Derived identity: In this exponential-rate model, \(W=Q\) for every sample with \(S>0\). In terms of \(t=\hat{\lambda}/\lambda_0\), the likelihood-ratio statistic is \(G^2=2n\{\log t-1+1/t\}\), whereas the common Wald and score statistic is \(n(1-1/t)^2\).

To verify the identity, substitute \(S=n/\hat{\lambda}\) into the Wald statistic:

W=\frac{n(\hat{\lambda}-\lambda_0)^2}{\hat{\lambda}^{\,2}} =n\left(1-\frac{\lambda_0}{\hat{\lambda}}\right)^2.

For the score statistic, substitute \(S=n/\hat{\lambda}\) and \(I_n(\lambda_0)=n/\lambda_0^2\):

Q=\frac{(n/\lambda_0-n/\hat{\lambda})^2}{n/\lambda_0^2} =n\left(1-\frac{\lambda_0}{\hat{\lambda}}\right)^2 =W.

For the likelihood ratio, use \(S=n/\hat{\lambda}\) in the log-likelihood difference:

\begin{aligned} G^2 &=2\left\{n\log\left(\frac{\hat{\lambda}}{\lambda_0}\right) -S(\hat{\lambda}-\lambda_0)\right\}\\ &=2n\left\{\log t-1+\frac{1}{t}\right\}. \end{aligned}

This establishes the formulas directly from the model likelihood. It also shows why the likelihood-ratio statistic need not equal the other two: it depends on the log-likelihood difference, not just on the squared standardized distance from the null.

Worked Comparisons

In each example, take \(n=100\), \(\lambda_0=1\), and use the large-sample chi-square calibration. For a statistic \(T\), the approximate p-value is \(P(\chi^2_1\ge T)\). Since the square root of a chi-square random variable with one degree of freedom has the absolute standard-normal distribution, this can also be calculated as \(2\{1-\Phi(\sqrt{T})\}\), where \(\Phi\) is the standard normal distribution function.

Worked Example: A Sample Sum of 80

Suppose the observed sample has \(S=80\). Then \(\hat{\lambda}=100/80=1.25\), and \(t=1.25\).

Wald statistic.

W=100\left(1-\frac{1}{1.25}\right)^2 =100(0.2)^2=4.

Score statistic. The null score is \(U(1)=100-80=20\), and the null information is \(I_{100}(1)=100\). Therefore

Q=\frac{20^2}{100}=4.

Likelihood-ratio statistic.

G^2=2(100)\left\{\log(1.25)-1+\frac{1}{1.25}\right\} =200\{\log(1.25)-0.2\} \approx 4.6287.

The chi-square p-values are approximately \(0.0455\) for both Wald and score, and \(0.0314\) for the likelihood ratio. At the five-percent level, all three reject \(H_0\), but the likelihood-ratio test gives the smaller p-value.

Worked Example: A Sample Sum of 60

Now let \(S=60\). The unrestricted estimate is \(\hat{\lambda}=100/60=5/3\), so \(t=5/3\).

Wald and score. The common formula gives

W=Q=100\left(1-\frac{1}{5/3}\right)^2 =100(0.4)^2=16.

Likelihood ratio.

\begin{aligned} G^2 &=200\left\{\log(5/3)-1+\frac{3}{5}\right\}\\ &=200\{\log(5/3)-0.4\}\\ &\approx 22.1651. \end{aligned}

The chi-square p-values are approximately \(0.0000633\) for Wald and score and \(0.0000026\) for the likelihood ratio. All three provide strong evidence against the null, but the numerical difference is now more pronounced than in the first example. The likelihood-ratio test assigns a smaller tail probability to this departure.

Worked Example: Different Decisions Near Five Percent

For a borderline case, let \(S=80.5\). Then \(\hat{\lambda}=100/80.5=200/161\), so \(t=200/161\approx1.24224\).

Wald and score. Since \(\lambda_0/\hat{\lambda}=0.805\),

$$ W=Q=100(1-0.805)^2 =100(0.195)^2=3.8025. $$

Likelihood ratio. Here \(1/t=0.805\), so

\begin{aligned} G^2 &=200\{\log(200/161)-1+0.805\}\\ &=200\{\log(200/161)-0.195\}\\ &\approx 4.3826. \end{aligned}

The approximate p-values are about \(0.0512\) for Wald and score and \(0.0363\) for the likelihood ratio. The five-percent critical value for \(\chi^2_1\) is approximately \(3.841\). Thus Wald and score do not reject, while the likelihood-ratio test does. Near a decision threshold, the differences between the statistics can affect the reported conclusion, not just the p-value’s final digits.

Why the Statistics Differ

The exact Wald–score equality here comes from a special relationship between the estimate and the information: both statistics reduce to \(n(1-\lambda_0/\hat{\lambda})^2\). In other models, the information used by Wald and score need not yield the same value. The equality should therefore not be carried over to a different likelihood without checking its formulas.

The likelihood-ratio statistic also treats departures on the two sides of the null differently in this model. To see the source of this difference, write \(t=1+\delta\), with \(\delta\) close to zero. For \(|\delta|<1\), the Taylor expansions of the logarithm and reciprocal give

\begin{aligned} W=Q &=n\left(\frac{\delta}{1+\delta}\right)^2 =n\delta^2-2n\delta^3+O(n\delta^4),\\ G^2 &=2n\left\{\log(1+\delta)-1+\frac{1}{1+\delta}\right\}\\ &=n\delta^2-\frac{4}{3}n\delta^3+O(n\delta^4). \end{aligned}

The leading quadratic terms agree, while the cubic terms differ. A positive \(\delta\) and a negative \(\delta\) therefore need not produce matching finite-sample statistics. This local calculation is consistent with the first-order equivalence of the tests, but does not imply that their finite-sample p-values are identical. Their chi-square calibrations are large-sample approximations, so especially extreme or borderline results should be interpreted with that limitation in mind.

Common Pitfalls and Practical Reading

Using the wrong evaluation point. The score test uses \(U(\lambda_0)\) and information at \(\lambda_0\), not the score or information evaluated at \(\hat{\lambda}\).
Reversing the likelihood-ratio subtraction. The statistic is twice the unrestricted log-likelihood minus the restricted log-likelihood, so it is nonnegative. Reversing the order can produce an invalid negative value.
Treating the p-value as exact. The calculations above use a chi-square approximation under the null. They do not show that the finite-sample null distribution is exactly \(\chi^2_1\).
Assuming equal statistics must give equal conclusions across every model. Wald and score coincide here because of the exponential likelihood’s algebra. That is a model-specific result.
Reporting only a reject-or-not decision. The third example shows that test choice can change a decision near the cutoff. Report which statistic and calibration were used, along with the p-value and the significance level.

In practice, the three tests are often close when the sample is informative and the data are not far from the null. Their agreement is useful, but it does not make them interchangeable in every finite sample. This exponential example gives a precise benchmark: Wald and score agree exactly, while the likelihood ratio reflects the shape of the log-likelihood between the null and unrestricted fits.

Check Your Understanding

Use the exponential-rate model and test setup from this tutorial.

For \(n=100\), \(\lambda_0=1\), and \(S=90\), calculate \(\hat{\lambda}\), the Wald statistic, and the score statistic.
Why are the Wald and score statistics exactly equal in this model? Which quantities are evaluated at different parameter values?
Write the likelihood-ratio statistic in terms of \(t=\hat{\lambda}/\lambda_0\), and identify the term that makes its finite-sample value differ from Wald and score.
What chi-square distribution is used for the large-sample calibration here, and why does it have one degree of freedom?
In the borderline example, why can the likelihood-ratio test reject at five percent while Wald and score do not?