Choosing Among Wald, Score, and Likelihood-Ratio Inference

Why There Is More Than One Likelihood Procedure

A generalized linear model can offer several routes to inference about the same parameter. A Wald procedure measures how far an estimate lies from the null value in estimated standard-error units. A score procedure measures the likelihood’s slope at the null fit. A likelihood-ratio procedure compares the optimized likelihoods under the unrestricted and restricted models. Under regular large-sample conditions, these approaches agree to first order, but their calculations and finite-sample behavior can differ.

The distinction matters in practice. Wald inference is often convenient when an unrestricted estimate and its covariance matrix are already available. A score test requires fitting only the null model. A likelihood-ratio test requires both restricted and unrestricted fits, but compares their likelihoods directly. Profile likelihood extends the likelihood-ratio idea to a parameter of interest while optimizing over nuisance parameters.

Definitions for a Generalized Linear Model

Let $\theta=(\psi,\lambda)$ denote a finite-dimensional parameter, where $\psi$ is the parameter or vector of parameters being tested and $\lambda$ contains nuisance parameters. Consider the null hypothesis $H_0:\psi=\psi_0$. Write $\hat{\theta}$ for the unrestricted maximum-likelihood estimate and $\tilde{\theta}=(\psi_0,\tilde{\lambda})$ for the maximum-likelihood estimate subject to the null. Let $I(\theta)$ denote the Fisher information matrix. The definitions below assume that the relevant estimates exist.

Definition: For a scalar parameter $\psi$, the Wald statistic is the squared distance between $\hat{\psi}$ and $\psi_0$, divided by the estimated variance of $\hat{\psi}$. For a $q$-dimensional parameter, it is the corresponding quadratic form using the estimated covariance matrix. The score statistic uses the score at the restricted estimate, scaled by its efficient information after accounting for nuisance parameters. The likelihood-ratio statistic is twice the unrestricted minus restricted maximized log-likelihood: $$ W=(\hat{\psi}-\psi_0)^\mathsf{T}\widehat{\operatorname{Var}}(\hat{\psi})^{-1}(\hat{\psi}-\psi_0),\qquad S=U_{\psi\cdot\lambda}^{\mathsf T}I_{\psi\cdot\lambda}^{-1}U_{\psi\cdot\lambda},\qquad G^2=2\{\ell(\hat{\theta})-\ell(\tilde{\theta})\}. $$ Here $U_{\psi\cdot\lambda}$ is the efficient score and $I_{\psi\cdot\lambda}$ the efficient information, evaluated at the restricted estimate. Under the null, $U_\lambda(\tilde{\theta})=0$, so the efficient score is $U_\psi(\tilde{\theta})$; the efficient information is $I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}$, with information blocks evaluated at that estimate.

For a smooth scalar restriction $h(\theta)=0$, the Wald statistic instead uses $h(\hat{\theta})$ and its delta-method variance, $\nabla h(\hat{\theta})^\mathsf{T}\widehat{\operatorname{Var}}(\hat{\theta})\nabla h(\hat{\theta})$. This is useful for contrasts and other nonlinear hypotheses; the degrees of freedom are the number of independent restrictions. The displayed block formula assumes the nuisance-information block is invertible.

The usual large-sample reference distribution for each statistic is chi-square with $q$ degrees of freedom under the null. This calibration requires regularity: the true parameter is an identifiable interior point, the model is correctly specified, the log-likelihood is sufficiently smooth, information is nonsingular, and the estimates behave consistently and asymptotically normally as information grows. The restrictions must be smooth and have full rank. These conditions exclude, for example, boundary cases and models in which a finite maximum-likelihood estimate does not exist.

Three Tests in One Binomial Model

Worked Example: Wald, Score, and Likelihood Ratio for a Proportion

Suppose $X\sim\operatorname{Binomial}(10,p)$, and the observed count is $X=8$. Test $H_0:p=0.5$ against a two-sided alternative. The unrestricted estimate is $\hat p=8/10=0.8$. The null estimate is $p_0=0.5$. The following calculations use the usual large-sample chi-square reference distributions; with this small sample, they are approximations.

Wald. The estimated variance of $\hat p$ is $\hat p(1-\hat p)/10=0.8(0.2)/10=0.016$. Thus $$ W=\frac{(0.8-0.5)^2}{0.016}=5.625. $$ Against a $\chi^2_1$ reference distribution, the approximate two-sided p-value is about $0.018$.

Score. Under the null, the variance of the count is $10(0.5)(0.5)=2.5$, and its expected value is $5$. Therefore $$ S=\frac{(8-5)^2}{2.5}=3.6. $$ Its $\chi^2_1$ approximate p-value is about $0.058$.

Likelihood ratio. The binomial coefficient cancels in the log-likelihood difference, giving $$ G^2=2\left\{8\log\left(\frac{0.8}{0.5}\right)+2\log\left(\frac{0.2}{0.5}\right)\right\} =2\{8\log(1.6)+2\log(0.4)\}\approx 3.855. $$ The $\chi^2_1$ approximate p-value is about $0.050$. The three statistics differ here, and their approximate p-values fall on different sides of $0.05$. This is not a contradiction: the large-sample equivalence is not an assertion that the statistics are identical in a sample of size 10.

Why the Three Statistics Agree Asymptotically

The local agreement can be derived by expanding the log-likelihood near the restricted estimate. The derivation also shows where the efficient information enters. For clarity, take $\psi$ to be $q$-dimensional and $\lambda$ to be nuisance parameters. Assume the regularity conditions above, and that the log-likelihood’s third-order remainder is negligible on the local estimation scale.

Theorem: Under $H_0:\psi=\psi_0$ and the stated regularity conditions, the Wald, score, and likelihood-ratio statistics for this restriction differ by $o_p(1)$. Each converges in distribution to $\chi^2_q$.

Remove the nuisance displacement.
Write $\delta=\psi-\psi_0$ and $a=\lambda-\tilde{\lambda}$. A quadratic expansion around $\tilde{\theta}$ has linear term $U_\psi^\mathsf{T}\delta$, since the nuisance score is zero, and quadratic term formed from the information blocks. For fixed $\delta$, maximizing over $a$ gives $a= -I_{\lambda\lambda}^{-1}I_{\lambda\psi}\delta$. Substituting this value leaves the efficient information $I_e=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}$.

Maximize the resulting local likelihood.
The profiled quadratic increment is $U_\psi^\mathsf{T}\delta-\tfrac12\delta^\mathsf{T}I_e\delta$. Its maximizer is $\hat{\delta}=I_e^{-1}U_\psi$. Twice its maximized value is $U_\psi^\mathsf{T}I_e^{-1}U_\psi$, which is the score quadratic form. The neglected likelihood terms are $o_p(1)$, so the likelihood-ratio statistic equals this form plus $o_p(1)$.

Relate the Wald statistic to the same form.
The local unrestricted estimate satisfies $\hat{\delta}=I_e^{-1}U_\psi+o_p(n^{-1/2})$, and its covariance is $I_e^{-1}+o_p(n^{-1})$. Substitution into the Wald quadratic form gives $U_\psi^\mathsf{T}I_e^{-1}U_\psi+o_p(1)$. Thus all three statistics agree to $o_p(1)$.

Obtain the reference distribution.
Under the null, the efficient score divided by the square root of information is asymptotically $N_q(0,I_q)$. Its squared Euclidean length therefore converges to $\chi^2_q$. Since each test statistic differs from that squared length by $o_p(1)$, each has the same limiting distribution. $\square$

This is a local argument: it compares likelihood behavior in a shrinking neighborhood of the null. It explains asymptotic equivalence, not equal finite-sample results, and it does not justify chi-square calibration when the regularity conditions fail.

Profile Likelihood for a Parameter of Interest

When the target is a parameter $\psi$ and $\lambda$ is nuisance, define the profile log-likelihood by maximizing over the nuisance parameter separately for each fixed value of $\psi$:

\ell_p(\psi)=\sup_{\lambda}\ell(\psi,\lambda).

A profile-likelihood confidence set contains the values $\psi$ for which $2\{\ell_p(\hat{\psi})-\ell_p(\psi)\}$ does not exceed the chosen $\chi^2_1$ cutoff, under regular large-sample calibration. The nuisance estimate is allowed to change as $\psi$ changes. This often gives a more faithful representation of likelihood curvature than a symmetric interval formed from a single standard error, especially when the likelihood is asymmetric. Profile likelihood is a way to construct likelihood-ratio inference for a parameter of interest; it is not a fourth, unrelated test statistic.

Worked Example: Profiling a Poisson Rate Ratio

Consider independent Poisson counts $Y_1=2$ and $Y_2=8$, observed over equal exposures of 10 units. Let the rates be $\exp(\alpha)$ and $\exp(\alpha+\beta)$, so $\beta$ is the log rate ratio. Ignoring terms constant in the parameters, the log-likelihood is $$ \ell(\alpha,\beta)=2\alpha+8(\alpha+\beta)-10e^\alpha-10e^{\alpha+\beta}. $$

For fixed $\beta$, differentiating with respect to $\alpha$ gives $10-10e^\alpha-10e^{\alpha+\beta}=0$. Hence $$ \hat{\alpha}(\beta)=\log\left(\frac{10}{10+10e^\beta}\right). $$ At the unrestricted maximum, the fitted rates are $2/10=0.2$ and $8/10=0.8$, so $\hat{\beta}=\log(0.8/0.2)=\log 4$. At the null $\beta=0$, the common fitted rate is $(2+8)/(10+10)=0.5$. The likelihood-ratio statistic for this null is $$ G^2=2\left\{2\log(0.2/0.5)+8\log(0.8/0.5)\right\} =2\{2\log(0.4)+8\log(1.6)\}\approx 3.855. $$ The profile-likelihood curve is obtained by substituting $\hat{\alpha}(\beta)$ into $\ell(\alpha,\beta)$. Its confidence-set endpoints are found by solving $2\{\ell_p(\log 4)-\ell_p(\beta)\}=\chi^2_{1,0.95}$ for $\beta$; the nuisance intercept is re-optimized at every candidate endpoint. This is the essential computational distinction from a Wald interval, which uses only the estimate and its local estimated variance.

Choosing a Procedure in Practice

Procedure	What it needs	Practical strength	Main caution
Wald	Unrestricted estimate and estimated covariance	Convenient for many contrasts and nonlinear functions	Can behave poorly with a weak, boundary-near, or highly asymmetric estimate
Score	Restricted fit, score, and information	Tests the null without fitting the unrestricted model	Its result can differ from the other tests in finite samples
Likelihood ratio	Restricted and unrestricted fits	Directly compares maximized likelihoods of nested models	Requires both fits and regular chi-square calibration
Profile likelihood	Repeated nuisance-parameter optimization over target values	Represents likelihood shape for a parameter of interest	Can require more computation and remains subject to regularity limits

Worked Example: A Wald Test for a Linear Contrast

Suppose a fitted generalized linear model has coefficient estimate $\hat{\beta}=(0.4,0.7,-0.2)^\mathsf{T}$ and estimated covariance matrix $$ \widehat{\operatorname{Var}}(\hat{\beta})= \begin{pmatrix} 0.09&0.01&0\\ 0.01&0.0625&0.0125\\ 0&0.0125&0.04 \end{pmatrix}. $$ Test the contrast $H_0:\beta_2+\beta_3=0$. Set $c=(0,1,1)^\mathsf{T}$. The estimated contrast is $c^\mathsf{T}\hat{\beta}=0.7-0.2=0.5$, and its estimated variance is $$ c^\mathsf{T}\widehat{\operatorname{Var}}(\hat{\beta})c =0.0625+0.04+2(0.0125)=0.1275. $$ Therefore $W=0.5^2/0.1275\approx1.961$, or equivalently the signed Wald statistic is $0.5/\sqrt{0.1275}\approx1.400$. Using the $\chi^2_1$ approximation, the p-value is about $0.16$. The covariance term matters: omitting it would use the wrong variance for the sum of the two estimates.

Common Mistake

A common mistake is to conclude that Wald, score, and likelihood-ratio tests must give the same answer because all have an asymptotic chi-square distribution. The conclusion does not follow: a shared limiting distribution does not make their observed statistics equal at finite sample sizes. The binomial calculation above shows that their approximate p-values can even differ around a conventional decision threshold. Another mistake is to treat a Wald test as invariant to re-expression of the parameter. Wald inference depends on the estimate’s local standard error and can change under nonlinear transformations in finite samples. When the likelihood is notably asymmetric, a profile-likelihood analysis can reveal that shape rather than imposing a symmetric standard-error summary.

A useful practical rule is to match computation to the question and available fits: use a score test when fitting only the null is valuable; use a likelihood-ratio comparison when both nested fits are available and the likelihood comparison is appropriate; use Wald inference for convenient contrasts when estimates are well behaved; and consider profile likelihood when the parameter-specific likelihood shape is important. None of these choices repairs model misspecification or boundary irregularity.

Check Your Understanding

Use the definitions and conditions in this tutorial to answer the following.

Which two fits are required to calculate a likelihood-ratio test for a nested-model restriction?
Why does the efficient information, rather than just $I_{\psi\psi}$, appear when nuisance parameters are present?
In the binomial example, which statistic uses the variance evaluated under the null, and what was that variance for the count?
What changes in a profile-likelihood calculation as the candidate value of the parameter of interest changes?
Name two regularity conditions that can fail at a boundary and explain why standard chi-square calibration then requires caution.