Tutorials › Statistical Theory II - Mathematical Statistics and Statistical Inference › Computing the Three Tests from Restricted and Unrestricted Fits

Home  /  Tutorials  /  Wald, Score, and Likelihood-Ratio Tests

Wald, Score, and Likelihood-Ratio Tests · Tutorial 888 of 970

Computing the Three Tests from Restricted and Unrestricted Fits

Track exactly which fit and information estimate each test needs, then calculate and compare all three in constrained Gaussian models.

Advanced 12 min read

What You'll Learn

  • Identify the unrestricted estimate and covariance required for a Wald statistic.
  • Compute a score statistic from the restricted fit and its efficient information.
  • Compute a likelihood-ratio statistic by comparing maximized likelihoods under both hypotheses.
  • Work through the three tests in Gaussian regression with unknown variance.
  • Recognize when different computational routes give equal or unequal statistics.
  • Check residual sums of squares and variance estimates before comparing test results.

Three Tests, Three Computational Routes

For a regular nested-model test, Wald, score, and likelihood-ratio statistics target the same null hypothesis and have the same first-order large-sample calibration. Their calculations nevertheless use different information. The Wald test starts from the unrestricted estimate and its covariance; the score test evaluates the likelihood’s slope at the restricted fit; the likelihood-ratio test compares the maximized likelihoods from both fits. Keeping those inputs separate is essential when translating a model fit into a test statistic.

Let \(\theta\) be a \(p\)-dimensional parameter and consider \(H_0:r(\theta)=0\), where \(r\) has \(q\) components and its derivative has rank \(q\) near the null. Write \(\hat\theta\) for the unrestricted maximum likelihood estimate and \(\tilde\theta\) for the maximum likelihood estimate subject to \(H_0\). The calculations below use expected Fisher information for the score test and the estimated covariance of the unrestricted maximum likelihood estimate for the Wald test.

Definition: The Wald statistic measures how far the unrestricted estimate is from satisfying the restrictions, using its estimated covariance. The score statistic uses the score (the derivative of the log-likelihood) evaluated at the restricted estimate, scaled by information for movement away from the restricted model. The likelihood-ratio statistic is twice the difference between the unrestricted and restricted maximized log-likelihoods: $$ W=r(\hat\theta)^{T} \left\{R(\hat\theta)\widehat{\operatorname{Var}}(\hat\theta)R(\hat\theta)^{T}\right\}^{-1} r(\hat\theta), \qquad R(\theta)=\frac{\partial r(\theta)}{\partial\theta^{T}}, $$ $$ S=U_{\psi\cdot\lambda}(\tilde\theta)^{T} I_{\psi\cdot\lambda}(\tilde\theta)^{-1} U_{\psi\cdot\lambda}(\tilde\theta), \qquad LR=2\{\ell(\hat\theta)-\ell(\tilde\theta)\}. $$ Here the score expression is written for a partition \(\theta=(\psi^{T},\lambda^{T})^{T}\) with \(H_0:\psi=\psi_0\). The efficient score and information are \(U_{\psi\cdot\lambda}=U_\psi-I_{\psi\lambda}I_{\lambda\lambda}^{-1}U_\lambda\) and \(I_{\psi\cdot\lambda}=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\), evaluated at the restricted fit. When the nuisance score is zero there, the efficient score equals \(U_\psi\).

Under regularity conditions—including an interior true parameter under the null, identifiable parameters, differentiable likelihoods, nonsingular relevant information, and maximum likelihood estimates with the usual large-sample behavior—each statistic is asymptotically chi-square with \(q\) degrees of freedom under \(H_0\). This shared calibration does not make their finite-sample calculations interchangeable.

What Each Workflow Requires

TestFit or evaluation neededWhat is scaled or compared?
WaldUnrestricted fit \(\hat\theta\)Restriction evaluated at \(\hat\theta\), with unrestricted covariance
ScoreRestricted fit \(\tilde\theta\)Score at the restriction, with restricted-fit efficient information
Likelihood ratioBoth unrestricted and restricted fitsDifference in their maximized log-likelihoods

The score test does not require fitting the unrestricted model, although it does require evaluating the score for directions that leave the null. Conversely, the Wald test does not use the restricted maximum likelihood value. In practice, a software routine may internally compute additional quantities, but these are the statistical inputs that define each test.

Worked Example: Two Restrictions in Gaussian Regression

Worked Example: Two Restrictions in Gaussian Regression

Suppose \(Y_i\) are independent with \(Y_i\sim N(\alpha+\beta_1x_{i1}+\beta_2x_{i2},\sigma^2)\), where \(\sigma^2\) is unknown. Use the four observations with design columns \(1\), \(x_1=(1,-1,0,0)^T\), and \(x_2=(1,1,-2,0)^T\), and responses \(y=(3,1,1,3)^T\). Test \(H_0:\beta_1=\beta_2=0\). The two slope columns are centered and orthogonal, with squared lengths 2 and 6.

Restricted fit. Under the null, the fitted mean is the sample mean, \(\tilde\alpha=2\). The residuals are \((1,-1,-1,1)\), so \(RSS_0=1+1+1+1=4\). The Gaussian maximum likelihood estimate under the null is \(\tilde\sigma^2=RSS_0/n=4/4=1\).

Unrestricted fit. Orthogonality gives \(\hat\alpha=2\), \(\hat\beta_1=(x_1^Ty)/(x_1^Tx_1)=2/2=1\), and \(\hat\beta_2=(x_2^Ty)/(x_2^Tx_2)=2/6=1/3\). The fitted means are \((10/3,4/3,4/3,2)\), and subtracting these from \(y\) gives residuals \((-1/3,-1/3,-1/3,1)\). Therefore $$ RSS_1=\frac{1}{9}+\frac{1}{9}+\frac{1}{9}+1=\frac{4}{3}, \qquad \hat\sigma^2=\frac{RSS_1}{4}=\frac{1}{3}. $$

Wald statistic. The unrestricted covariance estimate for the two slope estimates is \(\hat\sigma^2\operatorname{diag}(1/2,1/6)=\operatorname{diag}(1/6,1/18)\). Thus $$ W=\frac{1^2}{1/6}+\frac{(1/3)^2}{1/18}=6+2=8. $$

Score statistic. At the restricted fit, the slope score is \((x_1^Ty,x_2^Ty)^T/\tilde\sigma^2=(2,2)^T\). Because the centered slope columns are orthogonal to the intercept, nuisance adjustment for \(\alpha\) leaves their information unchanged. At \(\tilde\sigma^2=1\), the efficient information for the slopes is \(\operatorname{diag}(2,6)\). Hence $$ S=(2,2) \begin{pmatrix}1/2&0\\0&1/6\end{pmatrix} \begin{pmatrix}2\\2\end{pmatrix} =2+\frac{2}{3}=\frac{8}{3}. $$

Likelihood ratio. For a Gaussian linear model with unknown variance, maximizing over \(\sigma^2\) gives \(\hat\sigma^2=RSS/n\) and a maximized log-likelihood of the form \(-\frac{n}{2}\log(RSS/n)\) plus terms common to both fits. Therefore \(LR=n\log(RSS_0/RSS_1)=4\log(4/(4/3))=4\log 3\approx4.394\). The three values differ here: \(W=8\), \(S=8/3\), and \(LR\approx4.394\). Each uses the fit specified by its definition.

Why the Gaussian Likelihood-Ratio Formula Works

The previous calculation used a convenient formula for a normal linear model with unknown variance. It follows directly by profiling out the variance; no additional approximation is needed for the identity itself.

Derivation: If the residual sum of squares at a fit is \(RSS\) for \(n\) independent normal observations, then $$ \ell=-\frac{n}{2}\log(2\pi\sigma^2)-\frac{RSS}{2\sigma^2}. $$ Differentiating with respect to \(\sigma^2\) gives \(-n/(2\sigma^2)+RSS/(2(\sigma^2)^2)=0\), so the maximizing value is \(\hat\sigma^2=RSS/n\). Substitution gives \(\ell_{\max}=-\frac{n}{2}\{\log(2\pi)+1+\log(RSS/n)\}\). The constants cancel between nested fits, yielding \(LR=n\log(RSS_0/RSS_1)\). Since the unrestricted fit contains the restricted one, \(RSS_1\leq RSS_0\), so this statistic is nonnegative. \(\square\)

Worked Example: A Location Test with Unknown Variance

Worked Example: A Location Test with Unknown Variance

Let \(Y_1,Y_2,Y_3\) be independent \(N(\mu,\sigma^2)\) observations, with both parameters unknown. For data \((1,-1,2)\), test \(H_0:\mu=0\).

The unrestricted estimate is \(\hat\mu=2/3\). Its residuals are \((1/3,-5/3,4/3)\), so \(RSS_1=(1+25+16)/9=14/3\), and \(\hat\sigma^2=(14/3)/3=14/9\). Under the null, \(\tilde\mu=0\), \(RSS_0=1+1+4=6\), and \(\tilde\sigma^2=6/3=2\).

For the Wald test, the estimated variance of \(\hat\mu\) is \(\hat\sigma^2/n=14/27\), giving \(W=(2/3)^2/(14/27)=6/7\). For the score test, the score for \(\mu\) at the null fit is \(\sum_i y_i/\tilde\sigma^2=2/2=1\). The score test uses expected Fisher information; in this normal model the expected cross-information between \(\mu\) and \(\sigma^2\) is zero, so the efficient information for \(\mu\) is \(n/\tilde\sigma^2=3/2\).

At the restricted fit, the expected Fisher information has \(I_{\mu\mu}=n/\sigma^2=3/2\), \(I_{\mu,\sigma^2}=0\), and \(I_{\sigma^2,\sigma^2}=n/(2(\sigma^2)^2)=3/8\). Thus \(I_{\mu\cdot\sigma^2}=3/2\), and \(U_{\mu\cdot\sigma^2}=U_\mu=1\), so \(S=1/(3/2)=2/3\). Finally, $$ LR=3\log\left(\frac{6}{14/3}\right)=3\log(9/7)\approx0.7539. $$ The example illustrates why nuisance adjustment in a score calculation must be checked rather than assumed; here the expected cross-information is zero, so the efficient information for \(\mu\) is \(3/2\).

Worked Example: Known-Variance Regression and a Shared Value

Worked Example: Known-Variance Regression and a Shared Value

Suppose \(Y_i\sim N(\alpha+\beta x_i,1)\) independently, with \(x=(-1,0,1)\) and \(y=(0,1,3)\). Test \(H_0:\beta=0\), treating the variance as known to be 1.

The unrestricted least-squares estimates are \(\hat\alpha=\bar y=4/3\) and \(\hat\beta=\sum_i x_i(y_i-\bar y)/\sum_i x_i^2=(3/2)\), since the numerator is 3 and the denominator is 2. The unrestricted residuals are \((1/6,-1/3,1/6)\), giving \(RSS_1=1/6\). The unrestricted slope variance is \(1/2\), so \(W=(3/2)^2/(1/2)=9/2\).

Under the null, \(\tilde\alpha=4/3\), and the residuals are \((-4/3,-1/3,5/3)\); their squared sum is \(14/3\). The restricted slope score is \(\sum_i x_i(y_i-\tilde\alpha)=3\), and its information is \(\sum_i x_i^2=2\), so \(S=3^2/2=9/2\). With known variance, the maximized log-likelihood difference is half the reduction in residual sum of squares. Therefore \(LR=RSS_0-RSS_1=14/3-1/6=9/2\). Here the statistics coincide exactly, a consequence of the quadratic Gaussian likelihood with known variance, not a general rule for other models.

Efficient Information When There Are Nuisance Parameters

For a score test, a nuisance parameter is estimated under the null, and the score for that nuisance parameter is zero at an interior restricted maximum. The score for the tested parameter must still be scaled for the fact that nuisance parameters could adjust when moving away from the null. The efficient-information expression in the definition accounts for that adjustment.

Derivation: Partition the expected information into blocks \(I_{\psi\psi},I_{\psi\lambda},I_{\lambda\psi},I_{\lambda\lambda}\). A local change in \(\psi\) accompanied by the information-optimal adjustment in \(\lambda\) has nuisance component \(-I_{\lambda\lambda}^{-1}I_{\lambda\psi}\) times the change in \(\psi\). Substituting this adjustment into the quadratic information form gives $$ I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}. $$ Likewise, the score along the adjusted direction is \(U_\psi-I_{\psi\lambda}I_{\lambda\lambda}^{-1}U_\lambda\). At the restricted maximum \(U_\lambda=0\), but the efficient information still accounts for nuisance adjustment. This is why evaluating the tested score at the null is not, by itself, the whole score-test calculation. \(\square\)

Common Pitfalls and an Audit Checklist

A small arithmetic error in a residual or variance estimate can alter all three statistics, so audit the fits before comparing test values. In the first example, for instance, the third unrestricted fitted mean is \(2+(1/3)(-2)=4/3\), not \(1/3\). The corresponding residual is \(-1/3\); using a different value would incorrectly change \(RSS_1\), the variance estimate, the Wald covariance, and the likelihood ratio.

  • Using the wrong fit. Wald uses the unrestricted estimate and covariance; score uses the restricted fit; likelihood ratio needs both maximized log-likelihoods.
  • Confusing variance conventions. The Gaussian maximum likelihood estimate is \(RSS/n\), not the degrees-of-freedom-corrected residual variance. If a covariance estimate uses a different convention, state it and use it consistently.
  • Ignoring nuisance adjustment. Use efficient information for a score test with nuisance parameters; do not automatically use the tested-parameter block alone.
  • Assuming the statistics must match. They can differ in finite samples, as the unknown-variance regression example shows. Equality in a special model does not establish equality in general.
  • Forgetting nesting checks. For nested Gaussian least-squares fits, verify \(RSS_1\leq RSS_0\). A larger unrestricted residual sum of squares signals a fitting or arithmetic problem.

In computational work, the clearest report records the restricted and unrestricted parameter estimates, the covariance or information matrix used, and the resulting statistic. That makes the test reproducible and makes it easier to diagnose whether a discrepancy comes from optimization, nuisance-parameter handling, or arithmetic.

Check Your Understanding

For each question, identify the relevant fit or quantity before calculating.

  1. Which fit and covariance estimate enter a Wald statistic?
  2. In a score test with nuisance parameters, why can the nuisance score be zero while nuisance adjustment still matters?
  3. For nested Gaussian linear models with unknown variance, which residual sums of squares determine the likelihood-ratio statistic?
  4. In the first worked example, verify the third fitted value and explain how an error there would affect the reported tests.
  5. Why do equal Wald, score, and likelihood-ratio statistics in the known-variance example not imply that the three procedures always produce the same value?

Previous

Next

Back to Tutorials
Clinical Biostats