Score-Type Tests From Estimating Equations

Why a Restricted Estimating Equation Can Test a Restriction

A Wald test starts with an unrestricted estimate and asks how far it lies from a proposed restriction. A score-type test takes a different route: fit the model while imposing the null, then evaluate the estimating equation for the parameters constrained by that null. If the restriction is compatible with the data, this equation should be small on its natural asymptotic scale.

The phrase score-type is important. In likelihood theory, a score is the derivative of a log-likelihood. Here the test can be built from more general unbiased estimating equations, including quasi-score equations. There need not be a likelihood, and the covariance calculation must account for both the variability of the equations and the effect of estimating nuisance parameters.

Definition: Let \(\theta=(\gamma^\mathsf{T},\beta^\mathsf{T})^\mathsf{T}\), where \(\gamma\) is a nuisance parameter of dimension \(q\) and \(\beta\) is the parameter being tested, of dimension \(r\). Suppose independent, identically distributed observations contribute estimating functions \(\psi_i(\theta)\), partitioned as \((\psi_{\gamma i}^\mathsf{T},\psi_{\beta i}^\mathsf{T})^\mathsf{T}\), with \(E[\psi_i(\theta_0)]=0\) under the true parameter \(\theta_0\). For the null hypothesis \(H_0:\beta=\beta_0\), let \(\hat\gamma_0\) solve \(\sum_{i=1}^n\psi_{\gamma i}(\gamma,\beta_0)=0\). The restricted score-type vector is \(S_n=\sum_{i=1}^n\psi_{\beta i}(\hat\gamma_0,\beta_0)\).

The restricted fit solves the nuisance equations, not generally the tested equations. Thus \(S_n\) need not equal zero, and it need not have expectation exactly zero in a finite sample. Its useful reference is its asymptotic distribution after scaling by \(\sqrt n\). That distribution depends on how fitting \(\gamma\) changes the tested equation.

Adjusting for Nuisance Estimation

Define the sensitivity and variability matrices at the null parameter by

H=-E\!\left[\frac{\partial \psi_i(\theta_0)}{\partial\theta^\mathsf{T}}\right], \qquad J=\operatorname{Var}\!\left(\psi_i(\theta_0)\right).

Partition these matrices conformably with \((\gamma,\beta)\). Assume \(H_{\gamma\gamma}\) is nonsingular. The restricted estimate of \(\gamma\) responds to the random fluctuation in its own estimating equation. That response feeds into the tested equation through \(H_{\beta\gamma}\).

Theorem: Suppose the observations are independent and identically distributed; \(E[\psi_i(\theta_0)]=0\); \(J\) is finite; the estimating functions are differentiable near \(\theta_0\); and the sample average of their derivatives converges locally to its expectation. Suppose \(H_{\gamma\gamma}\) is nonsingular, the restricted solution is consistent and root-\(n\) consistent, and the Taylor remainder for the restricted estimating equations is \(o_p(n^{-1/2})\) after division by \(n\). Then, under \(H_0\), \(S_n/\sqrt n\xrightarrow{d}N(0,C)\), where \(C=BJB^\mathsf{T}\) and \(B=(-H_{\beta\gamma}H_{\gamma\gamma}^{-1},\,I_r)\). If \(C\) is positive definite and a consistent estimate \(\hat C\) is used, then \(S_n^\mathsf{T}(n\hat C)^{-1}S_n\xrightarrow{d}\chi^2_r\).

The matrix \(B\) describes the nuisance adjustment. In particular, the asymptotic random vector is formed from the tested equation minus the part predictable, to first order, from nuisance-equation fluctuations. The covariance \(C\) is therefore not generally just \(J_{\beta\beta}\).

Expand the nuisance equation.
At the true null value, write \(U_{\gamma,n}=\sum_i\psi_{\gamma i}(\theta_0)\). A first-order expansion of the restricted equation \(0=\sum_i\psi_{\gamma i}(\hat\gamma_0,\beta_0)\) gives \(\sqrt n(\hat\gamma_0-\gamma_0)=H_{\gamma\gamma}^{-1}U_{\gamma,n}/\sqrt n+o_p(1)\).

Expand the tested equation.
Expanding around \(\theta_0\) gives \(S_n/\sqrt n=U_{\beta,n}/\sqrt n-H_{\beta\gamma}H_{\gamma\gamma}^{-1}U_{\gamma,n}/\sqrt n+o_p(1)\).

Apply the central limit theorem.
The vector \((U_{\gamma,n}^\mathsf{T},U_{\beta,n}^\mathsf{T})^\mathsf{T}/\sqrt n\) converges to a mean-zero normal vector with covariance \(J\). Multiplication by \(B\) gives covariance \(BJB^\mathsf{T}=C\).

Form the quadratic test.
Consistency of \(\hat C\) and the continuous mapping theorem give a quadratic form in an \(r\)-dimensional standard normal limit, which has a \(\chi^2_r\) distribution.

For completeness, the first expansion follows by writing the derivative of the sample nuisance equation as \(-nH_{\gamma\gamma}+o_p(n)\). Solving its Taylor expansion for \(\hat\gamma_0-\gamma_0\) yields the expression in Step 1. Substituting that expression into the Taylor expansion of the tested equation gives Step 2. The multivariate central limit theorem applies because the estimating functions have finite second moments. Finally, if \(Z\sim N(0,C)\) and \(C\) is positive definite, then \(Z^\mathsf{T}C^{-1}Z\sim\chi^2_r\); replacing \(C\) by a consistent estimate preserves this limit by Slutsky's theorem. \(\square\)

In practice, estimate \(H\) with the negative average derivative of the estimating functions at the restricted fit, and estimate \(J\) with the empirical variability of those functions there. Then construct \(\hat B\) and \(\hat C=\hat B\hat J\hat B^\mathsf{T}\). The estimated covariance of the total vector \(S_n\) is \(n\hat C\), which is why the quadratic statistic uses \(n\hat C\), not merely \(\hat C\).

Worked Score-Type Tests

Worked Example: A Robust Test of a Regression Slope

Consider the estimating equations for a straight-line mean \(E[Y_i]=\gamma+\beta x_i\), with independent observations and fixed \(x_i\): \(\psi_{\gamma i}=Y_i-\gamma-\beta x_i\) and \(\psi_{\beta i}=x_i(Y_i-\gamma-\beta x_i)\). Test \(H_0:\beta=0\) using the four observations \((x_i,y_i)=(0,1),(1,2),(2,2),(3,5)\). This small calculation illustrates the statistic; its chi-square approximation is an asymptotic calibration, not a claim of accurate finite-sample size for four observations.

Under the restriction, the intercept estimate is \(\hat\gamma_0=(1+2+2+5)/4=2.5\). The residuals are \((-1.5,-0.5,-0.5,2.5)\). The nuisance equation sums to zero, while the tested equation is \(S_n=\sum_i x_i(y_i-\hat\gamma_0)=0-0.5-1+7.5=6\).

For these equations, the sensitivity blocks at the restricted fit are \(H_{\gamma\gamma}=1\) and \(H_{\beta\gamma}=\bar x=1.5\). Thus the nuisance-adjusted contribution for observation \(i\) is \(\psi_{\beta i}-1.5\psi_{\gamma i}=(x_i-1.5)(y_i-2.5)\). The four values are \(2.25,0.25,-0.25,3.75\), which sum to \(6\), as required.

The empirical variability estimate for this adjusted equation is the average squared contribution: \(\hat C=(2.25^2+0.25^2+(-0.25)^2+3.75^2)/4=19.25/4=4.8125\). Therefore the estimated covariance of \(S_n\) is \(n\hat C=4(4.8125)=19.25\), and the score-type statistic is \(6^2/19.25\approx1.87\).

The calculation uses the observed variability of the estimating functions rather than assuming constant error variance. With a larger sample and the stated regularity conditions, the statistic is compared with a \(\chi^2_1\) reference distribution.

Worked Example: Sensitivity Does Not Determine Variability

Suppose a two-parameter estimating equation has one nuisance parameter \(\gamma\) and one tested parameter \(\beta\). At the null, suppose its per-observation sensitivity and variability matrices are \(H=\begin{pmatrix}2&1\\1&3\end{pmatrix}\) and \(J=\begin{pmatrix}4&0\\0&9\end{pmatrix}\), respectively. These are illustrative matrices for an estimating equation, not an assertion that \(J=H\).

Here \(H_{\gamma\gamma}=2\) and \(H_{\beta\gamma}=1\), so \(B=(-1/2,1)\). The adjusted variability is \(C=BJB^\mathsf{T}=(-1/2,1)\begin{pmatrix}4&0\\0&9\end{pmatrix}\begin{pmatrix}-1/2\\1\end{pmatrix}=1+9=10\). Using only the raw tested-equation variability would give \(J_{\beta\beta}=9\), which is not the correct adjusted covariance.

For example, if a restricted fit in a sample of \(n=100\) produced \(S_n=8\), the score-type statistic using these covariance values would be \(8^2/(100\times10)=0.064\). The example separates the calculation from its interpretation: the test's reference distribution is asymptotic, and a particular statistic must be judged against that distribution.

Worked Example: A Restricted Score Need Not Be Unbiased in Finite Samples

Let \(X_i\) be independent \(N(\gamma,\sigma^2)\) observations, with known \(\sigma^2\). Use the estimating functions \(\psi_{\gamma i}=X_i-\gamma\) and \(\psi_{\beta i}=\sigma^2-(X_i-\gamma)^2-\beta\), and test \(H_0:\beta=0\). At the null, the restricted estimate is \(\hat\gamma_0=\bar X\), so \(S_n=n\sigma^2-\sum_i(X_i-\bar X)^2\).

The normal-sample identity \(\sum_i(X_i-\bar X)^2/\sigma^2\sim\chi^2_{n-1}\) gives \(E[S_n]=n\sigma^2-(n-1)\sigma^2=\sigma^2\), not zero. Thus it would be incorrect to say that the restricted score has zero expectation exactly.

The asymptotic result concerns \(S_n/\sqrt n\). In this example, \[ \frac{S_n}{\sqrt n} =\sigma^2\left\{\frac{1}{\sqrt n} -\frac{\chi^2_{n-1}-(n-1)}{\sqrt n}\right\} \xrightarrow{d}N(0,2\sigma^4), \] because the centered chi-square term divided by \(\sqrt n\) converges to \(N(0,2)\). The finite-sample mean \(\sigma^2/\sqrt n\) of \(S_n/\sqrt n\) tends to zero, so it does not contradict the zero-mean asymptotic limit.

The covariance adjustment agrees with this result. At the null, \(H_{\gamma\gamma}=1\), \(H_{\beta\gamma}=-E[2(X_i-\gamma)]=0\), and \(J_{\beta\beta}=\operatorname{Var}(\sigma^2-(X_i-\gamma)^2)=2\sigma^4\). Hence \(C=2\sigma^4\), and the asymptotic statistic is \(S_n^2/(2n\sigma^4)\), with a \(\chi^2_1\) limit.

How This Differs From a Likelihood Score Test

For a regular, correctly specified likelihood, the estimating function is the likelihood score. The information identity then equates score variability and sensitivity, \(J=H\), and the sensitivity matrix is symmetric. In the partitioned setting above, the adjusted covariance reduces to the Schur complement \(H_{\beta\beta}-H_{\beta\gamma}H_{\gamma\gamma}^{-1}H_{\gamma\beta}\). That special structure underlies the classical likelihood score test.

For general estimating equations, neither \(J=H\) nor symmetry of \(H\) is guaranteed. The test still uses the same broad strategy—evaluate the tested equation at a restricted fit—but its covariance is \(BJB^\mathsf{T}\). This is the estimating-equation analogue, not a likelihood-ratio test, and its calibration requires a consistent estimate of the relevant variability.

Common Mistake

Wrong reasoning: “The null makes the tested equation unbiased, so its value at the restricted estimate must have expectation zero, and I can use its raw variability.”
Correction: Unbiasedness is stated at the true parameter. Evaluating at an estimated nuisance parameter changes the equation, and may produce a nonzero finite-sample expectation. The relevant general result is a zero-mean asymptotic normal limit for \(S_n/\sqrt n\), with covariance \(BJB^\mathsf{T}\), not necessarily the raw block \(J_{\beta\beta}\).

Key takeaway: A score-type test from estimating equations evaluates the tested equation at the restricted fit, then accounts for nuisance estimation through sensitivity and variability. Under regularity conditions, its robust quadratic statistic has an asymptotic chi-square distribution; finite-sample zero expectation and the likelihood information identity should not be assumed.

Check Your Understanding

Use the definitions and covariance adjustment above to answer these questions.

Why does the restricted fit solve the nuisance equations but not generally the tested equations?
In the theorem, what role does \(H_{\beta\gamma}\) play in the adjusted score vector?
Under what condition does the estimating-equation covariance reduce to the likelihood information Schur complement?
Why is the estimated covariance of the total restricted score \(n\hat C\), rather than \(\hat C\)?
In the normal example, how can \(E[S_n]\ne0\) while \(S_n/\sqrt n\) has a zero-mean asymptotic limit?