Why the Score Statistic Is Asymptotically Chi-Square

Why a Score Statistic Needs Standardization

A score measures how rapidly the log-likelihood changes when a parameter moves away from its null value. Under the null hypothesis, that score is centered near zero, but its scale depends on how much information the data contain. A score of a given size is more surprising when the likelihood is sharply concentrated than when it is relatively flat.

The score test accounts for that scale by dividing its quadratic magnitude by Fisher information. The asymptotic reason this works is precise: under regularity conditions, a standardized score is approximately a standard normal vector. The squared length of a standard normal vector has a chi-square distribution. With nuisance parameters, the relevant scale is the information remaining for the tested parameter after accounting for the nuisance parameters.

Score Test With Nuisance Parameters

Consider independent, identically distributed observations from a regular parametric model with parameter \((\psi,\lambda)\). The tested parameter \(\psi\) has dimension \(q\), and \(\lambda\) is a nuisance parameter of dimension \(r\). The null hypothesis is \(H_0:\psi=\psi_0\); the nuisance parameter is not fixed by that hypothesis. Write the per-observation Fisher information at the true null parameter \((\psi_0,\lambda_0)\) in blocks:

Assume \(i_{\lambda\lambda}\) is nonsingular. The information relevant to testing \(\psi\), after allowing for estimation of \(\lambda\), is the Schur complement \(i_{\psi\psi\cdot\lambda}=i_{\psi\psi}-i_{\psi\lambda}i_{\lambda\lambda}^{-1}i_{\lambda\psi}\). Under the null, fit \(\lambda\) subject to \(\psi=\psi_0\), obtaining the restricted maximum likelihood estimator \(\tilde\lambda\). The score test uses the score for \(\psi\) at this restricted fit and the corresponding efficient information.

For this limit, suppose the true parameter is an interior point of an open parameter space; the model has a common dominating measure; the log density is sufficiently differentiable; and differentiation under the integral is valid so that the score has mean zero and its covariance equals Fisher information. Assume finite, positive-definite Fisher information, a central limit theorem for the score, and a law of large numbers for the likelihood Hessian. Also assume the restricted estimator is consistent, has the usual first-order likelihood expansion, and the estimated efficient information converges in probability to \(i_{\psi\psi\cdot\lambda}\), which is positive definite. These conditions rule out, among other things, boundary parameters and singular information.

Worked Examples

Why the Chi-Square Limit Holds

The first step is to account for the fact that the nuisance parameter is estimated under the null. Let \(U_{\psi,n}\) and \(U_{\lambda,n}\) denote the two components of the score at the true null parameter. The restricted likelihood expansion gives \(\tilde\lambda-\lambda_0=i_{\lambda\lambda}^{-1}U_{\lambda,n}/n+o_p(n^{-1/2})\). A Taylor expansion of the tested score in \(\lambda\) then yields

Here is the derivation. The derivative of the score \(U_{\psi,n}\) with respect to \(\lambda\), divided by \(n\), converges in probability to \(-i_{\psi\lambda}\). Taylor expansion and the restricted-estimator expansion therefore give

The joint central limit theorem says that \(n^{-1/2}(U_{\psi,n}^T,U_{\lambda,n}^T)^T\) converges to a mean-zero normal vector with covariance \(i\). The leading term in the displayed expansion is the linear transformation with matrix \((I_q,\,-i_{\psi\lambda}i_{\lambda\lambda}^{-1})\). Its covariance is

The remainder divided by \(\sqrt n\) converges in probability to zero. The central limit theorem and Slutsky’s theorem now give the claimed normal limit. This calculation explains why using only \(i_{\psi\psi}\) can be wrong when the nuisance parameter and tested parameter are not orthogonal: estimating the nuisance parameter changes the score’s limiting covariance. \(\square\)

The final step converts that normal limit into a chi-square limit.

To prove the theorem, write \(\hat i_{\mathrm{eff}}=\hat i_{\psi\psi\cdot\lambda}\), and let \(i_{\mathrm{eff}}=i_{\psi\psi\cdot\lambda}\). Positive definiteness and consistency imply that \(\hat i_{\mathrm{eff}}^{-1}\xrightarrow{p}i_{\mathrm{eff}}^{-1}\). The previous result says that \(n^{-1/2}U_{\psi,n}(\psi_0,\tilde\lambda)\) converges to a normal vector \(Z\sim N_q(0,i_{\mathrm{eff}})\). Consequently, by Slutsky’s theorem and continuity of the quadratic-form map,

Because \(i_{\mathrm{eff}}\) is positive definite, it has a symmetric inverse square root. The vector \(i_{\mathrm{eff}}^{-1/2}Z\) has distribution \(N_q(0,I_q)\), and \(Z^T i_{\mathrm{eff}}^{-1}Z=\lVert i_{\mathrm{eff}}^{-1/2}Z\rVert^2\). By definition, the squared length of \(q\) independent standard normal coordinates has a \(\chi^2_q\) distribution. This proves the limit. \(\square\)

Common Mistake: Squaring the Raw Score

A raw score is not generally on a fixed scale. For example, in the Bernoulli model the score’s variance under the null is \(n/[p_0(1-p_0)]\); it changes with both sample size and the null probability. Squaring the score without dividing by its information does not produce a statistic with a standard chi-square limit. Standardization is essential.

With nuisance parameters, another tempting error is to use the tested block \(i_{\psi\psi}\) as though the nuisance parameter were known. When \(i_{\psi\lambda}\ne0\), that ignores the effect of estimating \(\lambda\). The correction is the efficient information \(i_{\psi\psi\cdot\lambda}\). Finally, the degrees of freedom are \(q\), the dimension of the tested parameter, not the total number of model parameters.