Deriving the Score Test at the Null Estimate

Why Evaluate the Score at the Null Fit?

A Wald test starts from an unrestricted estimate and asks how far it lies from the null in estimated standard-error units. A score test reverses the starting point: it fits the model under the null, then asks whether the likelihood gradient at that fit points strongly toward a better-fitting parameter value. Its practical attraction is that the unrestricted model need not be fitted.

The gradient alone is not a measure of evidence: its size depends on the scale of the parameter and the amount of information in the data. The score test standardizes that gradient by the information at the null-constrained estimate. With nuisance parameters, the relevant information must also account for the fact that those nuisance parameters are estimated under the null.

Definition: Let $\ell_n(\theta)=\sum_{i=1}^n \ell_i(\theta)$ be the log-likelihood for independent observations, with parameter $\theta$. The full-sample score is $U_n(\theta)=\partial\ell_n(\theta)/\partial\theta$, and the full-sample observed information is $J_n(\theta)=-\partial^2\ell_n(\theta)/\partial\theta\partial\theta^\mathsf{T}$. The full-sample Fisher information is $I_n(\theta)=E_\theta[J_n(\theta)]$, when the expectation exists. For identically distributed observations, the per-observation Fisher information is $I_1(\theta)=E_\theta[-\partial^2\ell_1(\theta)/\partial\theta\partial\theta^\mathsf{T}]$, so $I_n(\theta)=nI_1(\theta)$. For independent, non-identically distributed observations, $I_n$ is the sum of their individual information matrices.

Consider a parameter split as $\theta=(\psi^\mathsf{T},\lambda^\mathsf{T})^\mathsf{T}$, where $\psi$ has dimension $r$ and is the parameter block being tested, while $\lambda$ denotes nuisance parameters. Test $H_0:\psi=\psi_0$. Write $\tilde{\lambda}$ for the restricted maximum-likelihood estimate under this null. At the restricted fit, the nuisance score is zero, $U_\lambda(\psi_0,\tilde{\lambda})=0$, provided the restricted maximum is an interior point and the likelihood is differentiable there. The score for $\psi$, however, need not be zero.

Definition: Partition the full-sample Fisher information at the null-constrained estimate into blocks $I_{\psi\psi}, I_{\psi\lambda}, I_{\lambda\psi}, I_{\lambda\lambda}$. If $I_{\lambda\lambda}$ is invertible, define the nuisance-adjusted information for $\psi$ by $$ I_{\mathrm{eff}}=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}. $$ The Fisher score statistic is $$ S=U_\psi(\psi_0,\tilde{\lambda})^\mathsf{T} I_{\mathrm{eff}}^{-1}U_\psi(\psi_0,\tilde{\lambda}). $$ All score and information quantities in this definition are evaluated at $(\psi_0,\tilde{\lambda})$. When there are no nuisance parameters, this reduces to $S=U_n(\psi_0)^\mathsf{T}I_n(\psi_0)^{-1}U_n(\psi_0)$.

Why Nuisance Parameters Change the Information

The adjustment is not an optional correction. A change in $\psi$ can be partly offset by a change in $\lambda$, so the information relevant to moving in the tested direction is smaller than the unadjusted block $I_{\psi\psi}$. The following derivation makes this precise using the local quadratic approximation to the log-likelihood.

Derivation: Suppose the observed information at the restricted fit is positive definite and partition it into blocks $J_{\psi\psi},J_{\psi\lambda},J_{\lambda\psi},J_{\lambda\lambda}$. The second-order log-likelihood change for a small displacement $(\delta_\psi,\delta_\lambda)$ from that fit is $$ q(\delta_\psi,\delta_\lambda) =U_\psi^\mathsf{T}\delta_\psi -\frac{1}{2} \begin{pmatrix}\delta_\psi\\\delta_\lambda\end{pmatrix}^{\mathsf{T}} \begin{pmatrix}J_{\psi\psi}&J_{\psi\lambda}\\J_{\lambda\psi}&J_{\lambda\lambda}\end{pmatrix} \begin{pmatrix}\delta_\psi\\\delta_\lambda\end{pmatrix}, $$ because $U_\lambda=0$ at the restricted fit. For a fixed $\delta_\psi$, differentiating with respect to $\delta_\lambda$ gives $-J_{\lambda\psi}\delta_\psi-J_{\lambda\lambda}\delta_\lambda=0$. Thus the nuisance displacement maximizing this quadratic approximation is $\delta_\lambda=-J_{\lambda\lambda}^{-1}J_{\lambda\psi}\delta_\psi$. Substitution gives $$ q_{\mathrm{prof}}(\delta_\psi) =U_\psi^\mathsf{T}\delta_\psi-\frac{1}{2}\delta_\psi^\mathsf{T} \left(J_{\psi\psi}-J_{\psi\lambda}J_{\lambda\lambda}^{-1}J_{\lambda\psi}\right)\delta_\psi. $$ The maximizing displacement is the score direction standardized by this adjusted curvature: $$ \widehat{\delta}_\psi=J_{\mathrm{eff}}^{-1}U_\psi, \qquad 2q_{\mathrm{prof}}(\widehat{\delta}_\psi)=U_\psi^\mathsf{T}J_{\mathrm{eff}}^{-1}U_\psi. $$ This is a local quadratic calculation, not an identity for the exact likelihood away from the restricted fit. It shows why the score statistic measures a squared direction toward improvement, scaled by the curvature after allowing nuisance parameters to respond.

The observed-information version replaces $I_{\mathrm{eff}}$ by $J_{\mathrm{eff}}=J_{\psi\psi}-J_{\psi\lambda}J_{\lambda\lambda}^{-1}J_{\lambda\psi}$, also evaluated at the restricted fit. Under regularity conditions, expected and observed information give asymptotically equivalent score statistics. They need not give identical finite-sample values.

Worked Score Tests

Worked Example: Logistic Regression Slope with an Intercept Nuisance Parameter

Suppose there are four observations at $x=0$ and four at $x=1$. There is one success among the four observations at $x=0$, and three among the four at $x=1$. Fit the logistic model $\operatorname{logit}(p_i)=\alpha+\beta x_i$ and test $H_0:\beta=0$. The intercept $\alpha$ is nuisance.

Under the null, all eight observations have probability $p$. The restricted estimate is $\tilde p=(1+3)/8=1/2$, so $\tilde\alpha=\log\{\tilde p/(1-\tilde p)\}=0$. For one Bernoulli observation, the score for $\beta$ is $x_i(y_i-p_i)$. Therefore, at the restricted fit, $$ U_\beta=\sum_i x_i(y_i-\tilde p) =3-4(1/2)=1. $$

For logistic regression, the expected information contribution is $p_i(1-p_i)(1,x_i)^\mathsf{T}(1,x_i)$. Here $p_i(1-p_i)=1/4$, and the information blocks are $$ I_{\alpha\alpha}=8(1/4)=2,\qquad I_{\alpha\beta}=4(1/4)=1,\qquad I_{\beta\beta}=4(1/4)=1. $$ Thus $I_{\mathrm{eff}}=1-1(2^{-1})1=1/2$, and $$ S=\frac{U_\beta^2}{I_{\mathrm{eff}}}=\frac{1^2}{1/2}=2. $$ The raw slope score is 1, but the adjusted information is $1/2$, not $I_{\beta\beta}=1$: the intercept can move to accommodate some of a slope change.

Worked Example: Poisson Intercept with No Nuisance Parameter

Let $Y_1,\ldots,Y_4$ be independent Poisson observations with $\log E[Y_i]=\alpha$. The observed counts are $0,1,2,3$. Test $H_0:\alpha=\log 2$. There are no nuisance parameters.

The log-likelihood, up to terms not depending on $\alpha$, is $\ell(\alpha)=\alpha\sum_i y_i-4e^\alpha$. Its score is $U(\alpha)=\sum_i y_i-4e^\alpha$, and its Fisher information is $I_4(\alpha)=4e^\alpha$. At the null, the count total is $6$, so $U(\log 2)=6-4(2)=-2$, while $I_4(\log 2)=8$. The score statistic is $$ S=\frac{(-2)^2}{8}=\frac{1}{2}. $$ This calculation uses the total information for all four observations. The per-observation information at the null is $I_1(\log 2)=2$, and $4I_1(\log 2)=8$, as required.

Worked Example: Gaussian Slope with Known Variance

Consider $Y_i\sim N(\alpha+\beta x_i,1)$ independently, with $x=(-1,-1,1,1)$ and observed $y=(0,1,2,3)$. Test $H_0:\beta=0$, treating $\alpha$ as nuisance. The variance is known to be 1.

Under the null, the restricted estimate is $\tilde\alpha=\bar y=(0+1+2+3)/4=1.5$. The slope score is $\sum_i x_i(y_i-\tilde\alpha)$, which is $$ (-1)(-1.5)+(-1)(-0.5)+(1)(0.5)+(1)(1.5)=4. $$ The information blocks are $I_{\alpha\alpha}=4$, $I_{\alpha\beta}=\sum_i x_i=0$, and $I_{\beta\beta}=\sum_i x_i^2=4$. Consequently $I_{\mathrm{eff}}=4-0(4^{-1})0=4$, and the score statistic is $S=4^2/4=4$. Centering of the covariate makes the intercept and slope information orthogonal in this example; the nuisance adjustment leaves the slope information unchanged.

Asymptotic Calibration

A score statistic is often compared with a chi-square distribution. The result below gives the standard calibration for testing an $r$-dimensional parameter block. Its assumptions matter: in particular, it is a regular interior-point result under a correctly specified likelihood, not a general guarantee for boundary or otherwise nonregular problems.

Theorem: Suppose $Y_1,\ldots,Y_n$ are independent and identically distributed from a correctly specified regular parametric model with parameter $(\psi,\lambda)$, and the null value $(\psi_0,\lambda_0)$ is an interior point. Assume the log-likelihood is sufficiently differentiable in a neighborhood of that value; differentiation and expectation can be interchanged so that the score has mean zero and its covariance equals the Fisher information; the per-observation Fisher information is finite and positive definite; the score satisfies a central limit theorem; and the restricted maximum-likelihood estimator $\tilde\lambda$ is consistent and root-$n$ consistent under $H_0$. If the restricted-fit information blocks converge to their population limits and the nuisance block is invertible, then under $H_0$, $S\xrightarrow{d}\chi^2_r$.

Proof. Let $u_i(\theta)$ be the one-observation score and let $I_1$ be the per-observation Fisher information at $\theta_0=(\psi_0,\lambda_0)$. Write $I_1$ in $\psi,\lambda$ blocks. A Taylor expansion of the nuisance score around $\theta_0$, using the restricted score equation $U_\lambda(\psi_0,\tilde\lambda)=0$, gives $$ 0=U_\lambda(\theta_0)-nI_{\lambda\lambda,1}(\tilde\lambda-\lambda_0)+o_p(\sqrt n). $$ Here the remainder is $o_p(\sqrt n)$ because $\tilde\lambda-\lambda_0=O_p(n^{-1/2})$, and the normalized Hessian converges to its expectation under the stated regularity conditions. Since $I_{\lambda\lambda,1}$ is invertible, $$ \tilde\lambda-\lambda_0 =I_{\lambda\lambda,1}^{-1}\frac{U_\lambda(\theta_0)}{n} +o_p(n^{-1/2}). $$ Expand the $\psi$-score in the same way and substitute this expression: $$ U_\psi(\psi_0,\tilde\lambda) =U_\psi(\theta_0)-I_{\psi\lambda,1}I_{\lambda\lambda,1}^{-1}U_\lambda(\theta_0)+o_p(\sqrt n). $$ Define $A=I_{\psi\lambda,1}I_{\lambda\lambda,1}^{-1}$. The leading term is the sum of independent transformed scores $u_{\psi i}-A u_{\lambda i}$, each with mean zero. Its covariance per observation is $$ \begin{aligned} \operatorname{Var}(u_{\psi i}-A u_{\lambda i}) &=I_{\psi\psi,1}-I_{\psi\lambda,1}A^\mathsf{T} -AI_{\lambda\psi,1}+AI_{\lambda\lambda,1}A^\mathsf{T}\\ &=I_{\psi\psi,1}-I_{\psi\lambda,1}I_{\lambda\lambda,1}^{-1}I_{\lambda\psi,1} =:I_{\mathrm{eff},1}. \end{aligned} $$ The multivariate central limit theorem therefore gives $$ \frac{U_\psi(\psi_0,\tilde\lambda)}{\sqrt n} \xrightarrow{d}N(0,I_{\mathrm{eff},1}). $$ The full-sample adjusted information at the restricted fit, divided by $n$, converges in probability to $I_{\mathrm{eff},1}$. Applying the continuous mapping theorem to the quadratic form gives a chi-square limit with $r$ degrees of freedom: after standardization, it is the sum of squares of $r$ independent standard normal variables. Hence $S\xrightarrow{d}\chi^2_r$. $\square$

Interpretation and Common Pitfalls

The score statistic is the squared length of the score direction when lengths are measured in the information metric at the null fit. A large value indicates that the likelihood gradient points strongly away from the null relative to the local curvature and the information available. The null model supplies the fit, score, and information needed for this calculation; an unrestricted fit is not needed to form the statistic.

Using the unrestricted estimate: A score test is not a Wald test evaluated at the restricted estimate. Evaluate the score and information at the null-constrained fit.
Ignoring nuisance adjustment: Using $I_{\psi\psi}$ alone treats nuisance parameters as fixed. When they are estimated under the null, use the Schur complement $I_{\mathrm{eff}}$.
Mixing per-observation and total information: For iid data, the sample information is $nI_1$. Pair a full-sample score with full-sample information; do not divide by an extra $n$ or omit the factor $n$.
Claiming an exact chi-square distribution: The chi-square result is an asymptotic null distribution under regularity conditions. It is not generally an exact finite-sample result.

Check Your Understanding

Answer each question using the score and information at the null-constrained fit.

In an iid model, how does full-sample Fisher information relate to per-observation Fisher information?
Why does a test of a slope with an estimated intercept generally require adjusted rather than unadjusted slope information?
For a scalar tested parameter with no nuisance parameters, write the Fisher score statistic.
In the logistic example, what are the restricted intercept estimate, the slope score, and the adjusted information?
Name two conditions needed for the asymptotic chi-square calibration, and explain why it should not automatically be treated as an exact finite-sample distribution.