Score Tests for Gaussian Restrictions

Why Evaluate the Score at the Restricted Fit?

A score test asks whether the likelihood would increase, to first order, if we moved away from a parameter restriction. Its defining feature is that the calculation uses the fit under the null hypothesis. This can be useful when fitting the unrestricted model is difficult, or when the null model has a simpler form. In a Gaussian model, the score and Fisher information can often be calculated directly from residual sums or cross-products.

The score alone is not a test statistic with a common scale: its variability depends on the model and on the parameterization. Fisher information supplies that scale. With nuisance parameters, the appropriate scale is the information remaining for the tested parameter after accounting for nuisance-parameter estimation.

Definition and Construction

Definition: Let \(X_1,\ldots,X_n\) be independent observations from a regular parametric model with log-likelihood \(\ell(\psi,\lambda)\). The parameter of interest is the \(q\)-dimensional vector \(\psi\), and \(\lambda\) is a nuisance-parameter vector. To test \(H_0:\psi=\psi_0\), let \(\tilde\lambda\) maximize the likelihood subject to \(\psi=\psi_0\). The restricted score for the tested parameter is \(U_\psi=\left.\partial\ell(\psi,\lambda)/\partial\psi\right|_{(\psi_0,\tilde\lambda)}\). Write the Fisher information at that fit in blocks as \(I_{\psi\psi}, I_{\psi\lambda}, I_{\lambda\psi}, I_{\lambda\lambda}\). The efficient information for \(\psi\) is \(I_{\mathrm{eff}}=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\). The score statistic is \(T_S=U_\psi^{\mathsf T}I_{\mathrm{eff}}^{-1}U_\psi\).

Here the information is for the full sample, not for one observation. When there are no nuisance parameters, \(I_{\mathrm{eff}}=I_{\psi\psi}\). If the restricted nuisance estimate is an interior maximum, its nuisance score is zero. The efficient-information expression then also gives the quadratic form obtained by inserting the full score vector \((U_\psi^{\mathsf T},0^{\mathsf T})^{\mathsf T}\) into the inverse of the full information matrix.

Why Efficient Information Appears

The nuisance parameters are re-estimated under the null. A change in \(\psi\) can be partly offset by a change in \(\lambda\), so the relevant curvature for testing \(\psi\) is not generally just the \(\psi,\psi\) block of the information. The following identity makes the adjustment precise.

Result: If \(I_{\lambda\lambda}\) is invertible and \(I_{\mathrm{eff}}\) is invertible, then \[ \begin{pmatrix}U_\psi\\0\end{pmatrix}^{\mathsf T} \begin{pmatrix}I_{\psi\psi}&I_{\psi\lambda}\\I_{\lambda\psi}&I_{\lambda\lambda}\end{pmatrix}^{-1} \begin{pmatrix}U_\psi\\0\end{pmatrix} =U_\psi^{\mathsf T}I_{\mathrm{eff}}^{-1}U_\psi. \]

To verify it, solve the block linear system \(I(x^{\mathsf T},y^{\mathsf T})^{\mathsf T}=(U_\psi^{\mathsf T},0^{\mathsf T})^{\mathsf T}\). The second block equation gives \(I_{\lambda\psi}x+I_{\lambda\lambda}y=0\), so \(y=-I_{\lambda\lambda}^{-1}I_{\lambda\psi}x\). Substitution into the first block equation yields \((I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi})x=U_\psi\), or \(x=I_{\mathrm{eff}}^{-1}U_\psi\). The quadratic form on the left is \(U_\psi^{\mathsf T}x\), which is the right-hand side. \(\square\)

For regular models with independent, identically distributed observations, an interior true parameter, nonsingular information, and the usual differentiability and moment conditions for likelihood asymptotics, \(T_S\xrightarrow{d}\chi^2_q\) under \(H_0\). The reason is that the restricted score, after nuisance estimation, is asymptotically centered normal with covariance \(I_{\mathrm{eff}}\). Scaling that normal vector by the inverse square root of its covariance gives \(q\) independent standard normal coordinates; their squared sum has a \(\chi^2_q\) distribution.

More explicitly, per-observation information can be denoted by \(I\), so full-sample information is \(nI\). Under the null, the unrestricted score at the true parameter has asymptotic covariance \(nI\). A first-order expansion of the tested score at the restricted nuisance estimate subtracts the component explained by the nuisance score, leaving covariance \(nI_{\mathrm{eff}}\). Replacing this information by its consistent estimate at the restricted fit gives the stated chi-square limit. This is an asymptotic calibration; the statistic does not in general have an exact chi-square distribution in a finite sample.

Worked Gaussian Covariance Tests

For Gaussian covariance restrictions, it is essential to evaluate both the score and information at the null-constrained estimates. The mean may be a nuisance parameter too, and centering the data correctly is part of the calculation.

Worked Example: Testing a Univariate Variance with Unknown Mean

Let \(X_1,\ldots,X_n\) be independent \(N(\mu,v)\) observations. Test \(H_0:v=1\), with \(\mu\) unrestricted. The observations are \(-1,0,1,2\), so \(n=4\). Calculate the score statistic.

The restricted estimate of the mean is the sample mean, \(\tilde\mu=\bar X=(-1+0+1+2)/4=1/2\). The residuals are \(-3/2,-1/2,1/2,3/2\), and hence their squared sum is \(R=(-3/2)^2+(-1/2)^2+(1/2)^2+(3/2)^2=9/4+1/4+1/4+9/4=5\).

For fixed \(\mu\), the log-likelihood, apart from no omitted parameter-dependent terms, is \(\ell(\mu,v)=-\frac n2\log(2\pi v)-R(\mu)/(2v)\), where \(R(\mu)=\sum_i(X_i-\mu)^2\). Its variance score is \(U_v=-n/(2v)+R(\mu)/(2v^2)\). At the restricted fit, this is \(U_v=-4/2+5/2=1/2\).

The Fisher information for \(v\) is \(I_{vv}=n/(2v^2)\), so at \(v=1\) it equals \(4/2=2\). The mean and variance scores have zero expected cross-information in this normal model; thus the nuisance adjustment does not change this information. Therefore \(T_S=(1/2)^2/2=1/8=0.125\).

Under the regular large-sample null approximation, the reference distribution is \(\chi^2_1\). The numerical statistic is small, but with only four observations the asymptotic calibration should not be mistaken for an exact finite-sample result.

Worked Example: Testing Zero Covariance with Unknown Marginal Variances

Suppose four independent observations come from a bivariate normal distribution with an unrestricted mean vector and covariance matrix \(\Sigma=\begin{pmatrix}a&c\\c&b\end{pmatrix}\). Test \(H_0:c=0\), with \(a\) and \(b\) unknown. The observed pairs are \((-1,-1),(0,0),(1,1),(0,-1)\).

The coordinate means are \(\bar X_1=0\) and \(\bar X_2=-1/4\). Define the centered sums \(S_{jk}=\sum_{i=1}^4(X_{ij}-\bar X_j)(X_{ik}-\bar X_k)\). Then \(S_{11}=(-1)^2+0^2+1^2+0^2=2\). For the second coordinate, the centered values are \(-3/4,1/4,5/4,-3/4\), giving \(S_{22}=(9+1+25+9)/16=44/16=11/4\). The cross-product is \(S_{12}=(-1)(-3/4)+0(1/4)+1(5/4)+0(-3/4)=2\).

Under the null \(c=0\), the restricted maximum-likelihood estimates are \(\tilde a=S_{11}/n=2/4=1/2\) and \(\tilde b=S_{22}/n=(11/4)/4=11/16\). At \(c=0\), differentiating the bivariate Gaussian log-likelihood with respect to \(c\) gives \(U_c=S_{12}/(ab)\). Thus \(U_c=2/[(1/2)(11/16)]=2/(11/32)=64/11\).

The expected Fisher information for \(c\) at \(c=0\) is \(I_{cc}=n/(ab)\). At the restricted fit, \(I_{cc}=4/(11/32)=128/11\). The information cross-terms between \(c\) and the two marginal variances are zero at \(c=0\), so this is also the efficient information for \(c\). The statistic is \(T_S=(64/11)^2/(128/11)=32/11\), approximately \(2.91\), with an asymptotic \(\chi^2_1\) reference distribution.

The centering and restricted estimates matter: using an uncentered second-coordinate sum of squares would give the wrong value of \(\tilde b\), and therefore the wrong score statistic.

Worked Example: A Gaussian Variance Restriction with Known Mean

Let \(X_1,\ldots,X_4\) be independent \(N(0,v)\) observations, with the mean known to be zero. The data are \(1,2,2,3\); test \(H_0:v=2\). Here there is no mean nuisance parameter to estimate.

The squared-residual sum about the known mean is \(R=1^2+2^2+2^2+3^2=18\). The variance score at \(v=2\) is \(U_v=-n/(2v)+R/(2v^2)=-4/4+18/8=-1+9/4=5/4\).

The Fisher information is \(I_{vv}=n/(2v^2)=4/(2\cdot4)=1/2\). Consequently \(T_S=(5/4)^2/(1/2)=25/16\cdot2=25/8=3.125\).

This example uses the same Gaussian variance score formula as the unknown-mean case, but the restricted residuals are measured from the known mean, not from a fitted sample mean. The statistic is again compared with \(\chi^2_1\) only as a large-sample approximation.

Practical Interpretation and Common Pitfalls

A score test measures the slope of the log-likelihood away from the null, in units determined by the null-model information. If the restricted score is near zero relative to its standard error, the likelihood has little first-order incentive to move away from the restriction. The unrestricted maximum-likelihood estimate is not needed to form this statistic; the restricted fit, score, and information are sufficient.

Using the unrestricted estimate in place of the restricted one. The score test is evaluated at the null fit. Using the unrestricted estimate changes the construction.
Ignoring nuisance estimation. In general, use efficient information, not merely \(I_{\psi\psi}\). The two coincide in special cases, such as the covariance examples above at the null.
Forgetting to center observations when the mean is estimated. The Gaussian covariance estimates and their score use centered sums and cross-products. The bivariate example shows how an incorrect mean changes the calculation.
Calling the chi-square approximation exact. The \(\chi^2_q\) reference is an asymptotic result under regularity conditions. It need not be accurate in a small sample.
Mixing up score, likelihood-ratio, and Wald calculations. The score test uses the null fit; the other procedures use different ingredients. Their large-sample agreement does not make their finite-sample statistics interchangeable.

Key takeaway: Fit the null model, evaluate the tested score there, and scale its quadratic size by efficient Fisher information. For a scalar restriction the calculation is \(U^2/I_{\mathrm{eff}}\); for \(q\) regular restrictions its null reference is asymptotically \(\chi^2_q\).

Check Your Understanding

Work through these questions by identifying the restricted fit before calculating a statistic.

In a model with tested parameter \(\psi\) and nuisance parameter \(\lambda\), what is the role of the restricted nuisance estimate in a score test?
When does the efficient information equal the \(\psi,\psi\) block of the full information matrix?
For the univariate variance example, verify the residual sum of squares from the four observations and explain why it is centered at \(1/2\).
In the bivariate example, calculate the second-coordinate mean and explain why the centered sum of squares is \(11/4\), not \(3\).
Why is a chi-square reference distribution for a score statistic generally an approximation rather than an exact finite-sample law?