Score Tests from Restricted Estimates

Why Test at the Restricted Estimate?

A likelihood-ratio test compares the likelihood at restricted and unrestricted maximum-likelihood estimates. A score test takes a different computational route: it asks whether the likelihood has a substantial slope away from the estimate obtained under the null hypothesis. Its defining feature is that the score and information are evaluated at the restricted estimate. The unrestricted model need not be fitted.

This can be useful when the null model is straightforward to estimate but the unrestricted model is large or computationally difficult. It does not remove the need to specify a valid likelihood or to account for nuisance parameters. In particular, the raw score for a tested parameter must be scaled using information that adjusts for nuisance parameters.

Definition: Let $\ell(\theta)$ be a log-likelihood, with parameter vector $\theta$, score $U(\theta)=\partial\ell(\theta)/\partial\theta$, and Fisher information $I(\theta)$. Under a null hypothesis $H_0$, the restricted maximum-likelihood estimate $\tilde{\theta}$ maximizes $\ell(\theta)$ subject to $H_0$. A score test measures the score at $\tilde{\theta}$, scaled by the information for departures from the null. Under regularity conditions and $H_0$, the resulting statistic has an asymptotic chi-square distribution.

Partitioned Parameters and Nuisance Adjustment

Write the parameter as $\theta=(\eta',\lambda')'$, where $\lambda$ contains the $q$ parameters being tested and $\eta$ contains nuisance parameters. Consider $H_0:\lambda=\lambda_0$. The restricted estimate $\tilde{\eta}$ maximizes the likelihood with $\lambda$ fixed at $\lambda_0$, and $\tilde{\theta}=(\tilde{\eta}',\lambda_0')'$. If the nuisance parameters are interior and the restricted likelihood is differentiable, the first-order condition gives $U_\eta(\tilde{\theta})=0$. The tested-parameter score $U_\lambda(\tilde{\theta})$ need not be zero.

Partition the information matrix into blocks corresponding to $\eta$ and $\lambda$. The information relevant to testing $\lambda$, after accounting for estimation of $\eta$, is the Schur complement below. This is sometimes called the efficient information for $\lambda$.

I_{\lambda\cdot\eta} = I_{\lambda\lambda} - I_{\lambda\eta}I_{\eta\eta}^{-1}I_{\eta\lambda}.

Theorem: For testing $H_0:\lambda=\lambda_0$, the score or Lagrange-multiplier statistic is $$ LM = U_\lambda(\tilde{\theta})' I_{\lambda\cdot\eta}(\tilde{\theta})^{-1} U_\lambda(\tilde{\theta}). $$ Under $H_0$, standard likelihood regularity conditions, and nonsingular information, $LM$ converges in distribution to $\chi^2_q$, where $q$ is the number of restrictions. The asymptotic p-value is $P(\chi^2_q\ge LM)$.

The adjustment matters because changes in $\lambda$ can be partially offset by changes in $\eta$. The raw block $I_{\lambda\lambda}$ treats $\eta$ as if it were fixed. The Schur complement instead measures the information left for $\lambda$ when nuisance parameters can adjust. In a one-parameter test, $LM=U_\lambda(\tilde{\theta})^2/I_{\lambda\cdot\eta}(\tilde{\theta})$.

Why the Statistic Has This Form

A local quadratic expansion of the log-likelihood explains both the nuisance adjustment and the statistic. Use $a$ for a small displacement in $\eta$ and $b$ for a displacement in $\lambda$ from the restricted estimate. In the expansion, write $I$ for the information evaluated at $\tilde{\theta}$. Since $U_\eta(\tilde{\theta})=0$, the log-likelihood change is approximately

\ell(\tilde{\theta}+(a',b')')-\ell(\tilde{\theta}) \approx U_\lambda'b -\frac{1}{2} \left( a'I_{\eta\eta}a +2a'I_{\eta\lambda}b +b'I_{\lambda\lambda}b \right).

For a fixed $b$, maximize this quadratic expression over $a$. Differentiating with respect to $a$ and setting the derivative to zero gives

-I_{\eta\eta}a-I_{\eta\lambda}b=0, \qquad a^*=-I_{\eta\eta}^{-1}I_{\eta\lambda}b.

Substituting $a^*$ into the quadratic expression leaves $U_\lambda'b-\frac{1}{2}b'I_{\lambda\cdot\eta}b$. Its maximum occurs at $b^*=I_{\lambda\cdot\eta}^{-1}U_\lambda$, and the maximized increase is $\frac{1}{2}U_\lambda'I_{\lambda\cdot\eta}^{-1}U_\lambda$. Twice this increase is the score statistic. This argument is a local approximation: it justifies the asymptotic test, not an exact finite-sample chi-square result.

For a full parameter vector and $q$ independent linear restrictions $R\theta=r$, the corresponding form is

LM = U(\tilde{\theta})'I(\tilde{\theta})^{-1}R' \left\{RI(\tilde{\theta})^{-1}R'\right\}^{-1} RI(\tilde{\theta})^{-1}U(\tilde{\theta}).

Here $\tilde{\theta}$ maximizes the likelihood subject to $R\theta=r$, and $R$ has row rank $q$. The formula projects the score onto directions that violate the restrictions, using the information to scale those directions. For a partitioned test, the nuisance first-order conditions reduce the expression to the efficient-score form above.

Worked Calculations

Worked Example: Testing Equality of Two Poisson Rates

Suppose two independent groups have Poisson event counts $Y_0$ and $Y_1$, with log means $\alpha$ and $\alpha+\beta$, respectively. Test $H_0:\beta=0$, so the groups have the same rate. The observed counts are $y_0=18$ and $y_1=30$. The model assumes independent Poisson counts, positive means, and an interior parameter value. The common-rate null is an interior restriction, and the restricted estimate below is positive.

Under $H_0$, both group means equal $\exp(\alpha)$. Maximizing the restricted likelihood gives $\exp(\tilde{\alpha})=(18+30)/2=24$. At this restricted estimate the score for $\beta$ is the observed count in group 1 minus its fitted mean, so $U_\beta=30-24=6$.

The expected information matrix for $(\alpha,\beta)$, evaluated under the restricted model, is

I(\tilde{\theta}) = \begin{pmatrix} 48 & 24\\ 24 & 24 \end{pmatrix}.

The nuisance parameter is $\alpha$, so the efficient information for $\beta$ is $I_{\beta\cdot\alpha}=24-(24)(48^{-1})(24)=12$. Therefore,

LM=\frac{6^2}{12}=3.000.

Against a $\chi^2_1$ reference distribution, the asymptotic p-value is approximately $0.0833$. At the 5% significance level, this test does not reject equal rates. The result is evidence about the compatibility of the observed difference with the null model; it does not establish that the rates are equal. The unrestricted model was not fitted.

Worked Example: Testing a Normal Mean with Unknown Variance

Let $Y_1,\ldots,Y_n$ be independent observations from $N(\mu,\sigma^2)$, with both $\mu$ and $\sigma^2$ unknown. Test $H_0:\mu=10$ using the hypothetical observations $8,10,11,13$. The normal model has an interior mean and positive variance; the variance is a nuisance parameter.

Under the null, the restricted maximum-likelihood estimate of the variance is the average squared deviation from 10: $\tilde{\sigma}^2=[(-2)^2+0^2+1^2+3^2]/4=14/4=3.5$. The score for $\mu$, evaluated at the restricted estimate, is

U_\mu(\tilde{\theta}) = \frac{\sum_i(Y_i-10)}{\tilde{\sigma}^2} = \frac{-2+0+1+3}{3.5} = \frac{2}{3.5} \approx 0.5714.

For this normal model, the expected information for $\mu$ is $n/\sigma^2$, and its expected-information cross term with $\sigma^2$ is zero. Thus the efficient information for $\mu$ is $4/3.5\approx1.1429$. The score statistic is

LM = \frac{(2/3.5)^2}{4/3.5} = \frac{4}{14} \approx 0.2857.

The asymptotic $\chi^2_1$ p-value is approximately $0.593$, so the test does not reject $H_0$ at the 5% level. This is an asymptotic calibration; with only four observations, the chi-square approximation may be inaccurate. The computation uses the restricted variance estimate, not a variance estimate from an unrestricted fit.

Worked Example: Testing a Logistic Group Effect

In two independent groups, suppose the number of successes is binomial with group size 20, with success probability $\operatorname{logit}^{-1}(\alpha+\beta x)$, where $x=0$ for group 0 and $x=1$ for group 1. The hypothetical success counts are 8 and 14. Test $H_0:\beta=0$. The model assumes independent Bernoulli outcomes with a correctly specified logistic mean; the null probability is interior, and the information is nonsingular.

Under the null, the common probability estimate is the pooled proportion $\tilde p=(8+14)/40=0.55$. The score for $\beta$ is successes in group 1 minus its restricted expected count: $U_\beta=14-20(0.55)=3$. Let $w=\tilde p(1-\tilde p)=0.55(0.45)=0.2475$. The information matrix for $(\alpha,\beta)$ is

I(\tilde{\theta}) = w \begin{pmatrix} 40 & 20\\ 20 & 20 \end{pmatrix}.

Adjusting for the nuisance intercept gives $I_{\beta\cdot\alpha}=20w-(20w)^2/(40w)=10w=2.475$. Hence

LM=\frac{3^2}{2.475}\approx3.636.

The asymptotic p-value from $\chi^2_1$ is approximately $0.0565$. At the 5% level, the test does not reject no group effect. The restricted fit requires only the pooled success proportion; no separate unrestricted probabilities need to be estimated.

Using and Interpreting a Score Test

A score test is most useful when the restricted likelihood is easier to fit than the unrestricted likelihood. Its calculation still requires the score and information of the specified full model, evaluated at the restricted estimate. A practical sequence is:

State the null and model.
Identify the restrictions, nuisance parameters, and likelihood assumptions. Check that the null is an interior restriction and that the relevant information is nonsingular.

Fit only the restricted model.
Find $\tilde{\theta}$, the maximum-likelihood estimate subject to $H_0$. Verify that the nuisance score is zero when the regular interior conditions apply.

Evaluate score and information.
Compute the score for the tested directions and use the efficient information, or the equivalent linear-restriction formula, at $\tilde{\theta}$.

Calibrate and interpret.
Compare the statistic with $\chi^2_q$ asymptotically. Report the restrictions, reference degrees of freedom, p-value, and conclusion in context.

The null chi-square approximation relies on the likelihood regularity conditions, including differentiability, identification, an interior true parameter, and adequate information. It can be unreliable in small samples, near parameter boundaries, or when information is nearly singular. A score statistic is not automatically a robust test: if the likelihood variance assumptions are wrong, the information used for scaling may also be wrong.

Common Mistake

A frequent error is to divide the tested score by $I_{\lambda\lambda}$ and ignore nuisance parameters. That treats the nuisance parameters as known. For example, in the Poisson calculation above this would use $36/24=1.5$, rather than the correct $36/12=3$. The correction is to use $I_{\lambda\cdot\eta}$, or the equivalent full restriction formula, because nuisance parameters are estimated under the null. Another error is to say that a nonsignificant score test proves the null; it only indicates that the observed score is not sufficiently large to reject under the chosen calibration.

Key takeaway: The score test evaluates the full model's slope at the restricted maximum-likelihood estimate and scales it by information for departures from the null, accounting for nuisance parameters. Its asymptotic chi-square statistic can be computed without fitting the unrestricted model.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

For a test of $q$ restrictions, what is the asymptotic reference distribution of the score statistic under the null?
Why is $I_{\lambda\lambda}$ generally not the correct information to use when nuisance parameters are estimated?
In the Poisson example, identify the restricted estimate, tested score, and efficient information.
What does a p-value of $0.0565$ imply at a 5% significance level, and what does it not imply?
Name one condition under which the asymptotic chi-square calibration may be unreliable.