Likelihood-Ratio Tests of Logistic Regression Restrictions

Testing Restrictions in a Logistic Model

In logistic regression, a coefficient describes a change in log odds, not a change in the mean response measured in the units of a continuous outcome. That distinction matters when testing hypotheses: the likelihood is Bernoulli, the variance depends on the success probability, and the Wald, score, and likelihood-ratio statistics are constructed from this model’s likelihood and information.

A coefficient test such as \(H_0:\beta_1=0\) asks whether a predictor has a zero conditional log-odds effect. A joint test such as \(H_0:\beta_1=\beta_2=0\) asks whether a set of effects can be removed together. Under standard regularity conditions, the three tests have chi-square reference distributions asymptotically, but they use different information: the unrestricted estimate for Wald, the restricted fit for score, and the difference in maximized log-likelihoods for likelihood ratio.

Definition: For independent binary responses \(Y_i\in\{0,1\}\), a logistic regression model specifies \(P(Y_i=1\mid x_i)=p_i\), with \(\operatorname{logit}(p_i)=\log\{p_i/(1-p_i)\}=x_i^\mathsf{T}\beta\). For a linear restriction \(H_0:R\beta=r\), where \(R\) has rank \(q\), the Wald test evaluates the restriction at the unrestricted maximum likelihood estimate; the score test evaluates the likelihood’s slope and information at the restricted maximum; and the likelihood-ratio test compares the restricted and unrestricted maximum log-likelihoods.

Likelihood, Score, and Information

Assume the observations are conditionally independent Bernoulli variables and the design vectors \(x_i\) are fixed. The log-likelihood, score vector, and observed negative Hessian are

\ell(\beta)=\sum_{i=1}^n\{y_i x_i^\mathsf{T}\beta-\log(1+\exp(x_i^\mathsf{T}\beta))\},\qquad U(\beta)=\sum_{i=1}^n x_i(y_i-p_i),\qquad J(\beta)=\sum_{i=1}^n p_i(1-p_i)x_ix_i^\mathsf{T}.

These expressions follow by writing the Bernoulli probability as \(p_i^{y_i}(1-p_i)^{1-y_i}\), taking logarithms, and differentiating. In particular, \(\partial p_i/\partial\beta=p_i(1-p_i)x_i\), so differentiating the score gives \(-J(\beta)\). The expected Fisher information equals \(J(\beta)\), since the latter depends on the covariates and model probabilities, not on the observed \(y_i\).

Write \(\hat\beta\) for the unrestricted maximum likelihood estimate and \(\tilde\beta\) for the maximum under \(R\beta=r\). For a scalar restriction, the Wald statistic is the squared estimated departure divided by its estimated variance. For a vector restriction, the corresponding quadratic form is

W=(R\hat\beta-r)^\mathsf{T} \{R\widehat{\operatorname{Var}}(\hat\beta)R^\mathsf{T}\}^{-1} (R\hat\beta-r).

The likelihood-ratio statistic and the score statistic can be written

\Lambda=2\{\ell(\hat\beta)-\ell(\tilde\beta)\},\qquad S=U(\tilde\beta)^\mathsf{T}J_{\mathrm{eff}}(\tilde\beta)^{-1}U(\tilde\beta).

Here \(J_{\mathrm{eff}}\) is the information for the tested coordinates after accounting for nuisance parameters. More precisely, if the parameters are partitioned into tested and nuisance coordinates and the information matrix has blocks \(J_{11},J_{12},J_{21},J_{22}\), then \(J_{\mathrm{eff}}=J_{11}-J_{12}J_{22}^{-1}J_{21}\). The score in the statistic is the correspondingly adjusted score. This adjustment is essential: nuisance parameters are estimated, not held fixed.

Regularity conditions: The usual chi-square calibration requires a correctly specified, identifiable model; a true parameter in the interior of the parameter space; a fixed number of parameters; nonsingular information; and regular maximum likelihood behavior. In logistic regression, complete or quasi-complete separation can prevent a finite maximum likelihood estimate, so these standard conditions must not be assumed automatically.

Worked Example: A Single Exposure Coefficient

Worked Example: Comparing Two Exposure Groups

Suppose 40 unexposed individuals include 8 outcomes and 32 non-outcomes, while 40 exposed individuals include 16 outcomes and 24 non-outcomes. Fit an intercept and a binary exposure indicator \(x\), and test \(H_0:\beta=0\).

The two fitted probabilities are \(8/40=0.2\) and \(16/40=0.4\). Their logits are \(\log(1/4)=-\log4\) and \(\log(2/3)\). Thus \(\hat\beta=\log\{(2/3)/(1/4)\}=\log(8/3)\), the log odds ratio. The estimated odds ratio is \(8/3\).

For independent binomial groups, the estimated variance of a fitted group logit is \(1/s+1/f\), where \(s\) and \(f\) are the success and failure counts. The intercept is the unexposed logit; the exposure coefficient is the difference of the two group logits. Therefore \(\widehat{\operatorname{Var}}(\hat\beta)=1/16+1/24+1/8+1/32=25/96\). The Wald statistic is

W=\frac{\{\log(8/3)\}^2}{25/96}\approx 3.818.

For the score test under \(H_0\), the pooled success proportion is \(24/80=0.3\). The exposure-group score, after accounting for the intercept, is \(16-40(0.3)=4\). The efficient information for the exposure contrast is \(40(40)/80\) times \(0.3(0.7)\), or \(4.2\). Hence

S=\frac{4^2}{4.2}\approx 3.810.

For the likelihood ratio, compare the two separately fitted probabilities, \(0.2\) and \(0.4\), with the restricted common probability \(0.3\). The log-likelihood difference is \(8\log(0.2/0.3)+32\log(0.8/0.7)+16\log(0.4/0.3)+24\log(0.6/0.7)\). The four terms are approximately \(-3.2437,\ 4.2730,\ 4.6029,\ -3.6996\), respectively. Thus

\Lambda=2(1.9326)\approx 3.865.

The tests are close here, but their numerical values differ. The coefficient concerns a conditional odds ratio; it is not a difference in outcome probabilities. A chi-square reference distribution with one degree of freedom is an approximation justified by regular large-sample theory, not an exact Bernoulli calculation.

Worked Example: A Joint Restriction

Worked Example: Testing Two Group Effects Together

Consider three groups of 40 observations each, with success counts 10, 20, and 30. Use group 1 as the reference and fit an intercept with indicators for groups 2 and 3. Test the joint restriction \(H_0:\beta_2=\beta_3=0\).

Under the unrestricted model, the fitted probabilities are \(0.25,\ 0.5,\ 0.75\). The restricted model has one common probability, \(60/120=0.5\). For the score test, the score coordinates for the group-2 and group-3 indicators at this restricted fit are \(20-20=0\) and \(30-20=10\). Thus the score vector is \((0,10)^\mathsf{T}\), not a vector based on the reference group’s residual alone.

At the restricted fit each observation has weight \(p(1-p)=0.25\). The intercept information is 30; each indicator has information 10 and has cross-information 10 with the intercept. Removing the intercept nuisance parameter gives \(J_{\mathrm{eff}}=\begin{pmatrix}20/3&-10/3\\-10/3&20/3\end{pmatrix}\). Its inverse is \(\begin{pmatrix}1/5&1/10\\1/10&1/5\end{pmatrix}\), so

S=(0,10) \begin{pmatrix}1/5&1/10\\1/10&1/5\end{pmatrix} \begin{pmatrix}0\\10\end{pmatrix}=20.

For the likelihood ratio, the middle group contributes zero log-likelihood difference because its fitted probability is already \(0.5\). The first group contributes \(10\log(0.25/0.5)+30\log(0.75/0.5)=10\log(0.5)+30\log(1.5)\approx5.2325\). The third group has the same contribution by symmetry. Therefore

\Lambda=2(5.2324814+5.2324814)\approx20.930.

This calculation shows how a joint test uses departures in both groups while allowing for the shared intercept. It also illustrates why one must compute the score under the restricted fit and adjust its information for nuisance parameters.

Worked Example: A Joint Wald Calculation

Worked Example: Using a Fitted Covariance Matrix

Suppose a logistic model has unrestricted estimates \((\hat\beta_2,\hat\beta_3)^\mathsf{T}=(\log3,2\log3)^\mathsf{T}\), and the estimated covariance matrix for these coefficients is \(V=\begin{pmatrix}7/30&2/15\\2/15&7/30\end{pmatrix}\). Test the joint null that both coefficients are zero.

The determinant of \(V\) is \((7/30)^2-(2/15)^2=11/900\). Direct inversion gives \(V^{-1}=\begin{pmatrix}70/11&-40/11\\-40/11&70/11\end{pmatrix}\). The Wald statistic is therefore

W=(\log3,2\log3)V^{-1}(\log3,2\log3)^\mathsf{T} =\frac{190}{11}(\log3)^2\approx20.8473.

The degrees of freedom are two because the null imposes two independent restrictions. The covariance between the estimates matters: replacing \(V\) with its diagonal would discard information and generally change the quadratic form. In a real analysis, use the covariance matrix from the stated fitted logistic model and verify that it corresponds to the coefficient parameterization being tested.

How to Interpret the Tests

For a logistic coefficient, \(\beta_j\) is a log odds ratio for a one-unit increase in predictor \(x_j\), holding the other predictors fixed. Exponentiating gives the odds ratio \(\exp(\beta_j)\). Thus \(H_0:\beta_j=0\) is equivalent to an odds ratio of one. A joint restriction tests several such log-odds effects at once; rejection says the specified restrictions are collectively inconsistent with the data under the model, not that every individual coefficient must be significant.

The three statistics answer the same regular large-sample testing question by different routes. Wald uses the unrestricted estimate and its covariance; score asks how steep the likelihood is at the restricted fit; likelihood ratio compares the best fit allowed by the null with the best unrestricted fit. For nested logistic models, the likelihood-ratio calculation is especially direct: maximize each model’s Bernoulli likelihood and take twice the log-likelihood difference.

Common Pitfall

Wrong: “A logistic coefficient is a change in probability, so testing whether it is zero is like testing a slope in normal linear regression.”
Correction: The coefficient is a change in log odds. The probability change depends on the starting probability, since \(p=\exp(x^\mathsf{T}\beta)/\{1+\exp(x^\mathsf{T}\beta)\}\). Use the Bernoulli likelihood and its information for the tests.
Wrong: “For a joint score test, use the raw tested-coordinate score and ignore the intercept.”
Correction: The score and information must account for nuisance parameters through the efficient score and Schur complement. Otherwise the statistic does not represent the tested restriction with nuisance parameters estimated.

Key takeaway: In logistic regression, coefficient restrictions are hypotheses about conditional log odds. Construct Wald, score, or likelihood-ratio statistics from the fitted Bernoulli model, account for nuisance parameters, and interpret a rejection in terms of the restrictions tested.

Check Your Understanding

Check whether you can connect the restriction, the model fit, and the statistic.

In logistic regression, what does \(H_0:\beta_j=0\) imply for the conditional odds ratio?
Why does the score test evaluate the score at the restricted maximum rather than the unrestricted maximum?
For a joint restriction, why must the covariance among coefficient estimates be included in the Wald statistic?
In the three-group example, why is the score for the group-2 indicator zero at the common-probability fit?
State one condition under which the usual chi-square calibration may fail for a logistic-regression test.