Joint Tests for Several Logistic Coefficients

Why Test Several Logistic Coefficients Together?

A logistic regression analysis may include several predictors, and a question may concern their combined contribution rather than any one coefficient alone. For example, a researcher might ask whether two measurements jointly provide evidence about the probability of an outcome after adjusting for other predictors. A joint test addresses that question directly: it tests whether a specified set of coefficients satisfies a set of restrictions.

This is not the same as looking at separate tests for each coefficient and counting how many are significant. A joint test uses the estimated covariance among coefficients and has its own test statistic and degrees of freedom. That distinction can change the conclusion.

Definition: In a logistic regression model, let $\boldsymbol{\beta}$ denote the vector of regression coefficients. A joint hypothesis has the form $H_0:R\boldsymbol{\beta}=\mathbf{r}$, where $R$ specifies the restrictions and $\mathbf{r}$ gives their hypothesized values. If $R$ has rank $q$, the null hypothesis imposes $q$ independent restrictions. The alternative is that at least one restriction does not hold.

For instance, testing whether two slopes are both zero uses $q=2$ restrictions. The intercept and any other coefficients not included in those restrictions remain unrestricted. The test therefore asks whether the two slopes add evidence beyond the rest of the specified model, not whether the entire model has no relationship with the outcome.

The Logistic Model and the Two Test Statistics

Suppose $Y_i$ is a binary outcome for observation $i$, with $Y_i=1$ indicating that the event occurs. For independent observations, a logistic regression model specifies

\log\left(\frac{p_i}{1-p_i}\right)=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}, \qquad p_i=P(Y_i=1\mid \mathbf{x}_i).

Here $\mathbf{x}_i$ is the predictor vector, including an intercept if one is fitted. The Bernoulli log-likelihood, evaluated at the model coefficients, is

\ell(\boldsymbol{\beta}) = \sum_{i=1}^{n} \left[ y_i\log(p_i)+(1-y_i)\log(1-p_i) \right].

The tests below use maximum likelihood estimates. Their usual reference distributions are large-sample approximations: the observations should be independent, the logistic model should be identifiable and suitably specified, and the maximum likelihood estimates should be finite with regularity conditions satisfied. If these conditions fail, such as when the likelihood has no finite maximum, the usual chi-square approximation may be unreliable.

Likelihood-ratio test: Fit the full model and the restricted model under $H_0$, using the same observations. If their maximized log-likelihoods are $\ell_{\text{full}}$ and $\ell_{\text{restricted}}$, respectively, then $$ G^2=2\left(\ell_{\text{full}}-\ell_{\text{restricted}}\right). $$ For $q$ independent restrictions, $G^2$ is approximately distributed as $\chi^2_q$ under $H_0$.

The models must be nested: the restricted model must be obtainable from the full model by imposing the null restrictions. Since the full model is allowed at least as much flexibility, its maximized log-likelihood cannot be smaller. Thus $G^2$ should be nonnegative, apart from tiny numerical discrepancies.

Wald test: Let $\widehat{\boldsymbol{\beta}}$ be the full-model maximum likelihood estimate and let $\widehat{\operatorname{Var}}(\widehat{\boldsymbol{\beta}})$ be its estimated covariance matrix. For $H_0:R\boldsymbol{\beta}=\mathbf{r}$, the Wald statistic is $$ W= (R\widehat{\boldsymbol{\beta}}-\mathbf{r})^\mathsf{T} \left[ R\widehat{\operatorname{Var}}(\widehat{\boldsymbol{\beta}})R^\mathsf{T} \right]^{-1} (R\widehat{\boldsymbol{\beta}}-\mathbf{r}). $$ Under $H_0$, $W$ is approximately distributed as $\chi^2_q$.

The covariance matrix matters: coefficients estimated from the same data are often related. In the common case of testing $q$ selected coefficients equal to zero, $R$ selects those coefficients and $\mathbf{r}=\mathbf{0}$. The Wald statistic then measures their joint distance from zero relative to their estimated uncertainty, accounting for their covariance.

Why the Reference Distribution Is Chi-Square

The likelihood-ratio result follows from the local quadratic shape of the log-likelihood near its maximum. For a sufficiently large sample and a regular model, the log-likelihood is approximately quadratic around the full-model estimate. Maximizing under the null restricts the estimate to satisfy $q$ independent conditions. The loss in maximized log-likelihood is therefore approximately a quadratic form in $q$ standardized directions. Twice that loss has an approximate $\chi^2_q$ distribution.

The Wald result has a closely related justification. Under the regularity conditions, the maximum likelihood estimate is approximately multivariate normal:

\widehat{\boldsymbol{\beta}} \ \approx\ N\!\left( \boldsymbol{\beta}, \operatorname{Var}(\widehat{\boldsymbol{\beta}}) \right).

Applying the linear transformation $R$ gives an approximately normal estimate of $R\boldsymbol{\beta}$, with covariance $R\operatorname{Var}(\widehat{\boldsymbol{\beta}})R^\mathsf{T}$. Under the null, subtracting $\mathbf{r}$ gives a vector with mean zero. Its squared length after standardizing by this covariance matrix is the Wald quadratic form, which is approximately $\chi^2_q$. This also explains why the inverse in the formula requires the tested restrictions to be independent and their estimated covariance matrix to be nonsingular.

Worked Examples

Worked Example: Likelihood-Ratio Test of Two Slopes

A hypothetical study models whether a policyholder renews a small equipment-protection contract. The full logistic model includes an intercept and two continuous predictors, $x_1$ and $x_2$. The null hypothesis is $H_0:\beta_1=\beta_2=0$; the intercept remains unrestricted. The full model has maximized log-likelihood $-83.2$, and the restricted model has maximized log-likelihood $-89.5$. Test at the 5% level.

Method and restrictions. Use a likelihood-ratio test because the restricted model is obtained from the full model by setting two slopes to zero. There are $q=2$ independent restrictions.

Calculate the statistic.

G^2=2\left[(-83.2)-(-89.5)\right] =2(6.3)=12.6.

Find the p-value and decide. Under the null, compare $12.6$ with a $\chi^2_2$ distribution. For two degrees of freedom, the upper-tail probability is $\exp(-G^2/2)$, so

P(\chi^2_2\ge 12.6)=\exp(-12.6/2)=\exp(-6.3)\approx 0.0018.

The p-value is below $0.05$, so reject $H_0$. The data provide evidence that at least one of the two slopes differs from zero, conditional on the model and its other terms. This result does not establish that both slopes are nonzero.

Check and common mistake. The full log-likelihood is larger, as it must be for nested models; the statistic is positive. A common error is to use only the difference $6.3$, omitting the factor of 2. The likelihood-ratio statistic is $12.6$, not $6.3$.

Worked Example: Wald Test Using the Covariance Matrix

In a hypothetical logistic model for whether a customer accepts an optional device-repair plan, two fitted slopes have estimates $\widehat{\beta}_1=0.40$ and $\widehat{\beta}_2=0.20$. Their estimated covariance matrix is

\widehat{\operatorname{Var}} \begin{pmatrix} \widehat{\beta}_1\\ \widehat{\beta}_2 \end{pmatrix} = \begin{pmatrix} 0.04&0.01\\ 0.01&0.01 \end{pmatrix}.

Test $H_0:\beta_1=\beta_2=0$ using a Wald test at the 5% level.

Method and restrictions. There are $q=2$ restrictions. The estimated coefficient vector is $(0.40,0.20)^\mathsf{T}$, and the null vector is zero. First invert the covariance matrix. Its determinant is

(0.04)(0.01)-(0.01)(0.01)=0.0003, \qquad \widehat{\operatorname{Var}}^{-1} = \frac{1}{0.0003} \begin{pmatrix} 0.01&-0.01\\ -0.01&0.04 \end{pmatrix} = \begin{pmatrix} 33.3333&-33.3333\\ -33.3333&133.3333 \end{pmatrix}.

Calculate the quadratic form. Multiplying the inverse matrix by the estimate vector gives approximately $(6.6667,13.3333)^\mathsf{T}$. Therefore,

$$ W= (0.40)(6.6667)+(0.20)(13.3333) =2.6667+2.6667 =5.3333. $$

Find the p-value and decide. With $q=2$, the approximate p-value is

P(\chi^2_2\ge 5.3333) =\exp(-5.3333/2) \approx 0.0695.

Since $0.0695>0.05$, do not reject the joint null hypothesis at the 5% level. This is not proof that both slopes are zero; it means the joint test does not provide sufficient evidence against that null at the chosen level.

Check and common mistake. The covariance matrix is positive definite because its determinant is positive and its diagonal entries are positive. Ignoring the off-diagonal covariance and adding two squared coefficient-to-standard-error ratios would give a different statistic. The Wald test requires the full covariance matrix.

Worked Example: Joint and Separate Tests Can Disagree

A hypothetical logistic model for whether a farm sensor sends an alert has two estimated slopes, $\widehat{\beta}_1=0.30$ and $\widehat{\beta}_2=-0.30$. Each has estimated variance $0.04$, and their estimated covariance is $0.03$. Compare the separate two-sided Wald tests with the joint Wald test of $H_0:\beta_1=\beta_2=0$, using a 5% level.

Separate tests. Each standard error is $\sqrt{0.04}=0.20$. Thus the separate test statistics are

z_1=\frac{0.30}{0.20}=1.50, \qquad z_2=\frac{-0.30}{0.20}=-1.50.

Each two-sided normal p-value is approximately $0.1336$, so neither separate test rejects at the 5% level.

Joint test. For the two estimates, the covariance matrix is

V= \begin{pmatrix} 0.04&0.03\\ 0.03&0.04 \end{pmatrix}, \qquad V^{-1} = \frac{1}{0.04(0.04)-0.03(0.03)} \begin{pmatrix} 0.04&-0.03\\ -0.03&0.04 \end{pmatrix} = \begin{pmatrix} 57.1429&-42.8571\\ -42.8571&57.1429 \end{pmatrix}.

Multiplying $V^{-1}$ by $(0.30,-0.30)^\mathsf{T}$ gives approximately $(30,-30)^\mathsf{T}$. The joint statistic is

$$ W=(0.30)(30)+(-0.30)(-30)=18. $$

There are two restrictions, so the p-value is $\exp(-18/2)=\exp(-9)\approx 0.0001$. The joint test rejects at the 5% level, even though neither separate test does. The estimates move in opposite directions, and their positive covariance makes that particular joint pattern comparatively unlikely under the joint null.

Check and common mistake. The covariance matrix has determinant $0.0007>0$, so it is invertible. Do not conclude that the joint test cannot reject because neither individual test rejects: the joint statistic depends on the covariance as well as the two individual standard errors.

Choosing and Interpreting a Joint Test

The likelihood-ratio and Wald tests target the same null hypothesis but use different information from the fitted model. The likelihood-ratio test compares maximized likelihoods from two nested fits. The Wald test evaluates how far the full-model estimate is from the null restrictions, standardized by its estimated covariance. In large samples under regularity conditions, both have the same approximate $\chi^2_q$ reference distribution. In finite samples, their results need not agree exactly.

Feature	Likelihood-ratio test	Wald test
Calculation	Compares full and restricted maximized log-likelihoods	Uses the full-model coefficient estimate and covariance matrix
Models needed	Both nested models must be fitted	Uses the unrestricted fitted model
Degrees of freedom	Number of independent null restrictions	Number of independent null restrictions
Common practical issue	Comparisons are invalid if the models are not nested or use different data	Results can be sensitive to an estimate being far from the null or to an unstable covariance estimate

For either test, the p-value is the upper-tail probability of the observed statistic under $\chi^2_q$. Reject at significance level $\alpha$ when the p-value is less than $\alpha$, or equivalently when the statistic exceeds the corresponding upper-tail critical value. State the restrictions and degrees of freedom explicitly; a statistic without its reference distribution is not enough to determine a conclusion.

A frequent trap is to test each coefficient separately and treat those results as a substitute for the requested joint test. Separate tests answer separate one-restriction questions. They do not generally account for the covariance among estimates in the way the joint test does. Another trap is to use the number of coefficients mentioned rather than the number of independent restrictions: the degrees of freedom are $q=\operatorname{rank}(R)$, not automatically the number of terms in a written expression.

Key takeaway: Formulate the null as $R\boldsymbol{\beta}=\mathbf{r}$, count its independent restrictions, and use a joint statistic with the matching $\chi^2_q$ reference distribution. A joint conclusion is not obtained by simply collecting the outcomes of separate coefficient tests.

Check Your Understanding

For each problem, identify the appropriate restriction count and use the stated asymptotic reference distribution.

A full logistic model and its restricted version have log-likelihoods $-51.6$ and $-56.1$, respectively. The restricted model fixes three slopes at zero and retains the intercept. Calculate the likelihood-ratio statistic and its approximate p-value. (Answer: $G^2=9.0$, $q=3$, p-value $\approx 0.0293$; reject at 5%.)
Two slopes have estimates $0.25$ and $-0.15$, with covariance matrix $\begin{pmatrix}0.01&0\\0&0.01\end{pmatrix}$. For the joint null that both are zero, calculate the Wald statistic and p-value. (Answer: $W=8.5$, $q=2$, p-value $\approx 0.0143$; reject at 5%.)
Two estimates are $0.30$ and $-0.30$, each with standard error $0.20$, and their covariance is $0.03$. What are the separate Wald $z$-statistics, and does the joint Wald test of both coefficients equal to zero reject at 5%? (Answer: separate $z$-statistics are $1.50$ and $-1.50$, neither separately significant; the joint statistic is $18$ with $q=2$, p-value $\approx 0.0001$, so it rejects.)
A fitted model has $\widehat{\beta}_1=0.20$, $\widehat{\beta}_2=0.10$, variances $0.04$ and $0.01$, and covariance $0.01$. Use a Wald test of the single restriction $H_0:\beta_1+2\beta_2=0$. (Answer: the estimated contrast is $0.40$, its estimated variance is $0.04+4(0.01)+4(0.01)=0.12$, so $W=0.40^2/0.12=1.3333$, $q=1$, p-value $\approx 0.2482$; do not reject at 5%.)

What You'll Learn