Functional-Form Tests for Logit and Probit

Why Test Functional Form?

A logit or probit model can use the appropriate binary-response link and still describe the covariates too simply. For example, a linear index may miss curvature in a continuous predictor, or it may omit a relationship between two predictors. Functional-form tests ask whether a specified set of extra terms is needed, conditional on the other terms in the model. They do not test every possible way the response probability could vary.

The practical approach is to specify a restricted model and a larger, nested model that includes the terms under consideration. Then test whether the added coefficients are jointly zero. A Wald test uses estimates and their covariance matrix from the larger model; a likelihood-ratio test compares the fitted likelihoods of the two models.

Models and Functional-Form Restrictions

Let \(Y_i\) be a binary outcome for observation \(i\), and let \(x_i\) denote a vector of observed covariates. In a logit model, the conditional success probability is linked to an index by the logistic cumulative distribution function. In a probit model, it is linked by the standard normal cumulative distribution function.

P(Y_i=1\mid x_i)=F(\eta_i), \qquad \eta_i=x_i^\mathsf{T}\beta,

Here \(F(t)=1/(1+\exp(-t))\) for logit and \(F(t)=\Phi(t)\) for probit, where \(\Phi\) is the standard normal cumulative distribution function. The index is a chosen function of the covariates and parameters. A test of functional form investigates whether adding particular terms to that index is supported by the data; it is not a test of whether the link itself is correct.

Definition: A restricted model is nested in an unrestricted model when the restricted model results from imposing one or more restrictions on the unrestricted model’s parameters. For a functional-form test, the null hypothesis commonly sets the coefficients of specified added terms to zero. If there are \(q\) such coefficients, the joint null imposes \(q\) restrictions.

For example, suppose \(x_i\) is a continuous covariate. A linear-index model and a model with a quadratic term can be written as

\eta_i=\beta_0+\beta_1 x_i \quad\text{and}\quad \eta_i=\beta_0+\beta_1 x_i+\gamma_2 x_i^2.

The test \(H_0:\gamma_2=0\) asks whether the added quadratic term improves this specified index. A test with both \(x_i^2\) and \(x_i^3\) added has the joint null \(H_0:\gamma_2=\gamma_3=0\). For an interaction between a continuous covariate \(x_i\) and a binary covariate \(D_i\), the usual hierarchical specification includes both main effects as well as the product:

\eta_i=\beta_0+\beta_1 x_i+\beta_2 D_i+\gamma x_iD_i.

Testing \(H_0:\gamma=0\) assesses that particular interaction conditional on the included main effects. Centering or scaling a continuous covariate can make polynomial terms easier to interpret and can reduce numerical difficulties, but does not change the basic testing logic.

Wald and Likelihood-Ratio Tests

Suppose the unrestricted model has parameter vector \(\theta\), and the null hypothesis is \(H_0:R\theta=r\), where \(R\) has rank \(q\). Let \(\hat\theta\) be the unrestricted maximum-likelihood estimate and let \(\widehat{\operatorname{Var}}(\hat\theta)\) be its estimated covariance matrix. The Wald statistic measures how far the estimated restrictions are from their null values relative to their estimated uncertainty.

W=(R\hat\theta-r)^\mathsf{T} \left[R\widehat{\operatorname{Var}}(\hat\theta)R^\mathsf{T}\right]^{-1} (R\hat\theta-r).

Under the null and standard large-sample maximum-likelihood conditions, \(W\) is approximately distributed as \(\chi^2_q\). For a single coefficient restriction, this is the square of the corresponding coefficient’s Wald \(z\)-statistic. For several restrictions, the covariance among the estimates matters; separate coefficient tests are not a substitute for the joint test.

The likelihood-ratio test fits both nested models to the same observations using the same response and likelihood specification. Write \(\ell_R\) for the maximized log-likelihood under the restrictions and \(\ell_U\) for the maximized log-likelihood in the unrestricted model. Its statistic is

LR=2(\ell_U-\ell_R).

Under the null and standard regularity conditions, \(LR\) is approximately \(\chi^2_q\), where \(q\) is the number of independent restrictions. Since the unrestricted model can reproduce the restricted model by setting the added coefficients to their null values, its maximized likelihood cannot be lower: \(\ell_U\geq \ell_R\). Thus \(LR\geq 0\), apart from numerical optimization or reporting-rounding issues.

Conditions for the usual reference distributions: The observations are independent with a correctly specified Bernoulli likelihood under the null; the restricted model is nested in the unrestricted model; the restrictions are identifiable; and the maximum-likelihood estimates are in a regular interior region with adequate information. In practice, severe separation, weak identification, very small samples, or a highly sparse design can make the usual chi-square approximations unreliable.

A brief justification explains why both tests use the same asymptotic reference distribution. Under regularity conditions, the unrestricted estimate is approximately multivariate normal around the true parameter, with covariance estimated by the inverse information matrix. Applying the linear restrictions \(R\theta=r\) gives an approximately \(q\)-dimensional normal deviation; its squared, covariance-standardized length is \(W\), hence the \(\chi^2_q\) limit. For the likelihood-ratio test, a second-order expansion of the log-likelihood around its maximum makes twice the loss in log-likelihood from imposing the \(q\) restrictions equal, asymptotically, to that same standardized quadratic distance. This is the likelihood-ratio form of the standard large-sample result; the finite-sample values of the Wald and likelihood-ratio statistics need not coincide.

Worked Examples

Worked Example: Testing a Quadratic Term With a Wald Test

Suppose a hypothetical study records an independent binary outcome for each of \(n=500\) people. A logit model includes an intercept and a continuous predictor \(x\), centered and scaled so that one unit corresponds to one sample standard deviation. The larger model adds \(x^2\). Assume the estimated quadratic coefficient is \(0.080\), with standard error \(0.035\). Test \(H_0:\gamma_2=0\) against a two-sided alternative at \(\alpha=0.05\).

The model comparison concerns the added coefficient while retaining the intercept and linear term. Under the null, the restricted model is nested in the quadratic model. Assume the independent Bernoulli likelihood is appropriate, the coefficient is estimable, and the large-sample approximation is reasonable.

For one restriction, the Wald statistic is

W=\left(\frac{\hat\gamma_2}{SE(\hat\gamma_2)}\right)^2 =\left(\frac{0.080}{0.035}\right)^2 =(2.2857)^2 =5.224.

The reference distribution is \(\chi^2_1\), giving a two-sided p-value of approximately \(0.0223\). Since \(0.0223<0.05\), reject the zero-quadratic-coefficient null at the 5% level. In this fitted logit index, the data provide evidence that the quadratic term adds information beyond the specified linear term. The estimate is on the index scale per squared standardized unit of \(x\); it is not a direct change in probability, and this test alone does not establish that the entire model is correctly specified.

Worked Example: Joint Test of Quadratic and Cubic Terms

Consider a hypothetical probit analysis of \(n=600\) independent binary outcomes. The restricted index contains an intercept and a continuous predictor \(x\), centered and scaled. The unrestricted index also contains \(x^2\) and \(x^3\). The null is \(H_0:\gamma_2=\gamma_3=0\), a joint test of \(q=2\) restrictions. Suppose the maximized log-likelihoods, calculated on the same observations, are \(\ell_R=-180.400\) and \(\ell_U=-175.275\).

Assume the probit Bernoulli likelihood is suitable under the null, both models converge to their maxima, the added coefficients are identifiable, and regular large-sample conditions hold. The unrestricted likelihood is higher, as expected for nested models. The likelihood-ratio statistic is

LR=2(\ell_U-\ell_R) =2[-175.275-(-180.400)] =2(5.125) =10.25.

Under \(H_0\), compare this statistic with \(\chi^2_2\). For two degrees of freedom, the upper-tail probability is \(\exp(-10.25/2)=\exp(-5.125)\approx 0.005946\). Because this p-value is below \(0.05\), reject the joint null. The evidence supports adding at least one of the two specified powers to this probit index, conditional on its other terms. This comparison does not identify which power is responsible; individual coefficients or a separate one-restriction test would address narrower hypotheses.

Worked Example: Testing an Interaction With a Wald Test

In a hypothetical logit model, \(D\) indicates participation in a program (one for participants, zero otherwise), and \(x\) is a centered, standardized baseline score. The index includes an intercept, \(x\), \(D\), and \(xD\). Suppose the estimated interaction coefficient is \(-0.30\) with standard error \(0.12\). Test whether the interaction coefficient is zero, assuming independent Bernoulli observations and a regular, adequately sized sample.

The main effects remain in the model, so the comparison is between the specified model with the interaction and its nested version without \(xD\). For the one restriction \(H_0:\gamma=0\),

W=\left(\frac{-0.30}{0.12}\right)^2 =(-2.5)^2 =6.25.

The \(\chi^2_1\) upper-tail p-value is approximately \(0.0124\). At the 5% level, reject the null of no interaction in this index. The negative estimate means that the interaction contributes negatively to the linear predictor as \(xD\) increases, relative to the model containing only the two main effects. It does not imply a constant probability-scale interaction: the mapping from the index to probability is nonlinear.

Choosing and Reporting a Test

The Wald and likelihood-ratio tests answer the same specified null asymptotically, but use different fitted quantities. The Wald test is convenient when the unrestricted model and its covariance estimate are available. The likelihood-ratio test requires fitting both models and a comparable likelihood for each. When the estimates are near the boundary of the parameter space or the likelihood is poorly behaved, neither standard chi-square approximation should be used without considering whether its regularity conditions apply.

A functional-form test should be planned as a comparison of explicit models, rather than as an indiscriminate search through many powers and products. State which terms are being added, whether the test is individual or joint, and how many restrictions are involved. For a polynomial, retain the lower-order terms when adding a higher power; for an interaction, retain its component main effects. These choices make the tested comparison interpretable and preserve the intended model hierarchy.

Report the model family, added terms, null hypothesis, test statistic, degrees of freedom, p-value, and conclusion in context. For example: “Adding the quadratic term to the logit index gave a Wald statistic of 5.224 on one degree of freedom (\(p\approx0.0223\)); the zero-coefficient null was rejected at the 5% level.” Avoid saying that a small p-value proves the larger model is true. It indicates incompatibility between the data and the specified restrictions under the model and approximation assumptions.

Common Pitfalls

Testing a coefficient and calling it an omnibus test. A test of one square term does not test all nonlinearities. Specify the terms and restrictions actually tested.
Testing interaction without its main effects. Omitting the component main effects changes the meaning of the interaction comparison. Include those effects unless the model’s scientific structure justifies a different specification.
Using unrelated samples for the LR comparison. The restricted and unrestricted log-likelihoods must be based on the same observations and likelihood setup for the nested comparison to have its standard interpretation.
Interpreting an index coefficient as a probability change. A logit or probit index coefficient is not itself a probability effect. The probability-scale effect depends on the covariate values and the link.
Treating non-rejection as proof of linearity. A test may have limited power, and it only assesses the specified added terms. Failure to reject does not establish that no other functional-form problem exists.

Key takeaway: A functional-form test compares a specified restricted binary-response model with a nested model that adds chosen covariate terms. Wald and likelihood-ratio statistics assess the same zero-restriction null asymptotically, but the conclusion is limited to those terms, model assumptions, and data.

Check Your Understanding

Use the model-comparison logic and interpretations developed above.

Write the joint null for adding \(x^2\) and \(x^3\) to a binary-response index. How many degrees of freedom does the usual test use?
In a Wald test of one added coefficient, the estimate is \(0.18\) and its standard error is \(0.06\). Calculate the Wald statistic.
Why must the restricted and unrestricted models in a likelihood-ratio test use the same observations?
In a model with \(x\), \(D\), and \(xD\), what does the null test of the interaction coefficient ask, and why should the main effects ordinarily remain included?
What does failure to reject a test of a quadratic term establish—and what does it not establish?