Probit Regression And Normal-Quantile Effects

From Response Probability to Normal-Quantile Scale

In a binary-response model, the probability of an outcome is between zero and one, while a linear predictor can take any real value. A link function connects these scales. In probit regression, that link is the inverse cumulative distribution function of the standard normal distribution. Consequently, each regression coefficient describes a change in the normal quantile corresponding to the response probability.

This interpretation is useful because it gives probit coefficients a direct, additive meaning on a common scale, even though their effects on probability are not generally additive. A coefficient is not, by itself, a fixed percentage-point change in risk. The probability change depends on the starting probability, or equivalently on the covariate profile.

Definition: Let $Y_i$ be a binary response, with $P(Y_i=1\mid \mathbf{x}_i)=p_i$, and let $\mathbf{x}_i$ be a vector of covariates. A probit regression model specifies $$ \Phi^{-1}(p_i)=\eta_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}, \qquad p_i=\Phi(\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}), $$ where $\Phi$ is the cumulative distribution function of $N(0,1)$, $\Phi^{-1}$ is its inverse on probabilities strictly between zero and one, and $\boldsymbol{\beta}$ is the coefficient vector. Equivalently, $Y_i\mid\mathbf{x}_i$ has a Bernoulli distribution with success probability $p_i$.

The standard normal density is denoted by $\phi$, so $\Phi'(z)=\phi(z)$. The standard normal quantile $\Phi^{-1}(p)$ is the value $z$ for which $\Phi(z)=p$. For example, a probability of $0.5$ corresponds to quantile zero, since $\Phi(0)=0.5$. Probit regression assumes that this quantile is a linear function of the covariates.

The phrase normal-quantile scale refers to the scale of $\Phi^{-1}(p)$. A one-unit change on this scale is one unit of the standard normal argument. This interpretation does not require a separate unobserved response variable: here it is a statement about the transformation of the modeled probability.

Coefficient Interpretation

Suppose the model contains an intercept and two predictors: $$ \Phi^{-1}(p)=\beta_0+\beta_1x_1+\beta_2x_2. $$ The intercept is the normal quantile of the modeled probability when both predictors equal zero. Holding $x_2$ fixed, increasing $x_1$ by one unit changes the normal quantile by $\beta_1$. More generally, increasing $x_1$ by $c$ units changes it by $c\beta_1$, provided the model is linear in $x_1$ and contains no interaction that changes its coefficient across values of another predictor.

Theorem: In a probit model with linear predictor $\eta=\mathbf{x}^{\mathsf{T}}\boldsymbol{\beta}$, changing the covariate vector from $\mathbf{x}$ to $\mathbf{x}+\boldsymbol{\Delta}$ changes the normal quantile of the response probability by $\boldsymbol{\Delta}^{\mathsf{T}}\boldsymbol{\beta}$. In particular, a one-unit increase in covariate $x_j$, with all other covariates fixed, changes that quantile by $\beta_j$.

Proof. At $\mathbf{x}$, the modeled probability is $p=\Phi(\mathbf{x}^{\mathsf{T}}\boldsymbol{\beta})$, so $\Phi^{-1}(p)=\mathbf{x}^{\mathsf{T}}\boldsymbol{\beta}$. At $\mathbf{x}+\boldsymbol{\Delta}$, the probability is $p'=\Phi((\mathbf{x}+\boldsymbol{\Delta})^{\mathsf{T}}\boldsymbol{\beta})$, so $$ \Phi^{-1}(p')-\Phi^{-1}(p) =(\mathbf{x}+\boldsymbol{\Delta})^{\mathsf{T}}\boldsymbol{\beta} -\mathbf{x}^{\mathsf{T}}\boldsymbol{\beta} =\boldsymbol{\Delta}^{\mathsf{T}}\boldsymbol{\beta}. $$ Taking $\boldsymbol{\Delta}$ to have a one in coordinate $j$ and zeros elsewhere gives the particular coefficient interpretation. $\square$

For a categorical predictor represented by an indicator, its coefficient is the adjusted normal-quantile difference between the category coded one and the reference category coded zero, at the same values of the other included covariates. If a model includes interactions, a coefficient for one predictor alone is not necessarily the change at every value of the interacting predictor: the relevant change is the sum of the applicable terms in the linear predictor.

Worked Example: A Continuous Predictor

Suppose a fitted model is $\Phi^{-1}(p)= -1+0.6x$, where $x$ is a continuous predictor. Compare the fitted probabilities at $x=0$ and $x=1$, and interpret the slope.

At $x=0$, the linear predictor is $-1+0.6(0)=-1$. Thus the fitted probability is $p_0=\Phi(-1)\approx0.158655$.

At $x=1$, the linear predictor is $-1+0.6(1)=-0.4$. Thus $p_1=\Phi(-0.4)\approx0.344578$. The probability difference is $p_1-p_0\approx0.344578-0.158655=0.185923$, or about 0.1859.

The slope $0.6$ means that a one-unit increase in $x$ raises the normal quantile of the response probability by $0.6$. It does not mean that the probability rises by $0.6$, or by the same fixed amount for every starting value of $x$.

Probability Effects Depend on the Starting Point

The model maps the linear predictor $\eta$ to probability using $p=\Phi(\eta)$. Since $\Phi$ is curved, equal changes in $\eta$ can correspond to different changes in probability. For a continuous predictor $x_j$ with coefficient $\beta_j$, the local probability effect is obtained by differentiating the fitted probability with respect to that predictor.

Theorem: In a probit model with linear predictor $\eta=\mathbf{x}^{\mathsf{T}}\boldsymbol{\beta}$, the partial derivative of the fitted probability with respect to a continuous covariate $x_j$ is $$ \frac{\partial p}{\partial x_j}=\beta_j\phi(\eta) =\beta_j\phi(\mathbf{x}^{\mathsf{T}}\boldsymbol{\beta}). $$ This derivative is a local probability change per unit of $x_j$, evaluated at the specified covariate values.

Proof. Write $p=\Phi(\eta)$, where $\eta=\sum_k x_k\beta_k$. Differentiating the linear predictor with respect to $x_j$ gives $\partial\eta/\partial x_j=\beta_j$. By the chain rule and $\Phi'(\eta)=\phi(\eta)$, $$ \frac{\partial p}{\partial x_j} =\frac{d\Phi(\eta)}{d\eta}\frac{\partial\eta}{\partial x_j} =\phi(\eta)\beta_j. $$ Substituting $\eta=\mathbf{x}^{\mathsf{T}}\boldsymbol{\beta}$ gives the stated expression. $\square$

Because the standard normal density is largest at zero and decreases as the absolute value of its argument increases, the magnitude of this local effect is largest when $\eta=0$, corresponding to $p=0.5$, and smaller toward probabilities near zero or one. If $\beta_j$ is positive, the fitted probability increases with $x_j$; if it is negative, the probability decreases. The size of the local change still depends on the other covariates through $\eta$.

Worked Example: Local Probability Effects at Two Profiles

Suppose $\Phi^{-1}(p)=-0.5+0.8x$, with $x$ continuous. Calculate the local probability effect of $x$ at $x=0$ and $x=2$.

The derivative is $dp/dx=0.8\phi(-0.5+0.8x)$. At $x=0$, the linear predictor is $-0.5$, so $$ \left.\frac{dp}{dx}\right|_{x=0} =0.8\phi(-0.5) \approx0.8(0.352065) \approx0.281652. $$ Thus, near $x=0$, a one-unit increase in $x$ corresponds locally to an increase of about 0.2817 in probability.

At $x=2$, the linear predictor is $-0.5+0.8(2)=1.1$, giving $$ \left.\frac{dp}{dx}\right|_{x=2} =0.8\phi(1.1) \approx0.8(0.217852) \approx0.174282. $$ The coefficient remains $0.8$ normal-quantile units per unit of $x$ at both profiles, but the local probability effect is smaller at $x=2$. A derivative is a local rate, not the exact probability difference for a finite one-unit change.

Adjusted Contrasts and Finite Probability Differences

For a finite change in a predictor, the probability difference should be calculated by evaluating the model at both covariate profiles. The quantile-scale contrast is linear and straightforward; transforming the endpoints through $\Phi$ produces a probability contrast that depends on those endpoints. This distinction is especially important for binary predictors, where the natural comparison is between two fitted probabilities rather than a derivative.

Worked Example: Treatment Contrasts at Two Ages

Consider the model $$ \Phi^{-1}(p)=-1+0.01(\text{age})+0.70T, $$ where age is measured in years and $T=1$ for treatment and $T=0$ for control. Calculate the treatment contrast at ages 50 and 75.

At age 50, the control linear predictor is $-1+0.01(50)+0.70(0)=-0.50$, while the treatment linear predictor is $-1+0.01(50)+0.70(1)=0.20$. The fitted probabilities are $\Phi(-0.50)\approx0.3085375$ and $\Phi(0.20)\approx0.5792597$. Their difference, computed from the unrounded values, is approximately $0.5792597-0.3085375=0.2707222$, or 0.2707 to four decimal places.

At age 75, the control linear predictor is $-1+0.01(75)=-0.25$, and the treatment linear predictor is $0.45$. The fitted probabilities are $\Phi(-0.25)\approx0.4012937$ and $\Phi(0.45)\approx0.6736448$. The difference is approximately $0.6736448-0.4012937=0.2723511$, or 0.2724 to four decimal places.

The treatment coefficient is the adjusted treatment contrast of $0.70$ normal-quantile units at either age because the model contains no treatment-by-age interaction. The probability differences are not identical: they are approximately 0.2707 and 0.2724. The small difference between them comes from applying the nonlinear function $\Phi$ at different starting points.

Common Mistakes and Practical Interpretation

A probit coefficient should first be interpreted on the scale on which the model is linear. When a report also needs a probability-scale statement, provide fitted probabilities or a probability contrast at specified covariate values. For continuous predictors, a derivative can summarize the local probability effect, but it too must be tied to a covariate profile.

Wrong: “A coefficient of $0.7$ means the probability rises by 70 percentage points.” Correction: It means the normal quantile rises by $0.7$ units for the stated predictor change. Calculate the probability change from the fitted probabilities.
Wrong: “A positive coefficient gives the same probability increase for everyone.” Correction: It gives a positive change on the quantile scale; the probability change depends on the starting linear predictor.
Wrong: “The derivative is the exact change for a whole-unit increase.” Correction: The derivative is local. For a finite change, evaluate $\Phi(\eta)$ at both endpoints and subtract.
Wrong: “The treatment coefficient is an unadjusted probability difference.” Correction: In a multiple-covariate probit model, it is an adjusted contrast on the normal-quantile scale, conditional on the other included covariates.

Key takeaway: Probit regression makes the normal quantile of the response probability linear in the covariates. A coefficient is a change on that normal-quantile scale; probability effects require the inverse-link transformation and depend on the covariate profile.

Check Your Understanding

Use the distinction between quantile-scale coefficients and probability-scale changes to answer these questions.

In a probit model, what quantity is linear in the covariates: the probability or the inverse standard normal CDF of the probability?
A continuous predictor has coefficient $-0.4$. State its one-unit interpretation on the normal-quantile scale, holding other covariates fixed.
Why does a probit coefficient not generally equal a fixed probability difference?
Write the formula for the local probability effect of a continuous predictor with coefficient $\beta_j$.
For a binary predictor, what two fitted quantities should be compared to report a probability difference at a specified covariate profile?