Delta-Method Inference for Binary-Model Effects

Why Effects Need Their Own Inference

In a logit or probit model, coefficients describe a linear index, but many questions concern quantities on the probability scale: the marginal effect of a continuous regressor, the probability change associated with a binary regressor, or an elasticity. These effects are nonlinear functions of the estimated coefficients. Their uncertainty therefore cannot usually be read directly from a coefficient’s standard error.

The delta method provides a general way to approximate the sampling variance of a smooth function of an estimator. Its basic inputs are an estimated coefficient vector, an estimated covariance matrix for that vector, and the gradient of the effect with respect to the coefficients. The result is an approximate standard error and, under the usual large-sample normal approximation, a confidence interval.

The Delta Method for an Estimated Effect

Let $\hat{\beta}$ be an estimator of the coefficient vector $\beta$, and let $\widehat V$ estimate the covariance matrix of $\hat{\beta}$. Suppose the effect of interest is $g(\beta)$, a differentiable scalar function. The first-order delta method estimates its variance using the gradient of $g$, evaluated at $\hat{\beta}$.

Definition: The delta-method variance estimate for $g(\hat{\beta})$ is $$ \widehat{\operatorname{Var}}\{g(\hat{\beta})\} = \nabla g(\hat{\beta})^\mathsf{T}\widehat V\nabla g(\hat{\beta}), $$ where $\nabla g(\hat{\beta})$ is the vector of partial derivatives of $g$ with respect to the coefficients. The delta-method standard error is the square root of this variance estimate.

For a large-sample two-sided confidence interval with confidence level $1-\alpha$, the usual normal approximation is

g(\hat{\beta}) \ \pm\ z_{1-\alpha/2} \sqrt{\nabla g(\hat{\beta})^\mathsf{T}\widehat V\nabla g(\hat{\beta})},

where $z_{1-\alpha/2}$ is the corresponding standard-normal quantile. For a 95% interval, $z_{0.975}$ is approximately 1.96. This is an approximate repeated-sampling interval for the effect defined by $g$, not a probability statement about a fixed parameter being inside the particular interval.

Gradients for Binary-Model Effects

Write the conditional probability in a binary-response model as $P(Y=1\mid x)=F(\eta)$, where $\eta=x^\mathsf{T}\beta$. In logit, $F=\Lambda$, the logistic cumulative distribution function; in probit, $F=\Phi$, the standard normal cumulative distribution function. Write $f=F'$, so $f$ is the logistic density in logit and the standard normal density $\phi$ in probit.

For a continuous regressor $x_j$, holding the other regressors fixed, the pointwise marginal effect is $\beta_j f(\eta)$. Its gradient has a contribution from the direct appearance of $\beta_j$, and another from the fact that changing any coefficient changes the index:

\frac{\partial\{\beta_j f(\eta)\}}{\partial\beta_k} = \mathbf{1}(j=k)f(\eta)+\beta_j f'(\eta)x_k.

Here $\mathbf{1}(j=k)$ equals 1 when $j=k$ and 0 otherwise. For logit, $f'(\eta)=f(\eta)\{1-F(\eta)-F(\eta)\}$; for probit, $f'(\eta)=-\eta\phi(\eta)$. Substituting the estimated coefficients and covariate values gives the gradient used in the variance formula.

A binary regressor is different: its effect is naturally a discrete probability change rather than a derivative. Let $x_1$ and $x_0$ denote the covariate vectors with the binary regressor set to 1 and 0, respectively, leaving other covariates unchanged. The change is $F(x_1^\mathsf{T}\beta)-F(x_0^\mathsf{T}\beta)$. Its gradient is

f(x_1^\mathsf{T}\beta)x_1-f(x_0^\mathsf{T}\beta)x_0.

An elasticity for a positive continuous regressor $x_j$ is the proportional change in the probability associated with a proportional change in $x_j$, evaluated at the specified covariates. It is $x_j\beta_j f(\eta)/F(\eta)$. For logit this simplifies to $\beta_j x_j\{1-F(\eta)\}$; for probit it is $\beta_j x_j\phi(\eta)/\Phi(\eta)$. Each is a function of the coefficient vector, so its gradient and the delta method apply in the same way.

Worked Inference Examples

Worked Example: A Pointwise Logit Marginal Effect

Suppose a hypothetical logit model has $P(Y=1\mid x)=\Lambda(\beta_0+\beta_1x)$. At $x=1.5$, the fitted coefficients are $\hat{\beta}_0=-0.4$ and $\hat{\beta}_1=0.8$. Suppose their estimated covariance matrix, in coefficient order $(\hat{\beta}_0,\hat{\beta}_1)$, is

\widehat V= \begin{pmatrix} 0.0400&0.0060\\ 0.0060&0.0100 \end{pmatrix}.

The index is $\hat{\eta}=-0.4+0.8(1.5)=0.8$, giving $\hat{p}=\Lambda(0.8)\approx0.6900$ and $\hat{p}(1-\hat{p})\approx0.2139$. The estimated marginal effect per one-unit increase in $x$ is

\hat E=\hat{\beta}_1\hat p(1-\hat p) =0.8(0.2139)\approx0.1711.

For this two-coefficient model, the gradient components are $\partial E/\partial\beta_0=\beta_1p(1-p)(1-2p)$ and $\partial E/\partial\beta_1=p(1-p)+x\beta_1p(1-p)(1-2p)$. At the estimates these are approximately $-0.06502$ and $0.1164$. Thus

\begin{aligned} \widehat{\operatorname{Var}}(\hat E) &=(-0.06502)^2(0.0400)+(0.1164)^2(0.0100)\\ &\quad+2(-0.06502)(0.1164)(0.0060)\\ &\approx0.0001691+0.0001355-0.0000908\\ &\approx0.0002137,\qquad SE(\hat E)\approx0.01462. \end{aligned}

A 95% delta-method interval is $0.1711\pm1.96(0.01462)$, or approximately $[0.1425,0.1998]$. In this hypothetical model, at $x=1.5$, the estimated probability increases by about 0.171 per one-unit increase in $x$, locally; the interval represents approximate sampling uncertainty in that marginal effect. The calculation assumes the fitted model and the stated covariance estimate are suitable for the sampling design.

Worked Example: A Probit Discrete Change for a Binary Regressor

Consider the hypothetical probit model $P(Y=1\mid d,z)=\Phi(\beta_0+\beta_1d+\beta_2z)$, where $d$ is binary. At $z=1$, suppose $(\hat{\beta}_0,\hat{\beta}_1,\hat{\beta}_2)=(-0.5,0.9,0.4)$, with estimated covariance matrix diagonal entries $0.0400, 0.0225, 0.0100$ and all off-diagonal entries zero. The probability change from $d=0$ to $d=1$ is

\hat\Delta=\Phi(-0.5+0.9+0.4)-\Phi(-0.5+0.4) =\Phi(0.8)-\Phi(-0.1) \approx0.7881-0.4602=0.3280.

The gradient, in coefficient order $(\beta_0,\beta_1,\beta_2)$, is $\phi(0.8)(1,1,1)-\phi(-0.1)(1,0,1)$. Since $\phi(0.8)\approx0.2897$ and $\phi(-0.1)\approx0.3970$, this is approximately $(-0.1073,0.2897,-0.1073)$. With a diagonal covariance matrix, the variance estimate is

\begin{aligned} \widehat{\operatorname{Var}}(\hat\Delta) &=(-0.1073)^2(0.0400)+(0.2897)^2(0.0225) +(-0.1073)^2(0.0100)\\ &\approx0.0004602+0.001888+0.0001151 =0.002463,\\ SE(\hat\Delta)&\approx\sqrt{0.002463}=0.04963. \end{aligned}

The 95% interval is $0.3280\pm1.96(0.04963)$, approximately $[0.2307,0.4253]$. It describes uncertainty in the modeled probability difference between $d=1$ and $d=0$ at $z=1$. It is not automatically a causal effect: that interpretation requires an appropriate causal design and assumptions beyond the probit specification.

Worked Example: A Logit Probability Elasticity

In a hypothetical logit model with $\eta=\beta_0+\beta_1x$, suppose $\hat{\beta}_0=-0.2$, $\hat{\beta}_1=0.6$, and the effect is evaluated at $x=2$. Use the covariance matrix from the first example. Here $\hat{\eta}=1$, $\hat p=\Lambda(1)\approx0.7311$, and the elasticity is

\hat\epsilon=\hat{\beta}_1x(1-\hat p) =0.6(2)(1-0.7311)\approx0.3227.

For logit, $\epsilon=\beta_1x(1-p)$. Differentiating with respect to $\beta_0$ and $\beta_1$ gives, respectively, $-\beta_1x p(1-p)$ and $x(1-p)-\beta_1x^2p(1-p)$. At the fitted values the gradient is approximately $(-0.2359,0.06601)$. Applying the covariance matrix yields

\begin{aligned} \widehat{\operatorname{Var}}(\hat\epsilon) &=(-0.2359)^2(0.0400)+(0.06601)^2(0.0100)\\ &\quad+2(-0.2359)(0.06601)(0.0060)\\ &\approx0.002226+0.00004358-0.0001869 \approx0.002083,\\ SE(\hat\epsilon)&\approx0.04564. \end{aligned}

The 95% interval is $0.3227\pm1.96(0.04564)$, approximately $[0.2333,0.4122]$. The point estimate says that, at $x=2$, a 1% increase in $x$ corresponds approximately to a 0.323% increase in the modeled probability, locally. The interval quantifies approximate sampling uncertainty in this elasticity, not uncertainty about an individual outcome.

Average Effects and the Delta-Method Justification

The target must be specified before computing an interval. A pointwise marginal effect is evaluated at one covariate vector. An average marginal effect (AME) averages individual marginal effects over the observations in a stated sample. If the sample covariates $x_i$ are treated as fixed for this calculation, the AME is a function of the coefficients, $\frac{1}{n}\sum_{i=1}^n \beta_j f(x_i^\mathsf{T}\beta)$. Its gradient is the average of the individual gradients. Insert that averaged gradient into the same variance formula. For a binary regressor, average the individual discrete changes and likewise average their gradients. An AME is not generally the same quantity as an effect evaluated at the sample means.

The first-order approximation follows from a Taylor expansion around the true coefficient vector. If $\hat{\beta}$ is close to $\beta$, then

g(\hat{\beta})\approx g(\beta)+\nabla g(\beta)^\mathsf{T}(\hat{\beta}-\beta).

The constant $g(\beta)$ has no sampling variance. The linear term has approximate variance $\nabla g(\beta)^\mathsf{T}\operatorname{Var}(\hat{\beta})\nabla g(\beta)$; replacing unknown quantities by their estimates gives the delta-method formula. In regular maximum-likelihood settings, coefficient estimates are approximately multivariate normal in large samples, which motivates the normal-based interval. Inference also requires a covariance estimate appropriate to the data and sampling structure. A robust or clustered covariance estimator may be needed when the corresponding dependence or variance assumptions call for it.

Common Pitfalls and Practical Checks

Using a coefficient standard error as an effect standard error. A marginal effect, change, or elasticity is a function of multiple coefficients, often with covariance terms. Differentiate the complete effect and use the full covariance matrix.
Dropping covariance terms. Unless the estimated coefficients are uncorrelated, the off-diagonal entries of $\widehat V$ contribute to the variance. The examples show that these terms can increase or decrease it.
Confusing a derivative with a binary-regressor change. A derivative describes a local change for a continuous regressor. For a binary regressor, compare the fitted probabilities at its two values.
Changing the estimand without noticing. A pointwise effect, an effect at the means, and an AME are different quantities. State which one the interval concerns and the covariate values or averaging sample used.
Treating a normal interval as exact. The delta method is a first-order, large-sample approximation. In small samples, or when the effect function is strongly curved over plausible coefficient values, its normal interval may be inaccurate.

For a calculation check, confirm that the gradient is evaluated at the same coefficient estimates and covariate values as the reported effect, that the covariance matrix uses the same coefficient order, and that the resulting variance is nonnegative apart from negligible rounding error. Interpret a probability change in probability points, an effect per unit in the regressor’s units, and an elasticity in percentage terms.

Check Your Understanding

Use the definitions and calculations above to answer the following questions.

Why does a nonlinear marginal effect require more than the standard error of $\hat{\beta}_j$?
For a binary regressor, what two model probabilities define the discrete change?
How would you obtain a delta-method variance estimate for an AME over a fixed sample?
What role do off-diagonal entries of the coefficient covariance matrix play?
What additional assumptions would be needed to interpret a probit discrete change causally?