Delta Method for GLM Parameter Functions

Why Transforming Coefficients Requires Propagating Uncertainty

GLM coefficients are often not the quantities of direct scientific interest. A coefficient contrast may describe a change in log rate or log odds, while a report calls for a rate ratio or odds ratio. A linear contrast may also be the target in its own right. In each case, uncertainty in the fitted coefficient vector must be carried through the function that defines the target.

The multivariate delta method provides this calculation when the coefficient estimator is asymptotically normal and the target is a sufficiently smooth function of the coefficients. Its central ingredient is the gradient: locally, the gradient describes how sensitive the target is to small changes in each coefficient.

Setup: Let $\beta\in\mathbb{R}^p$ be the GLM coefficient vector, and suppose, under the model and regularity conditions for asymptotic normality, that $$ \sqrt{n}(\hat{\beta}-\beta_0)\xrightarrow{d}N(0,\Sigma), $$ where $\Sigma$ is a finite, positive-semidefinite covariance matrix. Assume also that $\hat{\beta}\xrightarrow{p}\beta_0$. For a scalar function $g:\mathbb{R}^p\to\mathbb{R}$, write $\nabla g(\beta_0)$ for the column vector of its first partial derivatives.

The Multivariate Delta Method

The result applies when $g$ is differentiable at the true coefficient vector. This condition ensures that, for small changes in $\beta$, the first-order Taylor term captures the change in $g(\beta)$, while the remaining error is negligible at the asymptotic scale.

Theorem (multivariate delta method): Under the setup above, if $g$ is differentiable at $\beta_0$, then $$ \sqrt{n}\{g(\hat{\beta})-g(\beta_0)\} \xrightarrow{d} N\!\left(0,\nabla g(\beta_0)^\mathsf{T}\Sigma\nabla g(\beta_0)\right). $$ The asymptotic variance of $g(\hat{\beta})$ is therefore approximately $\nabla g(\beta_0)^\mathsf{T}\Sigma\nabla g(\beta_0)/n$.

The variance is a scalar quadratic form. It accounts for both the uncertainty of each coefficient and the covariances between coefficients. Replacing the full covariance matrix with only its diagonal entries is generally incorrect.

Specify the target.
Write the scientific quantity as a function $g(\beta)$, rather than treating a transformed coefficient as though it were still a coefficient.

Differentiate.
Find the gradient $\nabla g(\beta)$, keeping its dependence on $\beta$ explicit.

Propagate covariance.
Evaluate the gradient at the estimate and use the coefficient covariance matrix in the quadratic form.

Proof. Differentiability at $\beta_0$ gives the expansion $$ g(\hat{\beta})-g(\beta_0) = \nabla g(\beta_0)^\mathsf{T}(\hat{\beta}-\beta_0) +r(\hat{\beta}-\beta_0), $$ where $r(h)/\lVert h\rVert\to 0$ as $h\to 0$. By the assumed asymptotic normality, $\sqrt{n}(\hat{\beta}-\beta_0)=O_p(1)$, and hence $\hat{\beta}-\beta_0=O_p(n^{-1/2})$. Consistency and the defining property of the remainder imply $\sqrt{n}\,r(\hat{\beta}-\beta_0)=o_p(1)$. Therefore, $$ \sqrt{n}\{g(\hat{\beta})-g(\beta_0)\} = \nabla g(\beta_0)^\mathsf{T}\sqrt{n}(\hat{\beta}-\beta_0)+o_p(1). $$ The first term converges in distribution to a univariate normal random variable, since it is a fixed linear transformation of the limiting multivariate normal vector. Its variance is $\nabla g(\beta_0)^\mathsf{T}\Sigma\nabla g(\beta_0)$. Slutsky’s theorem gives the stated limit. $\square$

In practice, $\beta_0$ and $\Sigma$ are unknown. If $\hat{\Sigma}\xrightarrow{p}\Sigma$ estimates the asymptotic covariance of $\sqrt{n}(\hat{\beta}-\beta_0)$, and $g$ has a continuous gradient near $\beta_0$, then the estimated variance is $$ \widehat{\operatorname{Var}}\{g(\hat{\beta})\} = \frac{1}{n}\nabla g(\hat{\beta})^\mathsf{T}\hat{\Sigma}\nabla g(\hat{\beta}). $$ This plug-in estimate is consistent for the delta-method asymptotic variance. If the available covariance estimate is already on the scale of $\hat{\beta}$, rather than $\sqrt{n}\hat{\beta}$, use that covariance directly in the quadratic form; do not divide by $n$ a second time. When dispersion is estimated jointly with the coefficients, use the coefficient covariance block from the joint covariance estimate.

Linear Contrasts

A linear contrast has the form $g(\beta)=c^\mathsf{T}\beta$, where $c$ is a fixed $p$-vector. Its gradient is simply $c$, so the delta method reduces to the familiar variance formula for a linear combination: $$ \operatorname{Avar}(c^\mathsf{T}\hat{\beta}) = \frac{c^\mathsf{T}\Sigma c}{n}. $$ Unlike a nonlinear function, a linear contrast has no Taylor approximation error: the expression is exactly linear in the estimator. The asymptotic normality statement follows directly by transforming the multivariate normal limit.

Worked Example: A Linear Contrast With Correlated Coefficients

Suppose a model has coefficients $\beta=(\beta_0,\beta_1,\beta_2)^\mathsf{T}$, with estimate $\hat{\beta}=(0.20,0.40,-0.03)^\mathsf{T}$. The asymptotic covariance matrix for $\sqrt{n}(\hat{\beta}-\beta)$ is $$ \Sigma= \begin{pmatrix} 0.16&0.02&0\\ 0.02&0.09&-0.01\\ 0&-0.01&0.04 \end{pmatrix}, $$ and $n=400$. Find the estimate and standard error of $\beta_1-2\beta_2$.

Set $c=(0,1,-2)^\mathsf{T}$. The estimated contrast is $$ c^\mathsf{T}\hat{\beta} = 0(0.20)+1(0.40)-2(-0.03) = 0.46. $$ Its estimated asymptotic variance is $c^\mathsf{T}\Sigma c/n$. Expanding the quadratic form shows the contribution of the covariance: $$ c^\mathsf{T}\Sigma c = 0.09+(-2)^2(0.04)+2(1)(-2)(-0.01) = 0.09+0.16+0.04 = 0.29. $$ Thus the variance is $0.29/400=0.000725$, and the standard error is $\sqrt{0.000725}\approx 0.0269$. The covariance term contributes $0.04$; dropping it would give a different answer.

Exponentiating Contrasts: Rate Ratios and Odds Ratios

For a GLM with a log link, differences in linear predictors correspond to log rate ratios when the comparison uses the same exposure or offset convention. If $x_1$ and $x_0$ are two covariate vectors, the rate ratio is $g(\beta)=\exp\{(x_1-x_0)^\mathsf{T}\beta\}$. For logistic regression, the same expression gives the odds ratio between the two covariate profiles. In either case, put $c=x_1-x_0$: the target is $g(\beta)=\exp(c^\mathsf{T}\beta)$.

Differentiating gives $\nabla g(\beta)=\exp(c^\mathsf{T}\beta)c$. The delta-method variance is therefore $$ \operatorname{Avar}\{\exp(c^\mathsf{T}\hat{\beta})\} = \frac{\exp\{2c^\mathsf{T}\beta_0\}\,c^\mathsf{T}\Sigma c}{n}. $$ The corresponding plug-in standard error is $\exp(c^\mathsf{T}\hat{\beta})\sqrt{c^\mathsf{T}\hat{\Sigma}c/n}$. The factor from differentiating the exponential is essential.

Worked Example: A Log-Link Rate Ratio

In a Poisson regression with a log link, suppose the treatment coefficient is estimated as $\hat{\beta}_1=0.35$. The asymptotic variance of $\sqrt{n}\hat{\beta}_1$ is $0.04$, and $n=250$. For a comparison that changes treatment status but holds the other covariates and exposure convention fixed, calculate the estimated rate ratio and its delta-method standard error.

The log rate ratio is $\beta_1$, so the rate ratio function is $g(\beta_1)=\exp(\beta_1)$. Its estimate is $\exp(0.35)\approx 1.4191$. The variance of $\hat{\beta}_1$ is $0.04/250=0.00016$, and the derivative of $g$ at the estimate is $\exp(0.35)$. Hence $$ \widehat{\operatorname{Var}}\{\exp(\hat{\beta}_1)\} = \exp(0.70)(0.00016) \approx 0.0003222. $$ The estimated standard error is $\sqrt{0.0003222}\approx 0.0179$. The rate ratio estimate is therefore about $1.419$, with a delta-method standard error of about $0.018$. This standard error is on the rate-ratio scale, not the log-rate-ratio scale.

Worked Example: A Logistic-Regression Odds Ratio

Suppose a logistic regression has an exposure coefficient estimated as $0.80$. The asymptotic variance of $\sqrt{n}\hat{\beta}_1$ is $0.16$, and $n=500$. For a comparison that changes exposure status while holding other covariates fixed, find the estimated odds ratio and its delta-method standard error.

The estimated odds ratio is $\exp(0.80)\approx 2.2255$. The estimated variance of the coefficient is $0.16/500=0.00032$. Since the derivative of $\exp(\beta_1)$ is $\exp(\beta_1)$, the estimated variance on the odds-ratio scale is $$ \exp(1.60)(0.00032) \approx 4.9530(0.00032) \approx 0.001585. $$ The delta-method standard error is $\sqrt{0.001585}\approx 0.0398$. The calculation uses the coefficient variance and the squared derivative; it does not simply reuse $0.00032$ as the odds-ratio variance.

An alternative first-order interval is obtained on the log-odds-ratio scale and then exponentiated. Here the coefficient standard error is $\sqrt{0.00032}\approx 0.01789$, giving $\exp(0.80\pm1.96(0.01789))\approx(2.149,2.305)$. This is a transformed interval, whereas the delta-method standard error above is expressed directly on the odds-ratio scale.

Interpretation and Common Pitfalls

The delta method is useful whenever the reported parameter is a smooth transformation or combination of estimated GLM coefficients. For a contrast, it incorporates coefficient correlations through $c^\mathsf{T}\Sigma c$. For a ratio represented by an exponentiated contrast, it additionally scales uncertainty by the squared estimated ratio. These are different sources of calculation, and both matter.

Using only diagonal variances. The variance of a contrast includes covariance terms. Use the full coefficient covariance matrix unless the relevant covariances are known to be zero.
Forgetting the derivative. For $g(\beta)=\exp(c^\mathsf{T}\beta)$, the gradient is $\exp(c^\mathsf{T}\beta)c$, not just $c$. The exponential factor changes the variance on the ratio scale.
Confusing the two scales. The standard error of $c^\mathsf{T}\hat{\beta}$ is on the log scale for a rate ratio or odds ratio. It is not the standard error of the exponentiated quantity.
Ignoring the covariance scale. Check whether a covariance matrix estimates the variability of $\hat{\beta}$ itself or the asymptotic covariance of $\sqrt{n}(\hat{\beta}-\beta_0)$. The latter requires division by $n$.
Applying the result without smoothness. The stated theorem requires differentiability at the true parameter and the given asymptotic normality and consistency conditions. A nonsmooth target or a parameter at a nonregular boundary needs separate analysis.

Key takeaway: To obtain uncertainty for a smooth GLM coefficient function, differentiate the function, evaluate its gradient at the estimate, and propagate the full coefficient covariance through the resulting quadratic form. For exponentiated contrasts, the derivative contributes the estimated ratio itself.

Check Your Understanding

Use the definitions and calculations above to answer the following.

For $g(\beta)=c^\mathsf{T}\beta$, what is the gradient, and what asymptotic variance does it imply?
Why must coefficient covariances generally be included when calculating the variance of a linear contrast?
A log-link model gives a log rate ratio $c^\mathsf{T}\beta$. Write the gradient of the rate ratio with respect to $\beta$.
What changes if the covariance matrix provided estimates $\operatorname{Var}(\hat{\beta})$ directly rather than the asymptotic covariance of $\sqrt{n}(\hat{\beta}-\beta_0)$?
In logistic regression, why is the standard error of an exposure coefficient not also the standard error of its exponentiated odds ratio?