From Coefficient Uncertainty to Mean Uncertainty
A fitted GLM mean is not itself a coefficient: it is a function of the coefficients evaluated at a covariate value. Its uncertainty therefore depends both on how uncertain the coefficient estimates are and on how sensitive the mean is to changes in those estimates. The delta method combines these two ingredients.
Let \(\beta\in\mathbb{R}^p\) be the regression coefficient vector, let \(x_0\in\mathbb{R}^p\) be a specified covariate vector, and let \(g\) be the link function. The linear predictor and mean at that covariate value are \(\eta_0=x_0^T\beta\) and \(\mu_0=g^{-1}(\eta_0)\). Inference here concerns the model mean \(\mu_0\), not a future response drawn from the distribution with that mean.
The covariance matrix matters in its entirety. Coefficient estimates can be correlated, and those covariances contribute to the uncertainty of a linear predictor and, consequently, of the fitted mean.
Deriving the Fitted-Mean Formula
Suppose the model is regular, the true coefficient vector is an interior point, the likelihood identifies it, and the relevant information matrix is nonsingular. Assume the large-sample approximation for \(\hat\beta\) is asymptotically normal and that its estimated covariance \(\hat V_\beta\) consistently estimates the covariance on the corresponding asymptotic scale. For a fixed \(x_0\), the fitted-mean function \(m(\beta)=h(x_0^T\beta)\) is differentiable wherever the inverse link is differentiable.
Derivation. Write \(\eta=x_0^T\beta\). The \(j\)th partial derivative follows from the chain rule: \(\partial m(\beta)/\partial\beta_j=h'(\eta)\partial(x_0^T\beta)/\partial\beta_j=h'(\eta)x_{0j}\). Collecting these derivatives gives \(\nabla m(\beta)=h'(\eta)x_0\). The delta method propagates the coefficient covariance through this gradient: \[ \operatorname{Var}(\hat\mu_0)\ \approx\ \nabla m(\beta_0)^T V_\beta\nabla m(\beta_0) =h'(x_0^T\beta_0)^2x_0^TV_\beta x_0. \] Replacing the unknown coefficients and covariance by their estimates gives the stated variance estimate. This substitution is justified asymptotically by continuity of \(h'\), consistency of \(\hat\beta\), and consistency of the covariance estimate; the delta-method normal approximation and Slutsky’s theorem then give the corresponding studentized normal approximation when the limiting variance is positive. \(\square\)
The middle quantity \(x_0^T\hat V_\beta x_0\) is the estimated variance of the fitted linear predictor. The factor \(h'(x_0^T\hat\beta)^2\) converts uncertainty from the link scale to the response scale. If \(\hat V_\beta\) is obtained from information, use the coefficient covariance estimate appropriate to the fitted model; with nuisance dispersion, that is the coefficient block of the relevant joint covariance estimate.
Link-Scale and Response-Scale Uncertainty
A useful first calculation is often the standard error of the fitted linear predictor, \(\widehat{\operatorname{SE}}(\hat\eta_0)=\sqrt{x_0^T\hat V_\beta x_0}\). Applying the inverse link then gives the response-scale standard error. For common links, the derivative has a simple form.
| Link | Inverse link \(h(\eta)\) | Derivative \(h'(\eta)\) |
|---|---|---|
| Identity | \(\eta\) | 1 |
| Log | \(e^\eta\) | \(e^\eta=h(\eta)\) |
| Logit | \(1/(1+e^{-\eta})\) | \(h(\eta)\{1-h(\eta)\}\) |
For an identity link, response-scale and link-scale standard errors agree. For a log link, the response-scale standard error is approximately the fitted mean times the link-scale standard error. For a logit link, the derivative is largest near a fitted probability of one half and becomes small near zero or one.
Worked Example: A Logistic Fitted Probability
Suppose a logistic regression has \(\hat\beta=(-1,0.5)^T\), and the estimated coefficient covariance matrix is \(\hat V_\beta=\begin{pmatrix}0.04&-0.01\\-0.01&0.01\end{pmatrix}\). At \(x_0=(1,2)^T\), calculate the fitted probability and its standard error.
The fitted linear predictor is \(\hat\eta_0=1(-1)+2(0.5)=0\). Therefore \(\hat\mu_0=1/(1+e^0)=0.5\). The estimated linear-predictor variance is \(x_0^T\hat V_\beta x_0=0.04+2(2)(-0.01)+4(0.01)=0.04\), so its standard error is \(0.2\).
For the inverse logit, \(h'(0)=0.5(1-0.5)=0.25\). Thus the estimated response-scale variance is \(0.25^2(0.04)=0.0025\), and the standard error is \(\sqrt{0.0025}=0.05\). A first-order 95% Wald interval is \(0.5\pm1.96(0.05)=(0.402,0.598)\), to three decimal places. It is an approximate interval for the fitted probability, not for a future binary outcome.
Worked Example: A Log-Link Mean and a Wald Interval
At a specified covariate vector, suppose the fitted linear predictor is \(0.8\), and its estimated variance is \(0.0156\). For a log-link model, calculate the fitted mean, its response-scale standard error, and a 95% response-scale Wald interval.
The fitted mean is \(e^{0.8}=2.2255409\). The link-scale standard error is \(\sqrt{0.0156}=0.1249000\). Because \(h'(\eta)=e^\eta\), the response-scale standard error is \(e^{0.8}\sqrt{0.0156}=2.2255409(0.1249000)\approx0.2779700\).
The Wald interval is the fitted mean plus or minus 1.96 standard errors: \(2.2255409\pm1.96(0.2779700)\). The margin is approximately \(0.5448211\), giving endpoints \(1.6807198\) and \(2.7703621\). Rounded to four decimal places, the interval is \((1.6807,2.7704)\).
Worked Example: A Poisson Mean with Correlated Coefficients
For a Poisson GLM with log link, suppose \(\hat\beta=(-0.2,0.2)^T\), \(\hat V_\beta=\begin{pmatrix}0.04&0.01\\0.01&0.0225\end{pmatrix}\), and \(x_0=(1,1)^T\). Find the fitted mean and its estimated standard error.
Here \(\hat\eta_0=-0.2+0.2=0\), so \(\hat\mu_0=e^0=1\). The linear-predictor variance is \(0.04+2(0.01)+0.0225=0.0825\). Since the derivative of the inverse log link at zero is \(e^0=1\), the response-scale variance estimate is \(1^2(0.0825)=0.0825\), and the standard error is \(\sqrt{0.0825}\approx0.2872\). The estimated variance of a future Poisson response at this covariate value would be 1 under the model; it is not the variance \(0.0825\) of the estimated mean.
What This Uncertainty Does—and Does Not—Measure
The fitted-mean standard error describes how the estimated model mean would vary across repeated samples, to first order. It is different from the conditional variability of an individual response. For example, a Poisson response has conditional variance equal to its mean, while the estimated mean has an approximate variance obtained by propagating coefficient uncertainty. A prediction interval for a future response must account for response variability as well as uncertainty in the fitted mean.
The covariance estimate may be obtained by inverting an estimated information matrix, with the appropriate treatment of dispersion parameters. Whatever method supplies \(\hat V_\beta\), use its off-diagonal entries: dropping them changes \(x_0^T\hat V_\beta x_0\) and can either inflate or understate the result.
A response-scale Wald interval, formed as \(\hat\mu_0\pm z_{1-\alpha/2}\widehat{\operatorname{SE}}(\hat\mu_0)\), uses a local normal approximation. It can behave poorly when the fitted mean is close to a boundary or when the inverse link is strongly curved over the relevant uncertainty range. An interval formed on the link scale and transformed back is another option, but it is a different construction and need not have the same endpoints.
Common Pitfalls
- Using only diagonal entries of the coefficient covariance. This omits the terms involving coefficient covariances. Calculate the full quadratic form \(x_0^T\hat V_\beta x_0\).
- Forgetting the inverse-link derivative. The variance on the response scale is not generally the linear-predictor variance. Multiply by \(h'(\hat\eta_0)^2\).
- Confusing mean uncertainty with response variability. The delta-method result concerns the fitted conditional mean; it does not include the full variation of a new response.
- Treating a Wald interval as exact. It is a large-sample approximation. State its scale and interpret it as uncertainty about the model mean.
Check Your Understanding
Use the gradient formula and distinguish carefully between the link scale, response scale, and future-response variability.
- For an identity-link model, why does the fitted-mean standard error equal the linear-predictor standard error?
- For a log-link model with fitted linear predictor \(\hat\eta_0\), express the response-scale standard error using the fitted mean and the linear-predictor standard error.
- What role do off-diagonal entries of \(\hat V_\beta\) play in the fitted-mean variance?
- In a Poisson GLM, explain why the fitted-mean variance estimate is not generally the conditional variance of a future count.
- For a logistic model, at what fitted probability is the inverse-link derivative largest, and what is its value there?