Why Transforming an Estimate Changes Its Uncertainty
Maximum-likelihood estimation often produces estimates of parameters that are not the final quantities of interest. A model may estimate a log scale, while the reported quantity is a scale; it may estimate several coefficients, while the question concerns their ratio or a predicted function of them. A nonlinear transformation changes uncertainty as well as the point estimate. The delta method uses the local slope of that transformation to approximate the sampling variance of the resulting quantity.
The method is especially useful when the likelihood fit already provides an estimated covariance matrix for its parameter estimates. It does not require re-estimating the model for every transformed quantity. Instead, it combines that covariance matrix with derivatives of the target function. The approximation is local: it is most dependable when the estimator is approximately normal and the target is smooth near the true parameter.
Conditions and the Scalar Delta-Method Formula
Suppose the likelihood-based estimator satisfies the asymptotic normality result \(\sqrt{n}(\hat{\theta}-\theta_0)\) converges in distribution to a mean-zero normal vector with covariance matrix \(\Sigma\). Assume that \(\hat{\theta}\) is consistent for \(\theta_0\), that \(\theta_0\) is an interior parameter value, and that \(g\) is continuously differentiable in a neighborhood of \(\theta_0\). Also assume there is a consistent estimator \(\widehat{\operatorname{Var}}(\hat{\theta})\) of the covariance matrix of \(\hat{\theta}\). These are the conditions for the first-order approximation used here.
Write \(\nabla g(\theta)\) for the column vector of first derivatives of the scalar function \(g\). A first-order Taylor expansion gives \(g(\hat{\theta})\) approximately equal to \(g(\theta_0)+\nabla g(\theta_0)^\top(\hat{\theta}-\theta_0)\). Applying the asymptotic normality of \(\hat{\theta}\) to this linear term yields the variance approximation below. In practice, the unknown parameter and gradient are replaced by their estimates.
The covariance matrix in this formula estimates the covariance of \(\hat{\theta}\) itself. If an output instead reports an estimate of the asymptotic covariance of \(\sqrt{n}(\hat{\theta}-\theta_0)\), it must be divided by \(n\) to obtain the estimated covariance of \(\hat{\theta}\). Under correct likelihood specification, the covariance may be estimated using information-based methods; under misspecification, an appropriate sandwich covariance estimate can be used. The delta-method calculation then propagates whichever justified covariance estimate is supplied.
For a large-sample Wald confidence interval with confidence level \(1-\alpha\), use the estimate plus or minus a standard normal critical value times its estimated standard error:
For example, a 95% interval uses \(z_{0.975}\approx 1.96\). This is an asymptotic procedure; it does not generally call for a \(t\) critical value. The interval’s repeated-sampling interpretation is that, under the stated conditions and as sample size grows, intervals constructed by this procedure cover the fixed target \(g(\theta_0)\) at a rate approaching the stated confidence level.
Worked Example: Exponentiating a Log-Scale Estimate
Suppose a hypothetical likelihood fit estimates a log-scale parameter \(\beta\) as \(\hat{\beta}=0.400\), with estimated standard error \(0.200\). The estimated variance of \(\hat{\beta}\) is therefore \(0.200^2=0.0400\). The reporting target is the scale \(\lambda=\exp(\beta)\). Assume the estimate is asymptotically normal, the true parameter is interior, and the exponential function is smooth near it; these conditions support a first-order delta-method approximation.
Here \(g(\beta)=\exp(\beta)\), so \(g'(\beta)=\exp(\beta)\). The transformed estimate is \(\exp(0.400)=1.4918\), and the estimated standard error is
The direct 95% Wald interval is \(1.4918\pm1.96(0.2984)\). Its margin is \(0.5849\), giving endpoints \(0.9069\) and \(2.0767\), or approximately \((0.907,2.077)\). In context, this is an approximate confidence interval for the scale parameter, expressed in the scale’s units. Repeated use of this procedure under the assumptions would cover the fixed scale at a rate approaching 95%.
The delta method approximates the transformed estimator’s distribution as normal, even though an exponential is positive and its exact sampling distribution need not be normal. This example’s interval happens to have positive endpoints; that is not guaranteed in every application. The interval should therefore be checked for compatibility with the target’s possible values and substantive meaning.
Why the Gradient Determines the Result
The scalar formula is a direct consequence of local linearization. Near \(\theta_0\), a small change in \(\theta\) changes \(g(\theta)\) by approximately the gradient dotted with that change. Thus, uncertainty in directions where \(g\) changes rapidly contributes more to the target’s variance. Covariances matter as well: uncertainty in two parameter estimates can reinforce or offset each other depending on the signs of the corresponding derivatives.
For a small estimation error, \(g(\hat{\theta})-g(\theta_0)\) is approximately \(\nabla g(\theta_0)^\top(\hat{\theta}-\theta_0)\).
If the covariance of \(\hat{\theta}\) is approximately \(V\), the linear term has variance \(\nabla g(\theta_0)^\top V\nabla g(\theta_0)\).
Replace \(\theta_0\), its gradient, and \(V\) by \(\hat{\theta}\), \(\nabla g(\hat{\theta})\), and the estimated covariance matrix.
This justification also explains the method’s limitation. A first-order expansion neglects curvature and higher-order terms. If the target is sharply curved over the estimator’s uncertainty range, or if the target is not differentiable at the true parameter, the first-order normal approximation may be inaccurate.
Worked Example: A Ratio of Two Estimated Parameters
A hypothetical likelihood fit estimates two positive quantities \(a\) and \(b\) as \(\hat{a}=2.00\) and \(\hat{b}=4.00\). Their estimated covariance matrix is
The target is \(g(a,b)=a/b\), and its estimate is \(2.00/4.00=0.500\). The gradient is \(\nabla g(a,b)=(1/b,-a/b^2)^\top\), so at the estimates it is \((0.250,-0.125)^\top\). Assume the estimator is consistent and asymptotically normal, the true \(b\) is nonzero and interior, and the ratio function is smooth near the true values.
Substitution into the variance formula gives
and hence \(\widehat{SE}(\hat{a}/\hat{b})=\sqrt{0.006250}=0.07906\). The approximate 95% Wald interval is \(0.500\pm1.96(0.07906)\), which is \(0.500\pm0.15495\), or approximately \((0.345,0.655)\). If \(a\) and \(b\) are measured in the same units, the ratio is dimensionless; otherwise its units are the units of \(a\) divided by the units of \(b\).
The covariance term is negative because the derivative with respect to \(b\) is negative while the estimated covariance is positive. Omitting that term would give a different variance. The full covariance matrix, rather than only the individual standard errors, is needed for this calculation.
Several Nonlinear Targets at Once
The same reasoning applies when the target is a vector \(g(\theta)\) with \(r\) components. Let \(G(\theta)\) be its \(r\)-by-\(k\) Jacobian matrix, whose \(j\)th row contains the derivatives of the \(j\)th target component. The approximate covariance matrix of the transformed estimates is \(G(\hat{\theta})\widehat{\operatorname{Var}}(\hat{\theta})G(\hat{\theta})^\top\). Its diagonal entries give the estimated variances of individual components; its off-diagonal entries describe their estimated covariances.
Worked Example: Two Transformed Quantities and Their Covariance
Consider hypothetical estimates \(\hat{\theta}_1=1.50\) and \(\hat{\theta}_2=2.00\), with covariance estimate
The two targets are \(g_1(\theta)=\theta_1^2\) and \(g_2(\theta)=\log(\theta_2)\). Assume the estimator is consistent and asymptotically normal, the true \(\theta_2\) is positive and interior, and both functions are differentiable near the true parameter. The estimated targets are \(2.25\) and \(\log(2.00)=0.6931\). The Jacobian at the estimates is
Propagating the covariance gives
Therefore the estimated standard errors are \(\sqrt{0.360}=0.600\) for \(\theta_1^2\) and \(\sqrt{0.0225}=0.150\) for \(\log(\theta_2)\). Their approximate 95% Wald intervals are \(2.25\pm1.96(0.600)=(1.074,3.426)\) and \(0.6931\pm1.96(0.150)=(0.3991,0.9871)\). The off-diagonal value \(0.0150\) is the estimated covariance between the two transformed estimates; it is not needed to form either separate interval, but it matters for joint inference.
Practical Checks and a Common Pitfall
A reliable calculation makes the target, gradient, covariance scale, and units explicit. Before reporting an interval, check that the covariance matrix is the one for \(\hat{\theta}\), that the gradient uses the same parameter ordering, and that the final standard error has units consistent with the target. If a target has a natural restriction, such as being positive, inspect whether a direct Wald interval respects it. A confidence interval that conflicts with the parameter space is a warning that the normal approximation on the target scale may be poor; it is not a reason to silently truncate the endpoints.
- Pitfall: transforming only the point estimate. Computing \(g(\hat{\theta})\) but retaining the original parameter’s standard error ignores how quickly \(g\) changes. The correction is to calculate the gradient and use the covariance propagation formula.
- Pitfall: dropping covariance terms. For functions of multiple estimates, treating estimates as independent without justification can give an incorrect standard error. Use the full estimated covariance matrix.
- Pitfall: treating an approximation as exact. The result is first-order and asymptotic. If the target is very nonlinear near the estimate, or the estimator is far from its large-sample regime, the interval can be inaccurate.
Delta-method inference makes likelihood results interpretable on scientifically useful scales without changing the underlying fit. Its central discipline is to match a smooth target’s local sensitivity to the estimator’s full covariance. The result is an efficient approximation, not a substitute for checking assumptions, parameter constraints, and the behavior of the transformation.
Check Your Understanding
Work through these questions using the formulas and conditions in the tutorial.
- For a scalar target \(g(\theta)\), what three estimated quantities enter the delta-method variance formula?
- If a likelihood output reports the covariance of \(\sqrt{n}(\hat{\theta}-\theta_0)\), what adjustment is needed before using it as the covariance of \(\hat{\theta}\)?
- For \(g(a,b)=a/b\), why can the covariance between \(\hat{a}\) and \(\hat{b}\) affect the estimated standard error?
- What does a 95% delta-method confidence interval say about repeated sampling, and what does it not say about the probability that a fixed parameter lies in this particular interval?
- Give one reason a direct Wald interval for a transformed target might be misleading even when its standard error is calculated correctly.