Delta-Method Intervals for Smooth Estimands

Why Transform the Standard Error?

Many scientific questions concern a function of a parameter rather than the parameter itself. A study may estimate a rate parameter \(\theta\) but report \(e^\theta\), or estimate a success probability \(p\) but report the Bernoulli variance \(p(1-p)\). Even when an estimator of \(\theta\) is approximately normal, its transformed value need not have the same standard error. The derivative of the function describes how small estimation errors in \(\theta\) affect the quantity of interest.

The delta method turns that local sensitivity into a standard error and, under suitable conditions, a confidence interval. The interval is centered at the transformed estimate. Its standard error is not obtained by simply reusing the standard error of the original estimator: it includes the derivative of the transformation.

Definition: Let \(\hat{\theta}\) estimate a scalar parameter \(\theta\), and let \(g\) be a real-valued function defined near \(\theta\). The transformed estimand is \(g(\theta)\), and its plug-in estimator is \(g(\hat{\theta})\). A first-order delta-method interval uses the local derivative \(g'(\theta)\), estimated by \(g'(\hat{\theta})\), to propagate the uncertainty in \(\hat{\theta}\).

The Delta Method and Its Proof

Suppose \(\theta\) is an interior point of the parameter space and, for some \(V_\theta>0\),

\sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N(0,V_\theta).

This assumption gives both consistency of \(\hat{\theta}\) and an asymptotic standard error of approximately \(\sqrt{V_\theta/n}\). The first-order delta method states how this limit changes after applying a smooth function.

Theorem (First-order delta method): If \(g\) is differentiable at \(\theta\), then

\sqrt{n}\{g(\hat{\theta})-g(\theta)\} \xrightarrow{d} N\!\left(0,\{g'(\theta)\}^2V_\theta\right).

To prove the result, differentiability at \(\theta\) gives the expansion

g(\hat{\theta})-g(\theta) = g'(\theta)(\hat{\theta}-\theta)+r_n, \qquad \frac{r_n}{\hat{\theta}-\theta}\xrightarrow{p}0,

where the remainder is taken to be zero when \(\hat{\theta}=\theta\). The assumed convergence in distribution implies that \(\sqrt{n}(\hat{\theta}-\theta)=O_p(1)\), so \(\hat{\theta}-\theta=O_p(n^{-1/2})\). Therefore,

\sqrt{n}r_n = \left\{\frac{r_n}{\hat{\theta}-\theta}\right\} \sqrt{n}(\hat{\theta}-\theta) =o_p(1).

Multiplying the expansion by \(\sqrt{n}\), the first term converges in distribution to a normal random variable with mean zero and variance \(\{g'(\theta)\}^2V_\theta\). The remainder converges in probability to zero. Slutsky’s theorem then gives the stated limit. \(\square\)

If \(g'(\theta)\ne 0\), the first-order asymptotic standard error of \(g(\hat{\theta})\) is \(|g'(\theta)|\sqrt{V_\theta/n}\). In practice, \(\theta\) and \(V_\theta\) are unknown, so they are replaced by estimates. The absolute value is needed because a standard error is nonnegative, even when \(g\) is decreasing.

Practical standard error: If \(\hat V\) consistently estimates \(V_\theta\), and \(g'\) is continuous near \(\theta\), use

\widehat{\operatorname{se}}\{g(\hat{\theta})\} = |g'(\hat{\theta})|\sqrt{\frac{\hat V}{n}}.

Building an Interval With Asymptotic Coverage

Let \(0<\alpha<1\), and let \(z_{1-\alpha/2}\) denote the \(1-\alpha/2\) quantile of the standard normal distribution. If \(g'(\theta)\ne 0\), the delta-method interval for \(g(\theta)\) is

g(\hat{\theta}) \ \pm\ z_{1-\alpha/2} |g'(\hat{\theta})|\sqrt{\frac{\hat V}{n}}.

For this interval’s coverage claim, assume \(\hat V\xrightarrow{p}V_\theta>0\), \(\hat{\theta}\xrightarrow{p}\theta\), and that \(g\) is continuously differentiable in a neighborhood of \(\theta\). Also assume the asymptotic normality condition above and \(g'(\theta)\ne0\). These conditions make the estimated standard error consistent for the first-order standard error, after multiplication by \(\sqrt n\).

In detail, continuity of \(g'\), consistency of \(\hat{\theta}\), and consistency of \(\hat V\) imply

|g'(\hat{\theta})|\sqrt{\hat V} \xrightarrow{p} |g'(\theta)|\sqrt{V_\theta}.

Combining this with the delta-method limit and Slutsky’s theorem yields

\frac{g(\hat{\theta})-g(\theta)} {|g'(\hat{\theta})|\sqrt{\hat V/n}} \xrightarrow{d}N(0,1).

The interval contains \(g(\theta)\) exactly when the absolute value of this statistic is at most \(z_{1-\alpha/2}\). The standard normal distribution assigns probability \(1-\alpha\) to that interval, and has probability zero at its endpoints. Convergence in distribution therefore gives

P\!\left( g(\theta)\in \left[ g(\hat{\theta})-z_{1-\alpha/2}|g'(\hat{\theta})|\sqrt{\frac{\hat V}{n}}, \; g(\hat{\theta})+z_{1-\alpha/2}|g'(\hat{\theta})|\sqrt{\frac{\hat V}{n}} \right] \right) \longrightarrow 1-\alpha.

This is a large-sample coverage statement. It does not assert exact coverage at a particular finite sample size.

Worked Applications

Worked Example: An Exponential Estimand

Suppose \(Y_1,\ldots,Y_{100}\) are independent \(N(\theta,4)\) observations, and the sample mean is \(1.1\). Its standard error is \(2/\sqrt{100}=0.2\). Construct a first-order 95% delta-method interval for \(g(\theta)=e^\theta\), using the standard normal critical value \(1.96\).

The transformed estimate is \(g(\hat{\theta})=e^{1.1}=3.0041660239\). Since \(g'(\theta)=e^\theta\), the plug-in derivative is also \(e^{1.1}=3.0041660239\). The delta-method standard error is therefore

e^{1.1}(0.2)=0.6008332048.

The margin of error, calculated before rounding, is

1.96\{e^{1.1}(0.2)\}=1.1776330814.

Thus the interval is

\left[ e^{1.1}-1.96(0.2e^{1.1}), \; e^{1.1}+1.96(0.2e^{1.1}) \right] \approx [1.8265,\;4.1818].

Both endpoints use the unrounded estimate and margin before rounding. In particular, the lower endpoint is approximately \(1.82653294\), which rounds to \(1.8265\).

Worked Example: The Bernoulli Variance

In \(n=400\) independent Bernoulli trials, 120 are successes. Estimate the variance \(g(p)=p(1-p)\) of one trial and construct a first-order 95% interval. For \(0<p<1\), the sample proportion \(\hat p\) has asymptotic variance \(p(1-p)/n\).

The estimate is \(\hat p=120/400=0.30\), so the transformed estimate is

g(\hat p)=0.30(1-0.30)=0.21.

The derivative is \(g'(p)=1-2p\), giving \(g'(\hat p)=1-0.60=0.40\). The estimated standard error of \(\hat p\) is

\sqrt{\frac{0.30(0.70)}{400}} = \sqrt{0.000525} \approx 0.02291288.

Propagating this uncertainty gives

\widehat{\operatorname{se}}\{g(\hat p)\} = 0.40(0.02291288) \approx 0.00916515.

The 95% margin is \(1.96(0.00916515)\approx0.01796370\). Hence the interval is

[0.21-0.01796370,\;0.21+0.01796370] \approx [0.1920,\;0.2280].

The derivative reduces the standard error here because the variance function changes relatively slowly near \(\hat p=0.30\). Omitting the factor \(0.40\) would give the wrong uncertainty for the variance estimand.

Worked Example: Squaring a Mean

Suppose \(X_1,\ldots,X_{100}\) are independent \(N(\theta,1)\) observations with sample mean \(\bar X=2.1\). Estimate \(\theta^2\) using \(g(\bar X)=\bar X^2\), and form a first-order 95% interval. The standard error of \(\bar X\) is \(1/\sqrt{100}=0.1\).

The plug-in estimate is \(2.1^2=4.41\), and \(g'(x)=2x\), so \(g'(2.1)=4.2\). The delta-method standard error is

$$ 4.2(0.1)=0.42. $$

The margin of error is \(1.96(0.42)=0.8232\), yielding

[4.41-0.8232,\;4.41+0.8232]=[3.5868,\;5.2332].

This construction relies on the true \(\theta\) being nonzero, so the first derivative of \(g\) at the true parameter is nonzero. The observed sample mean alone cannot verify that condition; it is part of the model assumptions behind the approximation.

Common Pitfalls and Interpretation

Leaving out the derivative. The transformed estimate is \(g(\hat{\theta})\), but its standard error is approximately \(|g'(\hat{\theta})|\) times the standard error of \(\hat{\theta}\). The derivative measures how strongly the estimand responds to changes in the parameter.
Forgetting to square the derivative in a variance calculation. The asymptotic variance of \(g(\hat{\theta})\) is \(\{g'(\theta)\}^2V_\theta/n\); the standard error uses the absolute derivative.
Applying the usual formula when \(g'(\theta)=0\). If the first derivative vanishes, the first-order variance is zero and the studentized limit above does not apply. For example, at \(\theta=0\), the derivative of \(g(\theta)=\theta^2\) is zero. A first-order interval based on that derivative is not justified; a different approximation is needed.
Treating nominal coverage as exact. The interval is justified by a limiting distribution under stated conditions. In a finite sample, its actual coverage can differ from the nominal level.

The delta method is useful because many estimands are smooth functions of a parameter whose estimator already has a well-understood large-sample distribution. Its scope is also specific: it provides a first-order approximation at an interior parameter value with a nonzero derivative for the usual studentized interval. Checking that local condition is part of the inference, not a detail to skip.

Key takeaway: For a smooth scalar estimand \(g(\theta)\), center the interval at \(g(\hat{\theta})\) and propagate uncertainty with \(|g'(\hat{\theta})|\sqrt{\hat V/n}\). The resulting interval has asymptotic \(1-\alpha\) coverage when the normal approximation, variance estimate, smoothness, and nonzero-derivative conditions hold.

Check Your Understanding

Use the assumptions and formulas above to answer these questions.

If \(g'(\hat{\theta})=-2\) and the estimated standard error of \(\hat{\theta}\) is \(0.15\), what is the delta-method standard error for \(g(\hat{\theta})\)?
For \(g(\theta)=\theta^3\), what derivative factor appears in the delta-method standard error?
Why is \(g'(\theta)\ne0\) required for the usual studentized interval proof?
In the Bernoulli variance example, what error would result from using the standard error of \(\hat p\) directly as the standard error of \(\hat p(1-\hat p)\)?
Is the delta-method interval guaranteed to have exactly its nominal coverage for a fixed sample size? Explain.