The First-Order Delta Method

Why a First-Order Approximation Gives a Limit

An estimator may have a tractable asymptotic distribution while the quantity ultimately reported is a smooth function of that estimator. For example, an estimate of a probability might be converted to a log-odds, or an estimate of a positive mean might be converted to its reciprocal. The key question is how the estimator’s sampling variation changes under that transformation.

Near a fixed value $\theta$, differentiability says that a function $g$ is well approximated by its tangent line: $g(x)-g(\theta)$ is approximately $g'(\theta)(x-\theta)$. The first-order delta method makes this approximation precise for random estimators at their asymptotic scale. Its conclusion is that, to first order, the transformed estimator inherits the original limit multiplied by the derivative.

Definition: A sequence $T_n$ is asymptotically normal at rate $\sqrt n$ with center $\theta$, asymptotic mean zero, and asymptotic variance $\tau^2$ if $$ \sqrt n(T_n-\theta)\xrightarrow{d}N(0,\tau^2). $$ Here $\tau^2$ may be zero, although the usual nondegenerate case has $\tau^2>0$.

The same argument works at other deterministic rates. Let $a_n>0$ be a sequence with $a_n\to\infty$. Suppose $a_n(T_n-\theta)\xrightarrow{d}Z$, where $Z$ is a finite real-valued random variable. Let $I$ be an open interval containing $\theta$, assume $T_n\in I$ with probability one, and let $g:I\to\mathbb R$ be differentiable at $\theta$. These conditions ensure the transformed quantities are defined and the local linear approximation applies.

The Scalar First-Order Delta Method

Theorem (First-Order Delta Method): Under the conditions above, $$ a_n\{g(T_n)-g(\theta)\}\xrightarrow{d}g'(\theta)Z. $$ In particular, if $\sqrt n(T_n-\theta)\xrightarrow{d}N(0,\tau^2)$, then $$ \sqrt n\{g(T_n)-g(\theta)\}\xrightarrow{d}N\!\left(0,[g'(\theta)]^2\tau^2\right). $$ When $g'(\theta)=0$, the stated first-order limit is the constant zero; the theorem does not assert a nondegenerate normal limit.

The essential step is to show that the Taylor remainder disappears after multiplication by $a_n$. Differentiability at $\theta$ means that there is a remainder function $r$, with $r(0)=0$, such that

g(\theta+h)-g(\theta)=g'(\theta)h+r(h), \qquad \frac{r(h)}{h}\longrightarrow 0\quad\text{as }h\longrightarrow 0,\ h\ne 0.

We prove the theorem rather than treating the approximation as an informal substitution. Convergence in distribution of $a_n(T_n-\theta)$ implies that this sequence is tight, or bounded in probability. Since $a_n\to\infty$, it follows that $T_n-\theta\xrightarrow{p}0$: for any $\varepsilon>0$, tightness lets us choose $M$ so that $P(|a_n(T_n-\theta)|>M)<\varepsilon$ for all sufficiently large $n$; once $a_n\varepsilon_0>M$, the event $|T_n-\theta|>\varepsilon_0$ is contained in $|a_n(T_n-\theta)|>M$. Thus $T_n$ approaches $\theta$ in probability.

Set $D_n=T_n-\theta$, and define $R_n=r(D_n)$, taking $R_n=0$ when $D_n=0$. We next verify $a_nR_n\xrightarrow{p}0$. Write $U_n=a_nD_n$, which is tight. Given $\varepsilon>0$, choose $M<\infty$ so that $P(|U_n|>M)<\varepsilon$ for all sufficiently large $n$. For any $\eta>0$, differentiability gives a $\delta>0$ such that $|r(h)/h|<\eta/M$ whenever $0<|h|<\delta$ (if $M=0$, choose any positive bound instead). On the event $|D_n|<\delta$ and $|U_n|\le M$,

|a_nR_n| =|U_n|\left|\frac{r(D_n)}{D_n}\right| \le \eta.

Because $D_n\xrightarrow{p}0$, the probability of $|D_n|\ge\delta$ tends to zero. The probability that $|U_n|>M$ is at most $\varepsilon$ for all sufficiently large $n$. Consequently, $\limsup_n P(|a_nR_n|>\eta)\le\varepsilon$. Since $\varepsilon$ was arbitrary, $a_nR_n\xrightarrow{p}0$.

The Taylor identity now gives

a_n\{g(T_n)-g(\theta)\} =g'(\theta)a_n(T_n-\theta)+a_nR_n.

The first term converges in distribution to $g'(\theta)Z$, by the rule for multiplying a distributional limit by a constant. The second term converges in probability to zero. Slutsky’s theorem therefore yields the claimed limit. If $Z\sim N(0,\tau^2)$, multiplying by $g'(\theta)$ gives a normal variable with mean zero and variance $[g'(\theta)]^2\tau^2$, including the degenerate case when that variance is zero. $\square$

Worked Applications

Worked Example: The Logit of a Sample Proportion

Let $X_1,\ldots,X_n$ be independent Bernoulli random variables with success probability $p=0.4$, and let $\hat p_n=n^{-1}\sum_{i=1}^n X_i$. Find the asymptotic distribution of the logit estimate $\log\{\hat p_n/(1-\hat p_n)\}$. Since the sample proportion can equal zero or one for finite $n$, define the transformed statistic arbitrarily on those two events; their combined probability tends to zero, so this convention does not affect the limit.

The central limit theorem gives $\sqrt n(\hat p_n-p)\xrightarrow{d}N(0,p(1-p))$, and $p$ is in the interior of $(0,1)$. Set $g(x)=\log\{x/(1-x)\}$ for $0<x<1$. Differentiating,

g'(x)=\frac{1}{x}+\frac{1}{1-x} =\frac{1}{x(1-x)}.

At $p=0.4$, $g(p)=\log(0.4/0.6)=\log(2/3)$ and $g'(p)=1/(0.4\cdot0.6)=25/6$. The delta method therefore gives

\sqrt n\left[ \log\left\{\frac{\hat p_n}{1-\hat p_n}\right\} -\log(2/3) \right] \xrightarrow{d} N\left(0,\left(\frac{25}{6}\right)^2(0.4)(0.6)\right) =N\left(0,\frac{25}{6}\right).

The variance calculation is $(625/36)(6/25)=25/6$. The derivative magnifies the sample proportion’s local variation because the logit changes more rapidly than the identity function near $0.4$.

Worked Example: The Reciprocal of an Exponential Mean

Suppose $X_1,\ldots,X_n$ are independent exponential observations with mean $\mu=2$, so $E[X_i]=2$ and $\operatorname{Var}(X_i)=4$. The sample mean $\bar X_n$ estimates $\mu$; find the limit for its reciprocal $1/\bar X_n$, centered at $1/2$.

The central limit theorem yields $\sqrt n(\bar X_n-2)\xrightarrow{d}N(0,4)$. Choose $g(x)=1/x$ on $(0,\infty)$. Every $\bar X_n$ is positive with probability one, and $g'(x)=-1/x^2$, so $g'(2)=-1/4$. Hence

\sqrt n\left(\frac{1}{\bar X_n}-\frac12\right) \xrightarrow{d} N\left(0,\left(-\frac14\right)^2 4\right) =N\left(0,\frac14\right).

The negative derivative reverses the direction of local deviations: a sample mean above 2 produces a reciprocal below $1/2$. The limiting variance uses the square of that derivative, so its sign has no effect on the variance.

Worked Example: The Square Root of a Poisson Sample Mean

Let $X_1,\ldots,X_n$ be independent Poisson random variables with mean $\lambda=4$, and let $\bar X_n$ estimate $\lambda$. Determine the limiting distribution of $\sqrt{\bar X_n}$, centered at 2.

A Poisson variable with mean 4 has variance 4. The central limit theorem therefore gives $\sqrt n(\bar X_n-4)\xrightarrow{d}N(0,4)$. Take $g(x)=\sqrt{x}$ on $(0,\infty)$; because $\bar X_n=0$ is possible, define the transformed statistic on that event to equal zero. Since $P(\bar X_n=0)=P(X_1=\cdots=X_n=0)=e^{-4n}\to0$, this convention has no effect on the limit. On its domain, $g'(x)=1/(2\sqrt{x})$, so $g'(4)=1/4$. It follows that

\sqrt n(\sqrt{\bar X_n}-2) \xrightarrow{d} N\left(0,\left(\frac14\right)^2 4\right) =N\left(0,\frac14\right).

The centering is $g(4)=2$, and the variance is $4/16=1/4$. This is a first-order approximation around the positive value 4; the exceptional event where the sample mean is zero has vanishing probability.

Reading and Using the Limit

The theorem separates a transformed limit into two parts: the distributional behavior of the original estimator and the local slope of the function. If $\sqrt n(T_n-\theta)$ has asymptotic variance $\tau^2$, then the transformed variance is $[g'(\theta)]^2\tau^2$. A large absolute derivative amplifies local uncertainty; a small absolute derivative reduces it. This interpretation is local, at $\theta$, rather than a claim that $g$ is linear everywhere.

The centering in this result is $g(\theta)$. It is the function evaluated at the parameter around which $T_n$ fluctuates. The theorem does not replace that centering with $E[g(T_n)]$, nor does it say the transformed statistic is exactly normal for finite $n$. It establishes a limiting distribution after scaling.

Common pitfall: Applying the theorem but forgetting the derivative. If $T_n$ has asymptotic variance $\tau^2$, it is generally incorrect to give $g(T_n)$ the same asymptotic variance. The first-order variance is $[g'(\theta)]^2\tau^2$. For example, in the reciprocal-mean calculation the derivative at 2 is $-1/4$, so the variance is $4(1/4)^2=1/4$, not 4. Also, if $g'(\theta)=0$, the first-order theorem gives a zero limit; it does not justify reporting a nonzero normal variance.

The scalar delta method is useful whenever a target is a differentiable transformation of an estimator with a known asymptotic limit. Before applying it, identify the center $\theta$, check that the function is differentiable there, and compute its derivative at that point. Then multiply the original limiting random variable by that derivative, or multiply its variance by the squared derivative in the asymptotically normal case.

Check Your Understanding

For each question, identify the relevant derivative and state how it enters the limiting result.

If $\sqrt n(T_n-\theta)\xrightarrow{d}N(0,\tau^2)$ and $g'(\theta)=3$, what is the limiting variance of $\sqrt n\{g(T_n)-g(\theta)\}$?
For $g(x)=\log x$ at $\theta=5$, calculate $g'(\theta)$. If the original asymptotic variance is 10, what is the transformed asymptotic variance?
Why does differentiability at $\theta$ alone not imply that $g(T_n)$ is exactly normally distributed for finite $n$?
If $g'(\theta)=0$, what limit does the first-order delta method provide, and what does it leave unresolved?
For the reciprocal transformation $g(x)=1/x$, explain why a negative derivative still produces a nonnegative asymptotic variance.

What You'll Learn