The Scalar Delta Method for Transformed Estimators

Why Transform an Estimator?

An estimator may be easiest to analyze on one scale while the parameter of interest is reported on another. For example, a sample mean can estimate a model parameter whose value is the reciprocal of that mean. The scalar delta method tells us how a smooth transformation changes the estimator’s first-order limiting distribution. Its central calculation is simple: locally, the transformation acts like multiplication by its derivative. The conditions and the scale of the resulting variance, however, need careful attention.

Suppose $T_n$ estimates a scalar parameter $\theta$, and suppose its centered, scaled error has a limiting distribution. We want to understand the error in $g(T_n)$, which estimates $g(\theta)$. The method applies when $g$ is differentiable near $\theta$, and when $T_n$ is close to $\theta$ with high probability. That last condition matters: a local expansion cannot automatically be used at values of $T_n$ far from $\theta$.

Statement of the Scalar Delta Method

Theorem (Scalar Delta Method): Let $T_n$ be real-valued, let $\theta$ be a fixed real number, and suppose $$ \sqrt{n}(T_n-\theta)\xrightarrow{d} Z, $$ where $Z$ is a real-valued random variable. Let $g$ be defined on a neighborhood of $\theta$ and differentiable at $\theta$. Suppose also that $g(T_n)$ is defined with probability one for each $n$. Then $$ \sqrt{n}\{g(T_n)-g(\theta)\}\xrightarrow{d}g'(\theta)Z. $$ If $Z\sim N(0,\tau^2)$, the limit is $N(0,[g'(\theta)]^2\tau^2)$. This conclusion includes the case $g'(\theta)=0$, in which the stated first-order limit is degenerate at zero.

The condition $g(T_n)$ is defined can be weakened if necessary: when the transformation is only defined near $\theta$, define it arbitrarily outside that neighborhood. Since $T_n$ is consistent, the probability of using that arbitrary definition tends to zero, so it does not change the limiting distribution. Consistency follows here from the assumed scaled convergence: $\sqrt{n}(T_n-\theta)$ is bounded in probability, and hence $T_n-\theta\xrightarrow{p}0$.

The normal case is especially useful for inference. If the limiting variance of $\sqrt{n}(T_n-\theta)$ is $\tau^2$, the transformed estimator has first-order asymptotic variance $[g'(\theta)]^2\tau^2$ on the $\sqrt n$ scale. Equivalently, the usual unscaled variance approximation is $[g'(\theta)]^2\tau^2/n$. This is an asymptotic variance, not automatically an exact statement about $\operatorname{Var}(g(T_n))$ at finite $n$.

Why the Theorem Holds

Differentiability at $\theta$ gives a local remainder representation. There is a neighborhood $U$ of $\theta$ and a function $q$, defined on $U$, such that $q(\theta)=0$, $q(t)\to0$ as $t\to\theta$, and $$ g(t)-g(\theta)=g'(\theta)(t-\theta)+(t-\theta)q(t),\qquad t\in U. $$ The crucial point is that this identity is only being used for $t\in U$. Consistency makes $T_n\in U$ likely, but does not guarantee that it holds for every possible value of $T_n$.

Proof: Write $W_n=\sqrt{n}(T_n-\theta)$. By the assumed convergence in distribution, $W_n$ is bounded in probability. Also, $T_n\xrightarrow{p}\theta$, so $P(T_n\in U)\to1$. Define the scaled remainder $$ R_n=\sqrt{n}\{g(T_n)-g(\theta)\}-g'(\theta)W_n. $$ On the event $T_n\in U$, the local expansion gives $R_n=W_nq(T_n)$. On the complement, $R_n$ may not have this representation, but that complement has probability tending to zero. To see that $R_n\xrightarrow{p}0$, fix $\eta>0$ and choose $M>0$. Then $$ P(|R_n|>\eta) \leq P(T_n\notin U)+P(|W_n|>M) +P\bigl(|W_nq(T_n)|>\eta,\ T_n\in U,\ |W_n|\leq M\bigr). $$ The last probability is at most $P(|q(T_n)|>\eta/M,\ T_n\in U)$, which tends to zero because $q(t)\to0$ at $\theta$ and $T_n\xrightarrow{p}\theta$. Given any positive bound, boundedness in probability allows $M$ to be chosen so that $P(|W_n|>M)$ is eventually smaller than that bound. The first probability also tends to zero. Thus $R_n\xrightarrow{p}0$. Finally, $$ \sqrt{n}\{g(T_n)-g(\theta)\}=g'(\theta)W_n+R_n \xrightarrow{d}g'(\theta)Z $$ by Slutsky’s theorem. $\square$

For a normal limit, multiplying $Z$ by $g'(\theta)$ multiplies its variance by the square of that derivative. This explains why the derivative itself is not the variance multiplier: it scales errors, while variance is the expected squared deviation.

Worked Applications

Worked Example: Reciprocal of an Exponential Sample Mean

Let $X_1,\ldots,X_n$ be independent exponential observations with rate $\lambda=2$, so $E[X_i]=1/\lambda$ and $\operatorname{Var}(X_i)=1/\lambda^2$. Set $T_n=\bar X$, which estimates $\theta=1/\lambda$, and transform it with $g(t)=1/t$. The sample mean is positive with probability one, so $g(T_n)$ is defined.

The central limit theorem gives $$ \sqrt{n}(T_n-1/\lambda)\xrightarrow{d}N(0,1/\lambda^2). $$ Here $g'(\theta)=-1/\theta^2=-\lambda^2$. The transformed target is $g(\theta)=\lambda$, so the delta method yields $$ \sqrt{n}\{1/\bar X-\lambda\}\xrightarrow{d} N\left(0,(-\lambda^2)^2\frac{1}{\lambda^2}\right) =N(0,\lambda^2). $$ At $\lambda=2$, the limiting variance on the $\sqrt n$ scale is $4$, giving the first-order unscaled variance approximation $4/n$ and standard error $2/\sqrt n$. The negative derivative reverses the direction of small errors, but the variance remains positive because the derivative is squared.

Worked Example: A Logarithmic Transformation of a Sample Proportion

Let $X_1,\ldots,X_n$ be independent Bernoulli observations with success probability $p=0.4$, and let $T_n=\bar X$. Then $\theta=p$, and the central limit theorem gives $$ \sqrt{n}(T_n-p)\xrightarrow{d}N(0,p(1-p)). $$ Take $g(t)=\log(1+t)$. It is defined for every possible value $T_n\in[0,1]$, and $g'(p)=1/(1+p)$.

The transformed target is $\log(1.4)$. Its limiting variance is $$ [g'(p)]^2p(1-p)=\frac{0.4(0.6)}{(1.4)^2} =\frac{0.24}{1.96}=\frac{6}{49}. $$ Therefore, $$ \sqrt{n}\{\log(1+\bar X)-\log(1.4)\} \xrightarrow{d}N\left(0,\frac{6}{49}\right). $$ For a sample of size $n$, the first-order unscaled variance approximation is $6/(49n)$. This example also illustrates why it is useful to check the transformation’s domain for all possible sample values, not just values close to the true parameter.

Worked Example: Squaring a Normal Sample Mean

Suppose $X_1,\ldots,X_n$ are independent with distribution $N(1.5,4)$, and take $T_n=\bar X$. Then $$ \sqrt{n}(T_n-1.5)\xrightarrow{d}N(0,4). $$ To estimate the transformed parameter $\theta^2=1.5^2$, use $g(t)=t^2$. Since $g'(t)=2t$, we have $g'(1.5)=3$. The delta method gives $$ \sqrt{n}\{T_n^2-2.25\}\xrightarrow{d}N(0,3^2\cdot4)=N(0,36). $$ Thus the first-order unscaled variance approximation for $\bar X^2$ is $36/n$, with standard error $6/\sqrt n$. In this example $\bar X$ is normal exactly, but the variance calculation for its square here is a first-order asymptotic approximation, not an assertion that the exact finite-sample variance equals $36/n$.

Interpreting the Result and Avoiding Pitfalls

The derivative describes how much the transformed value changes for a small change in the estimator near the true parameter. A large absolute derivative magnifies local errors; a small one reduces them. For a normal limit, the transformation’s first-order asymptotic variance is $$ [g'(\theta)]^2\tau^2, $$ where $\tau^2$ is the variance of the limiting distribution of $\sqrt n(T_n-\theta)$. If the starting estimator has first-order unscaled variance $\tau^2/n$, the corresponding transformed approximation is $[g'(\theta)]^2\tau^2/n$.

A common mistake is to write $g'(\theta)\tau^2$ as the transformed variance. That uses the derivative only once. The correction is to transform the limiting error first: it is multiplied by $g'(\theta)$, and its variance is consequently multiplied by $[g'(\theta)]^2$. Another mistake is to treat the asymptotic variance formula as an exact finite-sample variance identity. Convergence in distribution alone does not guarantee convergence of variances; additional moment or uniform-integrability conditions are needed for such a claim.

Finally, if $g'(\theta)=0$, the theorem still says that the scaled transformed error converges to zero. It does not provide a useful nondegenerate normal approximation or a positive first-order variance. A different rate or a higher-order analysis may be needed; the first-order delta method does not settle that case.

Key takeaway: If $\sqrt n(T_n-\theta)\xrightarrow{d}N(0,\tau^2)$ and $g$ is differentiable at $\theta$, then the first-order transformed limit has variance $[g'(\theta)]^2\tau^2$. Apply the local expansion only where it is valid, and interpret the result as an asymptotic—not automatically exact—variance.

Check Your Understanding

Use the assumptions and scale carefully in each question.

If $\sqrt n(T_n-\theta)\xrightarrow{d}N(0,9)$ and $g'(\theta)=2$, what is the limiting distribution of $\sqrt n\{g(T_n)-g(\theta)\}$?
Why does consistency not justify applying a local Taylor expansion to every possible value of $T_n$ for each $n$?
If $g'(\theta)=-3$, by what factor is the first-order limiting variance multiplied?
What additional caution is needed before interpreting a delta-method variance as an approximation to the exact $\operatorname{Var}(g(T_n))$?
What does the first-order delta method conclude when $g'(\theta)=0$, and what does it fail to provide?