Two Routes to the Same Target
Suppose the parameter of interest is \(\psi=g(\theta)\). One route is to estimate \(\theta\) and then report \(g(\hat{\theta})\). Another is to construct an estimator \(\tilde{\psi}\) directly for \(g(\theta)\). These procedures target the same quantity, but they need not have the same sampling variability. The delta method makes the first route’s variability explicit and gives a criterion for comparing it with the second.
The comparison is about first-order asymptotic variance, not finite-sample equality. Two estimators can have matching first-order variances and different bias, distributions at finite \(n\), or observed values. Conversely, a direct estimator can be more or less variable than the transformed one.
The First-Order Comparison
The variance coefficient for \(g(\hat{\theta})\) is determined by the local slope of \(g\). If \(g\) is steep near \(\theta\), small errors in \(\hat{\theta}\) are magnified; if the slope is small, they are attenuated. A direct estimator has its own variance coefficient, which must be derived from its sampling behavior rather than inferred from the name of its target.
Proof. Write \(H_n=\hat{\theta}-\theta\). The assumed limit implies \(\sqrt{n}H_n=O_p(1)\), hence \(H_n=O_p(n^{-1/2})\) and \(H_n\xrightarrow{p}0\). Differentiability at \(\theta\) gives \[ g(\theta+H_n)-g(\theta)=g'(\theta)H_n+r_n, \] where \(r_n/H_n\xrightarrow{p}0\) (with the remainder taken as zero when \(H_n=0\)). Since \(H_n=O_p(n^{-1/2})\), it follows that \(\sqrt{n}r_n=o_p(1)\). Therefore \[ \sqrt{n}\{g(\hat{\theta})-g(\theta)\} =g'(\theta)\sqrt{n}(\hat{\theta}-\theta)+o_p(1). \] The assumed normal limit and Slutsky’s theorem give the claimed limit and variance. The direct estimator’s variance coefficient is \(W(\theta)\) by definition, so equality of the two first-order variance coefficients is precisely \(W(\theta)=\{g'(\theta)\}^2V(\theta)\). \(\square\)
When an estimator has an asymptotic linear representation, this comparison can be made especially transparently. Suppose \[ \hat{\theta}-\theta=\frac{1}{n}\sum_{i=1}^n\phi(X_i)+o_p(n^{-1/2}),\qquad \tilde{\psi}-g(\theta)=\frac{1}{n}\sum_{i=1}^n\chi(X_i)+o_p(n^{-1/2}), \] where \(\phi\) and \(\chi\) have mean zero and finite variances. The central limit theorem gives \(V(\theta)=E[\phi(X_1)^2]\) and \(W(\theta)=E[\chi(X_1)^2]\). The transformed estimator has first-order term \(g'(\theta)n^{-1}\sum_i\phi(X_i)\). Thus the variances agree if and only if \[ E[\chi(X_1)^2]=\{g'(\theta)\}^2E[\phi(X_1)^2]. \] This criterion compares the sizes of the two first-order fluctuations; it does not require their first-order terms to be the same random variable.
Worked Comparisons
Worked Example: Squaring a Binomial Proportion
Let \(X_1,\ldots,X_n\) be independent Bernoulli random variables with success probability \(p\), where \(0<p<1\). Write \(\hat{p}=n^{-1}\sum_iX_i\), and take \(g(p)=p^2\). Compare \(g(\hat{p})=\hat{p}^{\,2}\) with the direct pair estimator \[ \tilde{\psi}=\binom{n}{2}^{-1}\sum_{i<j}X_iX_j, \] which averages products over all distinct pairs.
Since \(\operatorname{Var}(\hat{p})=p(1-p)/n\) and \(g'(p)=2p\), the transformed estimator has first-order variance \[ \frac{\{2p\}^2p(1-p)}{n}=\frac{4p^3(1-p)}{n}. \] To check the direct estimator, put \(Z_i=X_i-p\). For each pair, \[ X_iX_j-p^2=pZ_i+pZ_j+Z_iZ_j. \] Averaging over pairs counts each \(Z_i\) exactly \(n-1\) times, so \[ \tilde{\psi}-p^2=2p\bar{Z}+R_n,\qquad R_n=\binom{n}{2}^{-1}\sum_{i<j}Z_iZ_j. \] Each product \(Z_iZ_j\) has mean zero and variance \(\{p(1-p)\}^2\). Distinct pair products have covariance zero: for disjoint pairs this follows from independence, and for pairs sharing one index the remaining factor has mean zero. Hence \[ \operatorname{Var}(R_n)=\frac{\{p(1-p)\}^2}{\binom{n}{2}}=O(n^{-2}), \] so \(R_n=o_p(n^{-1/2})\). The central limit theorem applied to \(2p\bar{Z}\) gives the direct estimator the same first-order variance, \(4p^3(1-p)/n\). The estimators agree to first order even though \(\tilde{\psi}\) is built from pairs rather than by squaring \(\hat{p}\).
Worked Example: Exponentiating a Normal Sample Mean
Suppose \(X_i\sim N(\theta,\sigma^2)\) independently, with known \(\sigma^2>0\), and the target is \(g(\theta)=e^\theta\). The transformed estimator is \(e^{\bar{X}}\). A different, direct estimator is \[ \tilde{\psi}=e^{-\sigma^2/2}\frac{1}{n}\sum_{i=1}^n e^{X_i}. \]
For the transformed sample mean, \(V(\theta)=\sigma^2\) and \(g'(\theta)=e^\theta\). Its first-order variance is therefore \[ \frac{e^{2\theta}\sigma^2}{n}. \] For a normal random variable, \(E[e^{X_i}]=e^{\theta+\sigma^2/2}\). Thus \(E[\tilde{\psi}]=e^\theta\), so this direct estimator is unbiased. Also \(E[e^{2X_i}]=e^{2\theta+2\sigma^2}\), which yields \[ \operatorname{Var}(e^{X_i}) =e^{2\theta+2\sigma^2}-e^{2\theta+\sigma^2} =e^{2\theta+\sigma^2}(e^{\sigma^2}-1). \] Scaling by \(e^{-\sigma^2/2}\) and averaging \(n\) independent terms gives \[ \operatorname{Var}(\tilde{\psi}) =\frac{e^{2\theta}(e^{\sigma^2}-1)}{n}. \] The lognormal terms have finite variance, so the central limit theorem gives this same coefficient as the direct estimator’s asymptotic variance. The coefficients match only if \(e^{\sigma^2}-1=\sigma^2\), which is false for \(\sigma^2>0\), since \(e^x>1+x\) for \(x>0\). For example, at \(\sigma^2=1\), the coefficients are \(e^{2\theta}\) for the transformed estimator and \(e^{2\theta}(e-1)\) for the direct estimator. Unbiasedness of the direct estimator does not imply that its first-order variance matches that of the transformed estimator.
Worked Example: Estimating the Square of a Poisson Rate
Let \(X_i\) be independent Poisson random variables with mean \(\lambda>0\). For \(g(\lambda)=\lambda^2\), compare \(\bar{X}^{\,2}\) with the direct pair estimator \(\tilde{\psi}=\binom{n}{2}^{-1}\sum_{i<j}X_iX_j\). The delta-method coefficient for \(\bar{X}^{\,2}\) is \[ \{2\lambda\}^2\operatorname{Var}(X_i)=4\lambda^3, \] because \(\operatorname{Var}(X_i)=\lambda\).
For the direct estimator, set \(Z_i=X_i-\lambda\). The identity \(X_iX_j-\lambda^2=\lambda Z_i+\lambda Z_j+Z_iZ_j\) gives \[ \tilde{\psi}-\lambda^2=2\lambda\bar{Z}+R_n,\qquad R_n=\binom{n}{2}^{-1}\sum_{i<j}Z_iZ_j. \] As in the binomial calculation, distinct pair products have covariance zero, and each has variance \(E[Z_i^2]E[Z_j^2]=\lambda^2\). Consequently \(\operatorname{Var}(R_n)=\lambda^2/\binom{n}{2}=O(n^{-2})\), so \(R_n=o_p(n^{-1/2})\). The central limit theorem applied to \(2\lambda\bar{Z}\) shows that the direct estimator has first-order variance \(4\lambda^3/n\), matching the transformed estimator. This conclusion concerns first-order variability; it does not say the two statistics are identical.
Using the Comparison for Inference
For a large-sample standard error, the transformed route uses the estimated slope and the estimated standard error for \(\hat{\theta}\). If \(\widehat{\operatorname{se}}(\hat{\theta})\) consistently estimates the standard error of \(\hat{\theta}\), the delta-method standard error is approximately \[ |g'(\hat{\theta})|\,\widehat{\operatorname{se}}(\hat{\theta}). \] A direct estimator instead uses a standard error derived from its own sampling variance. The theorem compares the limiting variance coefficients behind these calculations. It does not establish that either finite-sample standard error is exact.
If \(g'(\theta)=0\), the theorem reports a zero first-order variance for \(g(\hat{\theta})\). That is a statement about the first-order scale; it does not establish that the transformed estimator has no sampling variability. A finer approximation may be needed, but the first-order comparison alone does not provide it.
Common Mistake
- Wrong reasoning: “Both procedures estimate \(g(\theta)\), so their variances must agree.” A shared target places no such restriction on their sampling fluctuations.
- Correction: Calculate the transformed estimator’s coefficient \(\{g'(\theta)\}^2V(\theta)\), derive the direct estimator’s coefficient \(W(\theta)\), and compare them. Agreement holds exactly when those coefficients are equal.
- Another distinction: Equal variances do not mean equal estimators or equal confidence intervals. The comparison here is limited to first-order asymptotic variance.
Check Your Understanding
Use the first-order variance criterion to answer these questions.
- If \(\sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N(0,V(\theta))\), what is the first-order variance coefficient for \(g(\hat{\theta})\)?
- A direct estimator of \(g(\theta)\) has asymptotic variance coefficient \(W(\theta)\). What equation must hold for its first-order variance to match the transformed estimator’s?
- In the binomial example, why is the pair-product remainder negligible at the \(n^{-1/2}\) scale?
- Why does unbiasedness of the direct normal-model estimator not ensure variance agreement?
- What does a zero derivative imply for the first-order variance, and what does it not imply?