First-Order Asymptotic Variance and Efficiency

From a Limiting Distribution to Asymptotic Variance

An estimator’s variance at a fixed sample size describes its sampling variation at that sample size. First-order asymptotic variance is different: it describes the spread that remains after the estimation error is multiplied by $\sqrt n$ and a limiting distribution is taken. The distinction matters because convergence in distribution alone does not guarantee convergence of variances.

For an estimator $\hat{\theta}$ of a scalar parameter $\theta$, the usual first-order setup is that the scaled estimation error has a centered normal limit. The variance parameter of that limit is the asymptotic variance under the convention used here. In regular parametric models, it can then be compared with a lower bound determined by Fisher information.

Definition: If, at the true parameter value $\theta$, $$ \sqrt n(\hat{\theta}-\theta)\xrightarrow{d}N(0,V(\theta)), $$ then $V(\theta)$ is the first-order asymptotic variance of $\hat{\theta}$. Equivalently, the estimator’s first-order standard error is approximately $\sqrt{V(\theta)/n}$. This definition concerns the limiting distribution; it does not, by itself, assert that $n\operatorname{Var}(\hat{\theta})$ converges to $V(\theta)$.

The factor $n$ appears because many estimators have errors of order $n^{-1/2}$. If $\hat{\theta}$ is unbiased and its variance is exactly $c/n$, then the scaled error has variance $c$. But a limiting distribution is a statement about probabilities of sets, not directly about moments. To infer a limit for variances requires additional control of the tails.

Worked Example: The Bernoulli Sample Proportion

Let $X_1,\ldots,X_n$ be independent Bernoulli observations with success probability $p$, where $0<p<1$, and estimate $p$ by $\hat p=\bar X$.

The central limit theorem gives $$ \sqrt n(\hat p-p)\xrightarrow{d}N(0,p(1-p)). $$ Thus the asymptotic variance is $p(1-p)$. At every finite sample size, independence also gives $$ \operatorname{Var}(\hat p)=\operatorname{Var}(\bar X)=\frac{p(1-p)}{n}, \qquad n\operatorname{Var}(\hat p)=p(1-p). $$ In this example the scaled finite-sample variance equals the asymptotic variance exactly. That agreement is a property of this estimator and model, not a consequence of convergence in distribution alone.

When Do Finite-Sample Variances Converge?

A useful sufficient condition is uniform integrability of the squared scaled errors. This condition rules out a vanishing amount of probability carrying an unbounded amount of second-moment mass.

Theorem: Let $Z_n=\sqrt n(\hat{\theta}-\theta)$. Suppose $Z_n\xrightarrow{d}Z$, where $E[Z]=0$ and $E[Z^2]=V<\infty$, and suppose the family $\{Z_n^2:n\geq1\}$ is uniformly integrable. Then $$ n\operatorname{Var}(\hat{\theta})=\operatorname{Var}(Z_n)\longrightarrow V. $$

Proof. Uniform integrability of $Z_n^2$ implies uniform integrability of $Z_n$, since for $K>0$, $$ E\big[|Z_n|\mathbf{1}\{|Z_n|>K\}\big] \leq \frac{1}{K}E\big[Z_n^2\mathbf{1}\{|Z_n|>K\}\big]. $$ Convergence in distribution together with uniform integrability therefore gives $E[Z_n]\to E[Z]=0$ and $E[Z_n^2]\to E[Z^2]=V$. Hence $$ \operatorname{Var}(Z_n)=E[Z_n^2]-(E[Z_n])^2\longrightarrow V. $$ Since $Z_n=\sqrt n(\hat{\theta}-\theta)$, $\operatorname{Var}(Z_n)=n\operatorname{Var}(\hat{\theta})$. This proves the claim. $\square$

Worked Example: A Limiting Normal Law Does Not Fix the Variance

Consider independent observations $X_1,\ldots,X_n\sim N(\theta,\sigma^2)$, with known $\sigma^2$ and $n\geq2$. Write $\bar X$ for their mean and set $D=(X_1-X_2)/(\sqrt{2}\sigma)$. Then $D\sim N(0,1)$, and $D$ is independent of $\bar X$. Choose $a_n$ so that $P(D>a_n)=1/n$, and define the deliberately unusual estimator $$ T_n=\bar X+n\mathbf{1}\{D>a_n\}. $$

The event $\{D>a_n\}$ has probability $1/n$, which tends to zero. Thus $n^{3/2}\mathbf{1}\{D>a_n\}\xrightarrow{p}0$: for any fixed $\varepsilon>0$, its probability of exceeding $\varepsilon$ is at most $1/n$. Since $\sqrt n(\bar X-\theta)\sim N(0,\sigma^2)$, $$ \sqrt n(T_n-\theta)=\sqrt n(\bar X-\theta)+n^{3/2}\mathbf{1}\{D>a_n\} \xrightarrow{d}N(0,\sigma^2). $$ The asymptotic variance is therefore $\sigma^2$.

Yet independence gives $$ \operatorname{Var}(T_n)=\frac{\sigma^2}{n}+n^2\frac{1}{n}\left(1-\frac{1}{n}\right), \qquad n\operatorname{Var}(T_n)=\sigma^2+n^2\left(1-\frac{1}{n}\right). $$ The second expression diverges. A rare event has negligible probability for the limiting distribution but a large effect on the finite-sample variance. The squared scaled errors here are not uniformly integrable, so the theorem above does not apply.

Calculating Asymptotic Variances

A common route to an asymptotic variance is to start with a limiting distribution for a simpler statistic and then apply a differentiable transformation. The following result states the scalar delta method in the form needed here.

Theorem: Suppose $\sqrt n(T_n-\eta)\xrightarrow{d}N(0,W)$, where $W<\infty$, and $g$ is differentiable at $\eta$. Then $$ \sqrt n\{g(T_n)-g(\eta)\}\xrightarrow{d}N(0,[g'(\eta)]^2W). $$

Proof. The assumed convergence implies $T_n\xrightarrow{p}\eta$, so $T_n-\eta=O_p(n^{-1/2})$. Differentiability at $\eta$ gives $$ g(T_n)-g(\eta)=g'(\eta)(T_n-\eta)+r_n(T_n-\eta), $$ where $r_n\to 0$ in probability. Multiplying the remainder term by $\sqrt n$ gives $\sqrt n(T_n-\eta)r_n=o_p(1)$, because the first factor is bounded in probability. Slutsky’s theorem now gives the stated normal limit and its variance. $\square$

Worked Example: The Exponential Rate Estimator

Let $X_1,\ldots,X_n$ be independent exponential observations with rate $\lambda>0$, so $E[X_i]=1/\lambda$ and $\operatorname{Var}(X_i)=1/\lambda^2$. The maximum likelihood estimator of $\lambda$ is $\hat\lambda=1/\bar X$.

The central limit theorem gives $$ \sqrt n\left(\bar X-\frac{1}{\lambda}\right) \xrightarrow{d}N\left(0,\frac{1}{\lambda^2}\right). $$ For $g(x)=1/x$, $g'(1/\lambda)=-\lambda^2$. The delta method therefore yields $$ \sqrt n(\hat\lambda-\lambda)\xrightarrow{d} N\left(0,(-\lambda^2)^2\frac{1}{\lambda^2}\right) =N(0,\lambda^2). $$ The first-order asymptotic variance is $\lambda^2$.

The finite-sample calculation provides a useful check. If $S=\sum_{i=1}^nX_i$, then $S$ has a gamma distribution with shape $n$ and rate $\lambda$, and for $n>2$, $$ E[S^{-1}]=\frac{\lambda}{n-1}, \qquad E[S^{-2}]=\frac{\lambda^2}{(n-1)(n-2)}. $$ Consequently, $$ \operatorname{Var}(\hat\lambda) =n^2\left(E[S^{-2}]-E[S^{-1}]^2\right) =\frac{n^2\lambda^2}{(n-1)^2(n-2)}. $$ Multiplying by $n$ and taking the limit gives $n\operatorname{Var}(\hat\lambda)\to\lambda^2$, in agreement with the asymptotic variance.

First-Order Efficiency and the Information Bound

For an independent, identically distributed one-parameter model with density or mass function $f_\theta$, the Fisher information in one observation is $$ I(\theta)=E_\theta\left[\left\{\frac{\partial}{\partial\theta}\log f_\theta(X)\right\}^2\right], $$ when the derivative and expectation exist. Assume a smooth, identifiable model with finite positive information and the usual regularity conditions for the parametric asymptotic information bound. In this setting, a regular estimator has a limiting behavior that remains stable under local parameter changes of order $n^{-1/2}$. The precise role of that stability is developed separately; here it identifies the class of estimators to which the comparison applies.

For regular estimators with a centered normal first-order limit, the scalar asymptotic information bound is $I(\theta)^{-1}$. Thus $V(\theta)\geq I(\theta)^{-1}$; an estimator is first-order asymptotically efficient when equality holds. This is a comparison of first-order limiting variances, not necessarily of exact finite-sample variances.

Worked Example: Using Half the Normal Sample

Suppose $X_1,\ldots,X_n$ are independent $N(\theta,\sigma^2)$ observations with known $\sigma^2$. Use only the first $m=\lfloor n/2\rfloor$ observations and set $T_n=m^{-1}\sum_{i=1}^mX_i$. Since $T_n\sim N(\theta,\sigma^2/m)$, $$ \sqrt n(T_n-\theta)\sim N\left(0,\frac{n\sigma^2}{m}\right) \xrightarrow{d}N(0,2\sigma^2). $$ Its asymptotic variance is $2\sigma^2$.

For this model the score for one observation is $(X-\theta)/\sigma^2$, so $$ I(\theta)=E_\theta\left[\frac{(X-\theta)^2}{\sigma^4}\right]=\frac{1}{\sigma^2}. $$ The information bound is $I(\theta)^{-1}=\sigma^2$, and $2\sigma^2>\sigma^2$. The estimator is regular: under a local value $\theta+h/\sqrt n$, the distribution of $\sqrt n\{T_n-(\theta+h/\sqrt n)\}$ still converges to $N(0,2\sigma^2)$. It is therefore first-order less efficient than the full-sample mean, whose asymptotic variance is $\sigma^2$.

Common Mistakes and Practical Meaning

Calling $V(\theta)$ the finite-sample variance. It is the variance of the limiting scaled error. The finite-sample variance is $\operatorname{Var}(\hat{\theta})$, and identifying its scaled limit requires extra conditions.
Using convergence in distribution to assert moment convergence. The rare-event example shows why this is invalid. A condition such as uniform integrability of squared scaled errors supplies the needed tail control.
Comparing arbitrary estimators with the information bound. The bound stated here is for regular estimators under the model’s regularity conditions. Irregular estimators require separate analysis.
Forgetting the scale. If the limiting distribution is $N(0,V)$, the approximate variance of the estimator is $V/n$, not $V$.

Asymptotic variance is useful because it puts estimators with different constructions on a common first-order scale. The information bound gives a benchmark for regular estimation, while finite-sample variance and tail behavior remain important when assessing an estimator for actual use.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

If $\sqrt n(\hat{\theta}-\theta)\xrightarrow{d}N(0,9)$, what is the first-order approximate variance of $\hat{\theta}$ at sample size $n$?
Why does convergence in distribution alone not imply $n\operatorname{Var}(\hat{\theta})\to V(\theta)$?
State a sufficient condition used here to justify convergence of scaled finite-sample variances.
In the exponential rate example, what function is differentiated in the delta method, and what is the resulting asymptotic variance?
For the normal model with known variance $\sigma^2$, compare the information bound with the asymptotic variance of the estimator using the first half of the sample.