Worked Derivations for Consistency and Asymptotic Normality

From Estimator Construction to Limiting Distribution

Consistency and asymptotic normality answer different questions. Consistency says that an estimator approaches the target as the sample grows. Asymptotic normality describes the estimator’s centered sampling variation at a specified rate, often \(\sqrt{n}\). In practice, those conclusions are derived from the estimator’s defining equation or criterion, rather than from its name.

This tutorial works through three constructions: a root of a sample moment, a smooth extremum estimator, and a two-step plug-in estimator. In each case, the main ingredients are a population target identified by the model, a law of large numbers that brings sample quantities close to population quantities, and a central limit theorem for the leading random fluctuation.

Definition: Let \(Z_1,\ldots,Z_n\) be independent and identically distributed observations, and let \(\theta_0\) be a fixed parameter. An estimator \(\hat{\theta}_n\) is consistent if \(\hat{\theta}_n\) converges in probability to \(\theta_0\). It is asymptotically normal at rate \(\sqrt{n}\) if \(\sqrt{n}(\hat{\theta}_n-\theta_0)\) converges in distribution to a normal random variable with a finite variance.

Worked Derivation 1: A Sample-Moment Root

Suppose a scalar estimator solves a sample moment equation. Write \(P_n m(Z,\theta)=n^{-1}\sum_{i=1}^n m(Z_i,\theta)\), and let \(M(\theta)=E[m(Z_i,\theta)]\) denote its population counterpart. The population parameter is defined by \(M(\theta_0)=0\).

A standard route to consistency requires that \(\theta_0\) be the unique root of \(M\) in the parameter space, that the sample moments converge uniformly to their expectations, and that the sample root is chosen in that space. For asymptotic normality, additionally assume \(m(Z_i,\theta)\) is differentiable near \(\theta_0\), \(A=M'(\theta_0)\ne 0\), and \(m(Z_i,\theta_0)\) has finite variance. The observations are iid, so the central limit theorem applies to the sample moment at the true parameter.

Worked Example: A Ratio Identified by a Moment Equation

Let \(X_i=2\) for every observation, and suppose \(Y_i\) equals 4 or 8 with probability one half each. Consider the moment \(m((Y_i,X_i),\theta)=Y_i-\theta X_i\). The population moment is \(M(\theta)=6-2\theta\), whose unique root is \(\theta_0=3\). Assume a hypothetical sample of four observations has \(Y=(4,4,8,8)\) and \(X=(2,2,2,2)\).

The sample-moment equation gives

0=\frac{1}{4}\sum_{i=1}^4(Y_i-\hat{\theta}X_i) =\frac{24-8\hat{\theta}}{4}, \qquad \hat{\theta}=\frac{24}{8}=3.

For these data the estimate equals the population target. More generally, the sample equation has root \(\hat{\theta}=\bar{Y}/\bar{X}\). The denominator converges to \(E[X_i]=2\), and the population moment has the unique root 3, so the law of large numbers and continuity of the ratio imply consistency.

For the limiting distribution, \(M'(\theta_0)=-E[X_i]=-2\), and \(m((Y_i,X_i),\theta_0)=Y_i-6\), which is \(-2\) or \(2\) with equal probabilities. Its variance is 4. The root expansion therefore yields

\sqrt{n}(\hat{\theta}-3) = -\frac{1}{M'(3)}\frac{1}{\sqrt{n}}\sum_{i=1}^n(Y_i-6)+o_p(1) \ \xrightarrow{d}\ N(0,1).

The limiting variance is \(4/(-2)^2=1\), in squared units of \(\theta\). The result describes sampling variation around 3; it does not say that the estimate from every finite sample is normally distributed.

Why the Moment Expansion Works

The key step is to expand the sample moment around the true root. Since \(P_n m(Z,\hat{\theta})=0\), a mean-value expansion gives, for some intermediate value \(\tilde{\theta}\) between \(\hat{\theta}\) and \(\theta_0\),

0=P_n m(Z,\theta_0)+P_n m_\theta(Z,\tilde{\theta})(\hat{\theta}-\theta_0).

Consistency makes \(\tilde{\theta}\) approach \(\theta_0\). Under the differentiability and uniform convergence conditions, the derivative term approaches \(A=M'(\theta_0)\), which is nonzero. Rearranging, and applying the central limit theorem to the first term, gives

\sqrt{n}(\hat{\theta}-\theta_0) = -A^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n m(Z_i,\theta_0)+o_p(1) \ \xrightarrow{d}\ N\!\left(0,A^{-2}\operatorname{Var}[m(Z_i,\theta_0)]\right).

The derivative measures how quickly the population moment changes with the parameter. If that slope is close to zero, a given random fluctuation in the sample moment produces a larger change in the root. In vector problems, the corresponding derivative is a Jacobian and the scalar inverse is replaced by a matrix inverse.

Worked Derivation 2: A Smooth Extremum Estimator

An extremum estimator minimizes or maximizes a sample criterion \(Q_n(\theta)=P_n q(Z,\theta)\). For a minimizer, useful conditions include a compact parameter space, a continuous population criterion \(Q(\theta)=E[q(Z,\theta)]\) with a unique minimum at an interior point \(\theta_0\), and uniform convergence of \(Q_n\) to \(Q\). These conditions give consistency. For asymptotic normality, assume the criterion is twice differentiable near \(\theta_0\), its population Hessian \(H=E[q_{\theta\theta}(Z_i,\theta_0)]\) is nonsingular, and the criterion gradient has finite second moments. The sample minimizer is assumed to satisfy its first-order condition with probability approaching one.

Worked Example: Minimizing a Criterion in a Squared Parameter

Let \(W_i\) equal 1 or 9 with equal probability, and estimate a positive scalar \(\theta\) by minimizing \(Q_n(\theta)=\frac{1}{2}P_n[(\theta^2-W_i)^2]\) over \([1,3]\). The population mean of \(W_i\) is 5 and its variance is 16. The population criterion is uniquely minimized at \(\theta_0=\sqrt{5}\), which lies inside the parameter interval. For a hypothetical sample of four observations \(W=(1,1,9,9)\), the sample mean is 5.

Differentiating the sample criterion gives \(Q_n'(\theta)=2\theta(\theta^2-\bar{W})\). Since the minimizer is positive, its first-order condition implies \(\hat{\theta}=\sqrt{\bar{W}}=\sqrt{5}\), approximately 2.236. The law of large numbers gives \(\bar{W}\to 5\), and continuity of the positive square root gives consistency.

For the asymptotic variance, the individual criterion gradient at the target is \(q_\theta(W_i,\theta_0)=2\sqrt{5}(5-W_i)\), with mean zero. Its variance is \(J=4(5)\operatorname{Var}(W_i)=320\). The population Hessian is

H=E[q_{\theta\theta}(W_i,\sqrt{5})] =E[6\theta^2-2W_i]\big|_{\theta=\sqrt{5}} =30-10=20.

A Taylor expansion of the first-order condition gives \(\sqrt{n}(\hat{\theta}-\theta_0)=-H^{-1}n^{-1/2}\sum_i q_\theta(W_i,\theta_0)+o_p(1)\). Thus,

\sqrt{n}(\hat{\theta}-\sqrt{5}) \ \xrightarrow{d}\ N\!\left(0,H^{-2}J\right) = N(0,320/400) = N(0,0.8).

The criterion curvature \(H\) scales the gradient noise: greater curvature means the minimizer moves less for a given change in the gradient. Here the limiting variance is 0.8 in squared units of \(\theta\).

Worked Derivation 3: A Two-Step Plug-In Estimator

A plug-in estimator first estimates quantities and then substitutes them into a target function. Suppose \(\hat{\beta}\to_p\beta_0\), \(\hat{\eta}\to_p\eta_0\), and the target is \(\theta_0=g(\beta_0,\eta_0)\), with \(g\) continuously differentiable near \((\beta_0,\eta_0)\). Consistency of both first-stage estimates and continuity of \(g\) imply \(\hat{\theta}=g(\hat{\beta},\hat{\eta})\to_p\theta_0\).

For a limiting distribution, assume the estimates use the same iid observations and have a joint asymptotic normal distribution. A first-order expansion of \(g\) then gives a linear combination of their joint fluctuations. In particular, their covariance matters unless independence is justified.

Worked Example: A Product of Two Estimated Means

Suppose \(X_i\) and \(Y_i\) are independent within each iid observation. Each \(X_i\) equals 1 or 3 with equal probability, and each \(Y_i\) equals 2 or 6 with equal probability. Define \(\eta_0=E[X_i]=2\), \(\beta_0=E[Y_i]=4\), and the target \(\theta_0=\beta_0\eta_0=8\). Estimate \(\eta_0\) in the first stage by \(\hat{\eta}=\bar{X}\); in the second stage estimate \(\beta_0\) by \(\hat{\beta}=\bar{Y}\) and plug both estimates into \(\hat{\theta}=\hat{\beta}\hat{\eta}\).

For a hypothetical sample of four observations containing each pair \((X,Y)=(1,2),(1,6),(3,2),(3,6)\) once, \(\bar{X}=2\) and \(\bar{Y}=4\). Hence \(\hat{\theta}=4(2)=8\). Both sample means converge to their population means, so their product is consistent for 8.

For the asymptotic distribution, the gradient of \(g(\beta,\eta)=\beta\eta\), in the order \((\beta,\eta)\), is \((\eta,\beta)\). At the target this is \((2,4)\). The first-order expansion is

\sqrt{n}(\hat{\theta}-8) = 2\sqrt{n}(\bar{Y}-4)+4\sqrt{n}(\bar{X}-2)+o_p(1).

Here \(\operatorname{Var}(Y_i)=4\), \(\operatorname{Var}(X_i)=1\), and \(\operatorname{Cov}(X_i,Y_i)=0\) by the stated independence. The limiting variance is therefore

2^2(4)+4^2(1)+2(2)(4)(0)=16+16=32, \qquad \sqrt{n}(\hat{\theta}-8)\xrightarrow{d}N(0,32).

The variance is in squared units of the product target. If the two stages had nonzero covariance, the covariance term in this calculation would not vanish. When stages reuse observations, that dependence must be derived or estimated rather than ignored.

Common Pitfalls and Practical Checks

Using a central limit theorem before establishing consistency. A local expansion around \(\theta_0\) is useful only when the estimator is near \(\theta_0\). Establish identification and consistency, or otherwise justify that the selected solution is local to the target.
Dropping the derivative or curvature. For a moment root, the variance includes the inverse squared population derivative. For a smooth extremum estimator, it includes the inverse Hessian on both sides of the gradient variance. Randomness alone does not determine the estimator’s limiting variance.
Treating two estimated stages as independent by default. If they use the same observations, their limiting covariance may be nonzero. The plug-in expansion includes that covariance through the joint limiting distribution.
Claiming asymptotic normality from a finite-sample match. An estimate equal to its target in one hypothetical sample does not establish a limiting result. The distributional conclusion depends on the sampling assumptions, regularity conditions, and expansion.

A useful check is to identify the leading random term in the expansion. Its variance should reflect the moment variability, criterion-gradient variability, or joint first-stage variability actually present in the model. The assumptions must support both the law of large numbers used for consistency and the central limit theorem used for the limiting distribution.

Check Your Understanding

Use the assumptions and derivations above to answer the following questions.

For a scalar sample-moment root, which population quantity must be nonzero for the first-order expansion to identify the estimator’s rate and variance?
In the squared-parameter criterion example, what role does the population Hessian play in the asymptotic variance?
Why does consistency of both estimated means imply consistency of their product in the plug-in example?
If the first-stage and second-stage estimates in the product example are positively correlated, what happens to the covariance term in the variance calculation?
What is the limiting distribution of \(\sqrt{n}(\hat{\theta}-8)\) in the product example, and what is its variance?