Two-Step Estimators and Plug-In Nuisance Estimates

Why a Plug-In Estimate Changes the Answer

A two-step estimator first estimates a nuisance quantity and then substitutes that estimate into an equation for the target parameter. The second equation is therefore random not only because it uses the observations, but also because its nuisance input was estimated from them. Treating that input as fixed generally omits part of the estimator’s sampling variation.

The key calculation is a joint first-order expansion. It shows that the target estimator responds to both its own empirical estimating function and the first-stage estimating function. Depending on their covariance, the nuisance contribution can increase or decrease the asymptotic variance, or disappear at first order.

Definition: Let $Z_1,\ldots,Z_n$ be independent and identically distributed observations. A two-step estimator consists of a nuisance estimator $\hat{\eta}$, defined by a first estimating equation, and a target estimator $\hat{\theta}$, defined by a second equation that substitutes $\hat{\eta}$. Write the equations as $$ \frac{1}{n}\sum_{i=1}^n a(Z_i;\hat{\eta})=o_p(n^{-1/2}), \qquad \frac{1}{n}\sum_{i=1}^n \psi(Z_i;\hat{\theta},\hat{\eta})=o_p(n^{-1/2}). $$ Here $\eta\in\mathbb{R}^q$ and $\theta\in\mathbb{R}^p$. The negligible residuals allow exact roots as well as approximate roots accurate enough for first-order asymptotics.

The First-Order Expansion

Suppose the true values $\eta_0$ and $\theta_0$ satisfy $E[a(Z;\eta_0)]=0$ and $E[\psi(Z;\theta_0,\eta_0)]=0$. Define the expected derivative matrices, evaluated at those true values, by $$ A=E\!\left[\frac{\partial a(Z;\eta_0)}{\partial\eta^T}\right], \qquad B=E\!\left[\frac{\partial\psi(Z;\theta_0,\eta_0)}{\partial\theta^T}\right], \qquad C=E\!\left[\frac{\partial\psi(Z;\theta_0,\eta_0)}{\partial\eta^T}\right]. $$

The dimensions are $q\times q$, $p\times p$, and $p\times q$, respectively. Assume $A$ and $B$ are nonsingular. The observations are iid; both estimating functions have finite second moments at the true values; and the estimators are consistent. Also assume differentiability in neighborhoods of the true values, convergence of the empirical derivatives to $A,B,C$, and Taylor remainders that are $o_p(n^{-1/2})$ at the estimator values. These conditions justify the expansions below; in particular, they rule out nonsmooth behavior or uncontrolled changes in the derivatives near the truth.

Theorem (two-step linear expansion): Under the conditions above, define $D=CA^{-1}$. Then $$ \sqrt{n}(\hat{\theta}-\theta_0) = -B^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n \left\{\psi(Z_i;\theta_0,\eta_0)-D a(Z_i;\eta_0)\right\} +o_p(1). $$ Thus the first-stage estimating function contributes to the target estimator’s first-order variation through the adjusted function $\psi-Da$.

To prove the expansion, first Taylor-expand the first estimating equation around $\eta_0$. The assumed derivative convergence and remainder control give $$ 0= \frac{1}{n}\sum_{i=1}^n a(Z_i;\eta_0) +A(\hat{\eta}-\eta_0) +o_p(n^{-1/2}). $$ Multiplying by $\sqrt{n}$ and solving using nonsingularity of $A$ yields $$ \sqrt{n}(\hat{\eta}-\eta_0) = -A^{-1}\frac{1}{\sqrt{n}}\sum_{i=1}^n a(Z_i;\eta_0)+o_p(1). $$

Next expand the target equation around $(\theta_0,\eta_0)$. Its derivative with respect to $\eta$ is $C$, so $$ 0= \frac{1}{n}\sum_{i=1}^n \psi(Z_i;\theta_0,\eta_0) +B(\hat{\theta}-\theta_0) +C(\hat{\eta}-\eta_0) +o_p(n^{-1/2}). $$ Multiplying by $\sqrt{n}$, substituting the first-stage expansion, and solving for $\sqrt{n}(\hat{\theta}-\theta_0)$ gives the stated result. In particular, the nuisance term enters with a minus sign inside the adjusted function because the first-stage expansion itself has leading term $-A^{-1}a$. $\square$

Theorem (asymptotic distribution and covariance): If the stacked vector $(a(Z;\eta_0)^T,\psi(Z;\theta_0,\eta_0)^T)^T$ has finite second moments, the multivariate central limit theorem and the expansion imply $$ \sqrt{n}(\hat{\theta}-\theta_0)\xrightarrow{d}N(0,V), \qquad V=B^{-1}\operatorname{Var}(\psi-Da)B^{-T}, $$ where $a=a(Z;\eta_0)$, $\psi=\psi(Z;\theta_0,\eta_0)$, and $B^{-T}=(B^{-1})^T$. The normal limit may be degenerate if $V$ is singular.

For justification, $E[a]=E[\psi]=0$, so the adjusted function also has mean zero. The iid multivariate central limit theorem applied to $\psi-Da$, followed by multiplication by $-B^{-1}$, gives the displayed covariance. Expanding the variance makes the cross-covariance contribution explicit: $$ \operatorname{Var}(\psi-Da) = \operatorname{Var}(\psi) +D\operatorname{Var}(a)D^T -\operatorname{Cov}(\psi,a)D^T -D\operatorname{Cov}(a,\psi). $$ The cross terms matter when the two estimating functions use the same observations. Ignoring them is not generally justified, even if one correctly includes the variance of the nuisance estimate. $\square$

Worked Examples

Worked Example: A Ratio with an Estimated Denominator

Let $X$ and $Y$ be independent Bernoulli random variables with $P(X=1)=0.5$ and $P(Y=1)=0.4$. Estimate $\eta_0=E[X]$ by $\hat{\eta}=\bar{X}$, and define $\hat{\theta}$ by $\bar{Y}-\hat{\theta}\hat{\eta}=0$. On the event $\bar X=0$, define $\hat\theta$ arbitrarily; that event has probability $0.5^n$, which tends to zero. Find the limiting variance of $\sqrt n(\hat\theta-\theta_0)$.

The nuisance equation is $a(X;\eta)=X-\eta$, so $A=-1$. The target equation is $\psi(X,Y;\theta,\eta)=Y-\theta\eta$. Its derivatives are $B=-\eta_0=-0.5$ and $C=-\theta_0$. Since $\theta_0=E[Y]/E[X]=0.4/0.5=0.8$, we have $C=-0.8$, and $D=CA^{-1}=(-0.8)(-1)=0.8$.

The adjusted function is $$ \psi-Da =Y-0.8(0.5)-0.8(X-0.5) =Y-0.8X. $$ Independence gives $\operatorname{Var}(Y-0.8X)=\operatorname{Var}(Y)+0.8^2\operatorname{Var}(X)=0.4(0.6)+0.64(0.5)(0.5)=0.24+0.16=0.40$. Therefore $$ V=(-0.5)^{-2}(0.40)=4(0.40)=1.60. $$ The denominator estimate contributes even though $X$ and $Y$ are independent: its variation is represented by the term involving $X$ in the adjusted function.

Worked Example: A Plug-In Equation with Correlated Contributions

Now let $X$ and $Y$ again be independent Bernoulli variables with success probability $0.5$. Estimate $\eta_0=E[X]=0.5$ by $\bar X$, and define $\hat\theta$ from the equation $\overline{Y-\theta-\hat\eta X}=0$. Find the first-order variance.

Here $a=X-\eta$, so $A=-1$. The target function is $\psi=Y-\theta-\eta X$. Since $E[X]=0.5$, the true target is $\theta_0=E[Y]-\eta_0E[X]=0.5-0.25=0.25$. The derivatives are $B=-1$ and $C=-E[X]=-0.5$, giving $D=CA^{-1}=0.5$.

At the truth the adjusted function simplifies to $$ \psi-Da =Y-0.25-0.5X-0.5(X-0.5) =Y-X. $$ Thus $\operatorname{Var}(\psi-Da)=\operatorname{Var}(Y-X)=0.25+0.25=0.50$, because $X$ and $Y$ are independent. As $B^{-1}=-1$, the target covariance is $V=0.50$.

The exact estimator is $\hat\theta=\bar Y-\bar X^2$. The result shows why simply treating $\hat\eta$ as fixed would fail: changing $\bar X$ changes the target equation both through its plug-in value and through the observed $X$ term.

Worked Example: Estimating a Variance with an Estimated Mean

Let $X\sim N(\mu,4)$, and estimate $\eta_0=\mu$ by $\bar X$. Define the target $\theta_0=\operatorname{Var}(X)=4$ through the equation with $\psi(X;\theta,\eta)=(X-\eta)^2-\theta$. Determine whether estimating the mean affects the first-order variance of $\hat\theta=n^{-1}\sum_{i=1}^n(X_i-\bar X)^2$.

The nuisance function $a=X-\eta$ has $A=-1$. The target derivative with respect to $\theta$ is $B=-1$, while the expected derivative with respect to $\eta$ is $C=E[-2(X-\mu)]=0$. Hence $D=CA^{-1}=0$: the nuisance estimate contributes nothing at first order. The adjusted function is simply $(X-\mu)^2-4$.

For a centered normal variable with variance $4$, $E[(X-\mu)^4]=3(4^2)=48$. Therefore $$ \operatorname{Var}((X-\mu)^2-4) =E[(X-\mu)^4]-4^2 =48-16=32. $$ Since $B^{-1}=-1$, the asymptotic variance is $V=32$. This example is a first-order insensitivity result, not a claim that the estimated mean has no effect on the finite-sample value of the sample variance.

Using the Result and Avoiding a Common Mistake

The expansion is useful whenever a target equation includes fitted quantities such as a mean, scale, calibration term, or other nuisance parameter. It provides a direct route to standard errors: estimate the derivatives $A,B,C$, form $D=CA^{-1}$, and use empirical versions of the adjusted contributions $\psi_i-Da_i$ to estimate their covariance. The target covariance then has the same sandwich form as in the theorem, with the estimated $B$ on both sides.

Common mistake: “The target equation is $\sum_i\psi_i=0$, so its variance is just based on $\operatorname{Var}(\psi)$.” This treats the estimated nuisance value as if it were fixed. The correction is to expand the target equation in both $\theta$ and $\eta$, then substitute the first-stage expansion. The resulting $\psi-Da$ includes the nuisance variation and its covariance with the target contribution. Only when $C=0$, or when the nuisance contribution otherwise vanishes, can it be omitted at first order.

Key takeaway: A plug-in nuisance estimate changes the target estimator’s first-order behavior by adding its influence to the target estimating function. The adjusted function $\psi-CA^{-1}a$, not $\psi$ alone, determines the asymptotic variance.

Check Your Understanding

Use the two-step expansion to answer these questions.

In the theorem, what dimensions do $A$, $B$, and $C$ have?
Why does the adjusted function contain $-CA^{-1}a$ rather than $+CA^{-1}a$?
Which terms are omitted if one treats the estimated nuisance quantity as fixed when calculating variance?
In the variance-estimation example, why is the first-stage contribution zero at first order?
Can the nuisance estimate increase or decrease the target’s asymptotic variance? Explain using the cross-covariance terms.

Two-Step Estimators and Plug-In Nuisance Estimates

What You'll Learn