Two-Step Estimators And Estimated Nuisance Quantities

Why Estimating a Nuisance Quantity Changes Variance

A second-stage estimator can depend on a quantity that is not itself the target of inference. If that nuisance quantity is estimated rather than known, its sampling error usually enters the second-stage estimator at first order. Treating it as fixed can therefore give the wrong asymptotic variance. The key calculation is to expand the second-stage estimating equation in both the target parameter and the nuisance quantity.

The nuisance contribution does not always simply add a variance term. When both stages use the same observations, their random fluctuations can be correlated. The first-order variance includes this covariance, which can increase or reduce the total uncertainty.

First-Order Expansion for a Shared Sample

Suppose \(Z_1,\ldots,Z_n\) are independent and identically distributed observations. Write \(m(Z;\theta,\eta)\) for a scalar estimating function and \(P_n m(\theta,\eta)=n^{-1}\sum_{i=1}^n m(Z_i;\theta,\eta)\). The second-step estimate solves \(P_n m(\hat{\theta},\hat{\eta})=0\). Assume \(E[m(Z;\theta_0,\eta_0)]=0\).

To describe the first step, suppose \(\hat{\eta}\) has an asymptotic linear representation \(\sqrt n(\hat{\eta}-\eta_0)=n^{-1/2}\sum_{i=1}^n \phi_\eta(Z_i)+o_p(1)\), where \(\phi_\eta\) is a mean-zero vector with finite second moments. It is the first-stage influence function: it records how each observation contributes, to first order, to the nuisance estimate.

The local expansion is a regularity condition that makes precise the required differentiability and control of the empirical remainder near the true parameters. In applications, it is typically justified by differentiability of the population estimating equation and suitable uniform control of its sample version. Stating the expansion explicitly makes clear which approximation the variance calculation depends on.

Proof and Variance Decomposition

Substitute the assumed second-stage equation into the local expansion. The result is

The sample average \(P_n m(\theta_0,\eta_0)\) is \(O_p(n^{-1/2})\) by the central limit theorem, and the first-stage representation gives \(\hat{\eta}-\eta_0=O_p(n^{-1/2})\). Since \(A\ne0\), rearranging the display first gives \(\hat{\theta}-\theta_0=O_p(n^{-1/2})\), and then gives

Replacing the nuisance error by its asymptotic linear representation yields the stated sum of influence functions. The joint central limit theorem applies because the summands are independent and identically distributed with finite second moments; the continuous mapping theorem then gives the normal limit and its variance. \(\square\)

The variance can be displayed as three parts:

The first term is the direct variability in the second-stage estimating function. The second is the propagated nuisance-estimation variability. The third is their covariance, which is present when the same observations drive both stages. The signs of \(A\) and \(B\) are already accounted for in the influence function; the displayed variance uses the corresponding signed covariance.

When the First Stage Uses an Independent Sample

The shared-sample formula uses one summand per observation because both stages are based on the same \(Z_i\)'s. If the first-stage estimate instead comes from an independent sample, its contribution still propagates through the second-stage equation, but its scale depends on its sample size. In particular, it should not be treated as though it were based on \(n\) observations unless its sample size is actually \(n\).

To verify the scaling, the first-stage term in the second-stage expansion becomes \(\sqrt n(\hat{\eta}-\eta_0)=\sqrt{n/m}\,m^{-1/2}\sum_{j=1}^m\phi_\eta(W_j)+o_p(1)\). Its limiting variance is therefore \(\rho^{-1}\operatorname{Var}\{\phi_\eta(W)\}\). The direct term is based on the \(Z\)-sample and has variance \(\operatorname{Var}\{m(Z;\theta_0,\eta_0)\}\). Independence makes the covariance between these two terms zero. Multiplication by \(-A^{-1}\) gives the stated result. \(\square\)

Common Pitfall and Practical Interpretation

A common mistake is to calculate the second-stage variance while holding \(\hat{\eta}\) fixed, then add a nuisance variance term as if the two stages were independent. The first step omits propagated variability; the second can omit the covariance when the same observations are used. The correction is to expand the estimating equation jointly in \(\theta\) and \(\eta\), and then calculate the variance of the resulting combined influence function.

This calculation matters because the nuisance estimate can either increase or decrease uncertainty in the target estimate. The direction depends on how the estimating equation responds to the nuisance quantity and on the dependence between the two stages. In practice, a first-order variance calculation should identify the data used at each stage, the nuisance influence function, and any covariance implied by shared observations before a standard error is reported.