Plug-In Estimation with Nuisance Parameters

Why Nuisance Estimation Enters the Target

Many estimators involve a parameter that is not itself the main object of interest but must be estimated to calculate that object. Examples include a mean adjusted using an estimated scale, a prediction evaluated at estimated coefficients, or a moment that depends on an estimated calibration parameter. Such a parameter is called a nuisance parameter. Although it is not the target, its sampling error can affect the target estimator at first order.

The key calculation is a first-order expansion around the true nuisance value. It separates two sources of variation: the sample variation in the target calculation when the nuisance value is held fixed, and the change caused by estimating that nuisance value. This separation is also the basis for calculating the estimator’s asymptotic variance.

Definition: Let \(W_1,\ldots,W_n\) be an independent, identically distributed sample with distribution \(P\). Let \(\eta_0\in\mathbb R^q\) be a nuisance parameter, and let \(m(W;\eta)\) be a scalar function. The target is \(\tau_0=P m(W;\eta_0)=E[m(W;\eta_0)]\), while the plug-in estimator is \(\hat\tau=\mathbb P_n m(W;\hat\eta)\), where \(\mathbb P_n f=n^{-1}\sum_{i=1}^n f(W_i)\) and \(\hat\eta\) estimates \(\eta_0\).

The function \(m\) may depend directly on the observation and on \(\eta\). The target \(\tau_0\) is defined using the true nuisance value, whereas \(\hat\tau\) uses its estimate. In this tutorial, the first-order effect of nuisance estimation means the term in the expansion that is linear in \(\hat\eta-\eta_0\).

The First-Order Expansion

Suppose \(\eta\mapsto m(W;\eta)\) is differentiable near \(\eta_0\). Define the row-vector derivative

A=\left.\frac{\partial}{\partial\eta'}P m(W;\eta)\right|_{\eta=\eta_0} =E\left[\left.\frac{\partial m(W;\eta)}{\partial\eta'}\right|_{\eta=\eta_0}\right].

For the expansion below, assume the sample is independent and identically distributed, \(\hat\eta\) is consistent and has root-\(n\) rate, and the derivatives and Taylor remainders are sufficiently well behaved for sample averages of derivatives to converge to their expectations. In particular, assume the Taylor remainder and the centered empirical change in \(m\) are both \(o_p(n^{-1/2})\). These are standard smoothness and integrability requirements; they ensure that only the displayed linear terms matter at the root-\(n\) scale.

Theorem: Under these conditions,

\hat\tau-\tau_0 = (\mathbb P_n-P)m(W;\eta_0) + A(\hat\eta-\eta_0) + o_p(n^{-1/2}).

The first term is direct sample variation: it is the fluctuation in the sample average that would remain if the nuisance value were known. The second term is the nuisance-estimation contribution: it is the derivative of the target with respect to the nuisance parameter, multiplied by the nuisance estimation error. The derivative \(A\) describes sensitivity, while \(\hat\eta-\eta_0\) describes the size and direction of the error.

Why the Expansion Holds

Add and subtract \(\mathbb P_n m(W;\eta_0)\) and \(P m(W;\eta_0)\). This gives an exact decomposition:

\hat\tau-\tau_0 = (\mathbb P_n-P)m(W;\eta_0) + P\{m(W;\hat\eta)-m(W;\eta_0)\} + (\mathbb P_n-P)\{m(W;\hat\eta)-m(W;\eta_0)\}.

The first term is the direct sample fluctuation. A Taylor expansion of the second term around \(\eta_0\) gives \(A(\hat\eta-\eta_0)\), plus a remainder negligible at order \(n^{-1/2}\). For the last term, differentiability gives a change of order \(\|\hat\eta-\eta_0\|\), while the centered empirical average of the derivative is of order \(O_p(n^{-1/2})\). With a root-\(n\) nuisance estimate, their product is \(O_p(n^{-1})=o_p(n^{-1/2})\). Thus this last term does not add a first-order contribution under the stated conditions.

The derivation makes clear why consistency alone is not enough for root-\(n\) inference: the nuisance error must be small enough, and the Taylor remainder must be controlled at the same scale as the direct sampling fluctuation.

Worked Example: A Ratio Using an Estimated Denominator

Suppose the target is \(\tau_0=\mu_Y/\eta_0\), where \(\mu_Y=E[Y]\) and \(\eta_0\ne0\) is a nuisance parameter. The estimator is \(\hat\tau=\bar Y/\hat\eta\). For a concrete hypothetical sample, let the four observed \(Y\) values be 8, 10, 12, and 14, so \(\bar Y=11\). Suppose the true values used to describe the sampling error are \(\mu_Y=10\) and \(\eta_0=2\), and the nuisance estimate is \(\hat\eta=2.1\).

Here \(m(Y;\eta)=Y/\eta\), and \(A=-\mu_Y/\eta_0^2=-10/4=-2.5\). The direct term is \((\bar Y-\mu_Y)/\eta_0=(11-10)/2=0.5\). The nuisance term is \(A(\hat\eta-\eta_0)=(-2.5)(0.1)=-0.25\). The first-order expansion therefore gives a change of \(0.5-0.25=0.25\) from the target.

The target is \(10/2=5\), while the plug-in estimate is \(11/2.1=5.2381\), an actual change of \(0.2381\). The difference between the actual change and the first-order approximation is \(0.2381-0.25=-0.0119\), the finite-sample remainder. The example shows that the nuisance contribution can offset or reinforce direct sample variation.

Combining the Terms for Asymptotic Inference

A useful case is one in which the nuisance estimator itself has an asymptotically linear representation. Suppose

\sqrt n(\hat\eta-\eta_0) = \frac{1}{\sqrt n}\sum_{i=1}^n\phi(W_i)+o_p(1), \qquad E[\phi(W_i)]=0,

where \(\phi(W_i)\) is a \(q\)-vector with finite second moments. Substituting this representation into the expansion yields

\sqrt n(\hat\tau-\tau_0) = \frac{1}{\sqrt n}\sum_{i=1}^n \left\{m(W_i;\eta_0)-\tau_0+A\phi(W_i)\right\} +o_p(1).

The expression in braces is the combined influence contribution: the centered direct term plus the nuisance contribution. If it has finite variance, the central limit theorem gives an asymptotic normal distribution with variance

V=\operatorname{Var}\{m(W;\eta_0)-\tau_0+A\phi(W)\}.

This variance includes the covariance between direct sample variation and nuisance estimation. In general, the two terms use the same observations and are not independent. If \(D=m(W;\eta_0)-\tau_0\), then

V=\operatorname{Var}(D)+A\operatorname{Var}(\phi)A' +2\operatorname{Cov}(D,A\phi),

where the last term is the scalar covariance contribution. Omitting it is justified only when the covariance is zero, not merely because the two sources of variation have different names.

Worked Example: A Product of Two Sample Means

Consider the target \(\tau_0=\mu_Y\eta_0\), where \(\mu_Y=E[Y]\) is the direct target component and \(\eta_0=E[Z]\) is a nuisance parameter. Estimate it by \(\hat\tau=\bar Y\bar Z\), treating \(\hat\eta=\bar Z\) as the nuisance estimate. For hypothetical paired observations, take \(Y=(2,4,6,8)\) and \(Z=(1,3,2,4)\). Thus \(\bar Y=5\) and \(\bar Z=2.5\). Suppose \(\mu_Y=4.5\) and \(\eta_0=2.2\).

The target is \(4.5(2.2)=9.9\), and the estimate is \(5(2.5)=12.5\), a difference of \(2.6\). The direct contribution is \(\eta_0(\bar Y-\mu_Y)=2.2(0.5)=1.1\). The nuisance contribution is \(\mu_Y(\bar Z-\eta_0)=4.5(0.3)=1.35\). Their sum, \(2.45\), is the first-order approximation.

The exact algebra also identifies the remainder:

\bar Y\bar Z-\mu_Y\eta_0 = \eta_0(\bar Y-\mu_Y) +\mu_Y(\bar Z-\eta_0) +(\bar Y-\mu_Y)(\bar Z-\eta_0).

For these values the remainder is \((0.5)(0.3)=0.15\), so \(2.45+0.15=2.6\), as required. At the root-\(n\) scale, the product of two root-\(n\) estimation errors is typically of order \(n^{-1}\), which is smaller than \(n^{-1/2}\). Because \(Y\) and \(Z\) are paired, the two first-order contributions may be correlated; their covariance belongs in the asymptotic variance.

Worked Example: Keeping the Covariance in the Variance

Suppose a hypothetical plug-in estimator has a combined influence contribution \(D+A\phi\), with \(\operatorname{Var}(D)=4\), \(\operatorname{Var}(\phi)=1\), \(A=-2\), and \(\operatorname{Cov}(D,\phi)=-0.6\). These are assumed asymptotic moments for illustration; they are not estimates from a real study.

The variance of the nuisance contribution \(A\phi\) is \(A^2\operatorname{Var}(\phi)=(-2)^2(1)=4\). Its covariance with the direct term contributes \(2A\operatorname{Cov}(D,\phi)=2(-2)(-0.6)=2.4\). Therefore,

$$ V=4+4+2.4=10.4. $$

An asymptotic standard error for a sample of size \(n\) is \(\sqrt{10.4/n}\). For \(n=400\), this is \(\sqrt{10.4/400}=0.1612\). Assuming independence would instead give variance \(4+4=8\), which is incorrect here because the specified covariance is not zero.

When the Nuisance Error Is Not Root-\(n\)

The expansion also clarifies what to check when \(\hat\eta\) does not have a root-\(n\) rate. The nuisance term remains \(A(\hat\eta-\eta_0)\). If it is larger than \(n^{-1/2}\), it can dominate the direct sample variation, and the usual root-\(n\) approximation may fail. If it is smaller, it may be negligible at that scale. The outcome depends on both the nuisance estimator’s rate and the derivative \(A\); a small nuisance error can still matter when the target is sensitive to \(\eta\).

For more complicated nuisance objects, such as functions rather than finite-dimensional vectors, the same separation is conceptually useful, but differentiability and remainder control require appropriate function-space conditions. The finite-dimensional expansion here should not be applied automatically without checking those conditions.

Common Mistake

Wrong reasoning: “The nuisance estimate is only an intermediate calculation, so I can treat it as fixed when deriving the target estimator’s uncertainty.”
Correction: The target estimator changes when \(\hat\eta\) changes. Include \(A(\hat\eta-\eta_0)\) in the first-order expansion, and include its variance and covariance with the direct term when calculating uncertainty.

A related mistake is to add the direct and nuisance variances while assuming independence. Unless independence or zero covariance has been justified, the combined variance must include the covariance term.

Key takeaway: A plug-in estimator generally has both direct sampling variation and a first-order nuisance-estimation contribution. Expand around the true nuisance value, retain both terms, and calculate uncertainty from their combined variation.

Check Your Understanding

Use the expansion and assumptions in this tutorial to answer the following questions.

In the plug-in expansion, what does the derivative \(A\) measure, and what does \(\hat\eta-\eta_0\) measure?
For \(\hat\tau=\bar Y/\hat\eta\), with \(\eta_0\ne0\), find the derivative of the target with respect to \(\eta\).
Why can the covariance between the direct term and nuisance contribution be nonzero?
If the nuisance estimate converges more slowly than root-\(n\), what should be checked before using a root-\(n\) approximation for the target?
Under what condition can the covariance term be omitted from the combined asymptotic variance?