Tutorials › Statistical Theory III - Theories of Inference, Multivariate Methods, and Contingency Tables › Nuisance Parameters in Block Estimating Equations

Home  /  Tutorials  /  M-Estimation and Estimating Equations

M-Estimation and Estimating Equations · Tutorial 605 of 972

Nuisance Parameters in Block Estimating Equations

Learn to isolate a target parameter’s asymptotic variance from a joint estimating system and identify when estimating nuisance parameters changes its precision.

Advanced 12 min read

What You'll Learn

  • Partition a joint estimating equation into target and nuisance components.
  • Derive the target parameter’s asymptotic linear representation using a Schur complement.
  • Calculate its variance from the partitioned Jacobian and score covariance.
  • Distinguish nuisance-parameter effects from the special case of orthogonal equations.
  • Compare target precision when a nuisance parameter is known and when it is estimated.

Why Nuisance Parameters Affect Precision

An estimating equation for a parameter of interest often depends on additional unknown quantities. A target estimate may therefore be obtained not from one equation in isolation, but from a joint system that estimates the target and nuisance parameters together. Even when the target equation is unbiased, estimating the nuisance parameters can change its variability: the fitted nuisance values fluctuate, and those fluctuations can enter the target equation.

The central calculation is a block-matrix version of the sandwich variance. It separates the direct contribution of the target equation from the contribution transmitted through the nuisance equations. The result also clarifies an important qualification: nuisance estimation does not always increase variance for every possible estimating system. A precision penalty is guaranteed in the usual regular likelihood comparison when the nuisance parameter is unknown, but general estimating equations need not obey that comparison.

Definition: Let \(\eta=(\theta^\top,\lambda^\top)^\top\), where \(\theta\) is the target parameter and \(\lambda\) is a nuisance parameter. Suppose \(\eta\) is estimated by solving \(\sum_{i=1}^n \psi(W_i,\eta)=0\), with \(\psi=(\psi_\theta^\top,\psi_\lambda^\top)^\top\). At the true value \(\eta_0\), define the Jacobian \(A=E[\partial\psi(W_i,\eta_0)/\partial\eta^\top]\) and score covariance \(B=\operatorname{Var}(\psi(W_i,\eta_0))\). Partition them conformably as \(A=\begin{pmatrix}A_{\theta\theta}&A_{\theta\lambda}\\A_{\lambda\theta}&A_{\lambda\lambda}\end{pmatrix}\) and \(B=\begin{pmatrix}B_{\theta\theta}&B_{\theta\lambda}\\B_{\lambda\theta}&B_{\lambda\lambda}\end{pmatrix}\).

Here “score covariance” means the covariance of the estimating-function contributions; they need not be likelihood scores. Assume the observations are independent and identically distributed, \(E[\psi(W_i,\eta_0)]=0\), and \(B\) is finite. Also assume a consistent root \(\hat\eta\) exists, the estimating functions are differentiable near \(\eta_0\), and the empirical Jacobian converges to \(A\) with a Taylor remainder negligible at the \(n^{-1/2}\) scale. Take \(A\) to be nonsingular and assume the multivariate central limit theorem applies to \(\psi(W_i,\eta_0)\). These are standard conditions for the asymptotic normality of a Z-estimator.

The Block Derivation

The joint asymptotic linear representation is \(\sqrt n(\hat\eta-\eta_0)=-A^{-1}n^{-1/2}\sum_{i=1}^n\psi(W_i,\eta_0)+o_p(1)\). For the target component, it is more revealing to solve the two linearized equations by elimination. Suppose \(A_{\lambda\lambda}\) is nonsingular and define the Schur complement \(S=A_{\theta\theta}-A_{\theta\lambda}A_{\lambda\lambda}^{-1}A_{\lambda\theta}\). When \(S\) is nonsingular, the target fluctuation is driven by an adjusted estimating function.

Theorem: Under the preceding regularity conditions, if \(A_{\lambda\lambda}\) and \(S\) are nonsingular, then \[ \sqrt n(\hat\theta-\theta_0) = -S^{-1}\frac{1}{\sqrt n}\sum_{i=1}^n \left\{\psi_\theta(W_i,\eta_0) -A_{\theta\lambda}A_{\lambda\lambda}^{-1}\psi_\lambda(W_i,\eta_0)\right\} +o_p(1). \] Consequently, \(\sqrt n(\hat\theta-\theta_0)\xrightarrow{d}N(0,V_\theta)\), where \[ V_\theta = S^{-1}\operatorname{Var}\!\left( \psi_\theta-A_{\theta\lambda}A_{\lambda\lambda}^{-1}\psi_\lambda \right)S^{-\top}. \] The arguments of the variance are evaluated at \((W_i,\eta_0)\), and \(S^{-\top}=(S^{-1})^\top\).

Proof. Write \(Z_{\theta,n}=n^{-1/2}\sum_i\psi_\theta(W_i,\eta_0)\) and \(Z_{\lambda,n}=n^{-1/2}\sum_i\psi_\lambda(W_i,\eta_0)\). Taylor expansion of the two estimating equations gives \[ A_{\theta\theta}\sqrt n(\hat\theta-\theta_0) +A_{\theta\lambda}\sqrt n(\hat\lambda-\lambda_0) =-Z_{\theta,n}+o_p(1), \] \[ A_{\lambda\theta}\sqrt n(\hat\theta-\theta_0) +A_{\lambda\lambda}\sqrt n(\hat\lambda-\lambda_0) =-Z_{\lambda,n}+o_p(1). \] Solve the second equation for the nuisance fluctuation: \[ \sqrt n(\hat\lambda-\lambda_0) =-A_{\lambda\lambda}^{-1}Z_{\lambda,n} -A_{\lambda\lambda}^{-1}A_{\lambda\theta}\sqrt n(\hat\theta-\theta_0) +o_p(1). \] Substitute this expression into the first equation and collect the target terms. The coefficient of \(\sqrt n(\hat\theta-\theta_0)\) is \(S\), so \[ S\sqrt n(\hat\theta-\theta_0) =-Z_{\theta,n}+A_{\theta\lambda}A_{\lambda\lambda}^{-1}Z_{\lambda,n}+o_p(1). \] Multiplication by \(S^{-1}\) yields the stated representation. The multivariate central limit theorem gives joint convergence of \((Z_{\theta,n},Z_{\lambda,n})\) to a mean-zero normal vector with covariance \(B\). Taking the linear combination in the representation and applying Slutsky’s theorem gives the stated covariance. \(\square\)

The coefficient \(A_{\theta\lambda}A_{\lambda\lambda}^{-1}\) describes how nuisance-equation fluctuations enter the target’s effective estimating function. If \(A_{\theta\lambda}=0\), this direct coupling vanishes, but that alone does not establish that the target variance equals the variance obtained with known nuisance parameters: the covariance blocks and target sensitivity must also be considered. The theorem’s variance is exactly the target block of the full sandwich \(A^{-1}BA^{-\top}\), but its adjusted-equation form makes the mechanism explicit.

Worked Calculations

Worked Example: Orthogonal Mean and Variance Equations

Let \(X_i\sim N(\mu,v)\), independently, with \(v>0\). Estimate the target \(\theta=\mu\) and nuisance parameter \(\lambda=v\) from \(\psi_\theta=X-\mu\) and \(\psi_\lambda=(X-\mu)^2-v\). Calculate the target’s asymptotic variance.

The Jacobian entries are expectations of derivatives at \((\mu,v)\): \(A_{\theta\theta}=-1\), \(A_{\theta\lambda}=0\), \(A_{\lambda\theta}=E[-2(X-\mu)]=0\), and \(A_{\lambda\lambda}=-1\). Thus \(S=-1-0=-1\), and the adjustment coefficient is zero. The adjusted target contribution is simply \(X-\mu\).

Since \(E[(X-\mu)^2]=v\), its variance is \(v\). Therefore \(V_\theta=(-1)^{-2}v=v\), so \(\sqrt n(\hat\mu-\mu)\xrightarrow{d}N(0,v)\). Estimating the variance does not affect the first-order precision of the mean here. The zero coupling is supported by the normal distribution’s zero third central moment, which also makes the two estimating-function components uncorrelated.

Worked Example: A Mean Difference with an Estimated Nuisance Mean

Suppose \(X_i=\theta+\lambda+\epsilon_i\) and \(Z_i=\lambda+u_i\), where \(\epsilon_i\sim N(0,4)\), \(u_i\sim N(0,1)\), and the errors are independent of each other and across observations. The target is \(\theta\); \(\lambda\) is nuisance. Use \(\psi_\theta=X-\theta-\lambda\) and \(\psi_\lambda=Z-\lambda\).

The Jacobian is \(A=\begin{pmatrix}-1&-1\\0&-1\end{pmatrix}\). The nuisance block is \(A_{\lambda\lambda}=-1\), and \(S=-1-(-1)(-1)^{-1}(0)=-1\). The adjustment coefficient is \(A_{\theta\lambda}A_{\lambda\lambda}^{-1}=1\), so the adjusted target contribution is \(\psi_\theta-\psi_\lambda=\epsilon-u\). Its variance is \(4+1=5\), because the errors are independent. Since \(S^{-1}=-1\), the theorem gives \(V_\theta=5\).

The equations also show the result directly: \(\hat\lambda=\bar Z\) and \(\hat\theta=\bar X-\bar Z\), so \(\hat\theta-\theta=\bar\epsilon-\bar u\) and \(\operatorname{Var}(\hat\theta)=5/n\). If \(\lambda\) were known, the estimate \(\bar X-\lambda\) would have variance \(4/n\). Here estimating the nuisance parameter adds \(1/n\) to the target variance.

Worked Example: Unknown Second Mean in a Bivariate Normal Model

Let \((X,Y)^\top\sim N((\theta,\lambda)^\top,\Sigma)\), where \(\Sigma=\begin{pmatrix}1&0.6\\0.6&1\end{pmatrix}\) is known and observations are independent. Estimate the target mean \(\theta\) and nuisance mean \(\lambda\) by the likelihood score equations. Compare target variances when \(\lambda\) is unknown and known.

The inverse covariance matrix is \(\Sigma^{-1}=(1/0.64)\begin{pmatrix}1&-0.6\\-0.6&1\end{pmatrix}\). For \(R=(X-\theta,Y-\lambda)^\top\), the score is \(\psi=\Sigma^{-1}R\). The expected score Jacobian is \(A=-\Sigma^{-1}\), and the score covariance is \(B=\Sigma^{-1}\). Hence the joint sandwich covariance is \(A^{-1}BA^{-\top}=\Sigma\). Its target entry is \(V_\theta=1\). Equivalently, the joint maximum likelihood estimate of the mean vector consists of the two sample means, so the variance of \(\hat\theta\) is \(1/n\).

If \(\lambda\) is known, the target score is the first component of \(\Sigma^{-1}R\). Its information is \(I_{\theta\theta}=1/0.64\), giving variance \(I_{\theta\theta}^{-1}/n=0.64/n\). Thus estimating \(\lambda\) raises the target variance from \(0.64/n\) to \(1/n\). The second measurement helps estimate \(\theta\) when its mean is known; when that mean must also be estimated, some of that information is lost.

Interpretation and a Common Pitfall

In regular likelihood problems, the information for the target after accounting for an unknown nuisance parameter is a Schur complement of the full information matrix. The bivariate normal example illustrates the resulting loss relative to the known-nuisance problem. For general estimating equations, however, there is no universal rule that the variance must increase: the equations may not be likelihood-efficient, and their covariance and sensitivity can interact in different ways. The block formula, rather than an automatic “nuisance penalty” claim, determines the answer.

A common mistake is to use \(A_{\theta\theta}^{-1}B_{\theta\theta}A_{\theta\theta}^{-\top}\) as the target variance while ignoring the nuisance equations. That formula treats the target equation as though the nuisance value were fixed and known. When \(A_{\theta\lambda}\ne0\), nuisance fluctuations feed into the target equation; even when that block is zero, the joint covariance structure should be checked before declaring the full variance unchanged. Use the adjusted contribution and Schur complement, or calculate the target block of the full sandwich.

Key takeaway: The target’s first-order fluctuation depends on both the nuisance-adjusted estimating function and its effective sensitivity \(S\). Their combination—not the target equation alone—determines asymptotic precision.

Check Your Understanding

Use the block representation to answer these questions.

  1. What does the block \(A_{\theta\lambda}\) measure in the linearized target equation?
  2. Write the adjusted target estimating-function contribution when \(\theta\) is scalar and \(\lambda\) is vector-valued.
  3. In the mean-difference example, why does estimating \(\lambda\) add variance to \(\hat\theta\)?
  4. Does \(A_{\theta\lambda}=0\) by itself prove that estimating the nuisance parameter has no effect on target variance? Explain.
  5. In the bivariate normal example, why is the target variance smaller when the nuisance mean is known?

Previous

Next

Back to Tutorials
Clinical Biostats