Efficient Information With Nuisance Parameters

Why nuisance parameters change the information bound

A model may contain parameters that are not themselves the focus of an inference but still affect the distribution of the data. These are nuisance parameters. Estimating them can make a target parameter harder to estimate: some variation in the data that appears informative about the target may also be explained by changes in the nuisance parameters.

The Fisher information matrix records this interaction. Its off-diagonal blocks measure how target and nuisance scores co-vary. Accounting for those blocks gives the efficient information for the target: the information remaining after the component associated with nuisance scores is removed.

Setup: Let \(X_1,\ldots,X_n\) be independent and identically distributed observations from a regular parametric model with parameter \(\theta=(\psi^T,\lambda^T)^T\). The target parameter is \(\psi\in\mathbb{R}^p\), and the nuisance parameter is \(\lambda\in\mathbb{R}^q\). For one observation, let \(U_\psi=\partial\ell(\theta;X)/\partial\psi\) and \(U_\lambda=\partial\ell(\theta;X)/\partial\lambda\) be the score vectors. Assume the scores have mean zero and finite second moments, and that the Fisher information per observation is positive definite.

Write the information in blocks, where \(I_{\psi\lambda}=I_{\lambda\psi}^T\):

I(\theta)= \begin{pmatrix} I_{\psi\psi} & I_{\psi\lambda}\\ I_{\lambda\psi} & I_{\lambda\lambda} \end{pmatrix}, \qquad I_{ab}=E[U_aU_b^T].

Positive definiteness of the full information implies that \(I_{\lambda\lambda}\) is invertible. This allows the nuisance-score component of the target score to be identified and removed.

Definition: The efficient score for \(\psi\) is \(U_{\mathrm{eff}}=U_\psi-I_{\psi\lambda}I_{\lambda\lambda}^{-1}U_\lambda\). Its covariance matrix, \(I_{\mathrm{eff}}=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\), is the efficient information for \(\psi\). The expression for \(I_{\mathrm{eff}}\) is the Schur complement of \(I_{\lambda\lambda}\) in \(I(\theta)\).

Why the efficient score has this form

The nuisance score \(U_\lambda\) is a vector of directions in which the likelihood changes when the nuisance parameter changes. The efficient score subtracts from \(U_\psi\) its best linear prediction using those directions. The coefficient is chosen so that the remainder is uncorrelated with every nuisance-score component.

Theorem: Under the setup above, \(E[U_{\mathrm{eff}}]=0\), \(\operatorname{Cov}(U_{\mathrm{eff}},U_\lambda)=0\), and \(\operatorname{Var}(U_{\mathrm{eff}})=I_{\mathrm{eff}}\). If the full information is positive definite, then \(I_{\mathrm{eff}}\) is positive definite.

Proof. The scores have mean zero, so their linear combination \(U_{\mathrm{eff}}\) also has mean zero. For its covariance with the nuisance score, use the block definitions:

\operatorname{Cov}(U_{\mathrm{eff}},U_\lambda) =I_{\psi\lambda} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\lambda} =0.

Expanding the covariance of \(U_{\mathrm{eff}}\), and using \(I_{\lambda\psi}=I_{\psi\lambda}^T\), gives

\begin{aligned} \operatorname{Var}(U_{\mathrm{eff}}) &=I_{\psi\psi} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\\ &\quad +I_{\psi\lambda}I_{\lambda\lambda}^{-1} I_{\lambda\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\\ &=I_{\psi\psi} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi} =I_{\mathrm{eff}}. \end{aligned}

For positive definiteness, take any nonzero vector \(a\in\mathbb{R}^p\) and define \(b=-I_{\lambda\lambda}^{-1}I_{\lambda\psi}a\). The vector \((a^T,b^T)^T\) is nonzero, so positive definiteness of \(I(\theta)\) gives

0< \begin{pmatrix}a\\b\end{pmatrix}^{T} I(\theta) \begin{pmatrix}a\\b\end{pmatrix} =a^T I_{\mathrm{eff}}a.

Thus \(I_{\mathrm{eff}}\) is positive definite. \(\square\)

Efficient information and the asymptotic variance bound

For \(n\) independent observations, information is \(nI(\theta)\), so the corresponding target bound is \(I_{\mathrm{eff}}^{-1}/n\). This is an asymptotic bound for regular estimators, not a claim that every estimator has this variance or that the inequality holds without regularity conditions.

One precise form of the bound uses an asymptotically linear estimator. Suppose an estimator of \(\psi\) has an influence-function representation \(\sqrt{n}(\hat{\psi}-\psi)=n^{-1/2}\sum_{i=1}^n\phi(X_i)+o_p(1)\), where \(E[\phi]=0\), \(E[\phi\phi^T]\) is finite, and the regularity identities \(E[\phi U_\psi^T]=I_p\) and \(E[\phi U_\lambda^T]=0\) hold. These identities express the estimator’s first-order response to changes in the target and nuisance parameters.

Theorem: Under these conditions, \(\operatorname{Var}(\phi)-I_{\mathrm{eff}}^{-1}\) is positive semidefinite. Consequently, the asymptotic covariance matrix of the estimator, on the \(\sqrt n\) scale, is at least \(I_{\mathrm{eff}}^{-1}\) in the positive-semidefinite ordering.

Proof. The regularity identities imply \[ \operatorname{Cov}(\phi,U_{\mathrm{eff}}) =E[\phi U_{\mathrm{eff}}^T] =I_p. \] Since \(\operatorname{Var}(U_{\mathrm{eff}})=I_{\mathrm{eff}}\), define \(R=\phi-I_{\mathrm{eff}}^{-1}U_{\mathrm{eff}}\). Expanding its covariance yields

\begin{aligned} \operatorname{Var}(R) &=\operatorname{Var}(\phi) -\operatorname{Cov}(\phi,U_{\mathrm{eff}})I_{\mathrm{eff}}^{-1} -I_{\mathrm{eff}}^{-1}\operatorname{Cov}(U_{\mathrm{eff}},\phi) +I_{\mathrm{eff}}^{-1}I_{\mathrm{eff}}I_{\mathrm{eff}}^{-1}\\ &=\operatorname{Var}(\phi)-I_{\mathrm{eff}}^{-1}. \end{aligned}

A covariance matrix is positive semidefinite, so the stated difference is positive semidefinite. The asymptotic linear representation has limiting covariance \(\operatorname{Var}(\phi)\), proving the bound. \(\square\)

The same Schur complement also appears in the inverse information matrix:

[I(\theta)^{-1}]_{\psi\psi}=I_{\mathrm{eff}}^{-1}.

To verify this identity, solve \(I(\theta)(x^T,y^T)^T=(v^T,0^T)^T\) for an arbitrary \(v\in\mathbb{R}^p\). The second block equation gives \(y=-I_{\lambda\lambda}^{-1}I_{\lambda\psi}x\). Substitution into the first gives \(I_{\mathrm{eff}}x=v\), so \(x=I_{\mathrm{eff}}^{-1}v\). This is exactly the target block of the inverse. The earlier parametric information bound therefore gives the same target bound.

Worked calculations

Worked Example: A Correlated Bivariate Normal Mean

Let \(X=(X_1,X_2)^T\sim N((\mu,\nu)^T,\Sigma)\), where the covariance matrix is known: \(\Sigma=\begin{pmatrix}4&3\\3&9\end{pmatrix}\). The target is \(\mu\), and \(\nu\) is a nuisance parameter. Calculate the efficient information for one observation.

For a normal model with known covariance, the information for the mean vector is \(\Sigma^{-1}\). Here \(\det(\Sigma)=4(9)-3(3)=27\), and hence

I(\mu,\nu)=\Sigma^{-1} =\frac{1}{27}\begin{pmatrix}9&-3\\-3&4\end{pmatrix} =\begin{pmatrix}1/3&-1/9\\-1/9&4/27\end{pmatrix}.

Thus \(I_{\mu\mu}=1/3\), \(I_{\mu\nu}=-1/9\), and \(I_{\nu\nu}=4/27\). The nuisance-adjusted information is

I_{\mathrm{eff}} =\frac13-\frac{(-1/9)^2}{4/27} =\frac13-\frac1{12} =\frac14.

The asymptotic variance bound for estimating \(\mu\) from \(n\) observations is \(4/n\). If \(\nu\) were known, the information for \(\mu\) would instead be \(1/3\), giving the smaller bound \(3/n\). Treating an unknown nuisance parameter as known would therefore understate the target’s variance bound.

Worked Example: A Logistic Regression Intercept

Suppose \(Y_i\) are independent Bernoulli observations with \(P(Y_i=1)=\operatorname{expit}(\alpha+\beta x_i)\), where \(n\) is even and half the fixed covariates are \(0\), half are \(1\). The target is the intercept \(\alpha\), and the slope \(\beta\) is nuisance. Evaluate the information at \(\alpha=\beta=0\).

For a Bernoulli logistic model, an observation with covariate \(x\) contributes \(w(x)(1,x)^T(1,x)\) to the information, where \(w(x)=p(x)(1-p(x))\). At the stated parameter values, \(p(x)=1/2\) for both covariate values, so \(w(x)=1/4\). Averaging the two equally frequent design points gives the per-observation information

I(\alpha,\beta) =\frac12\left[ \frac14\begin{pmatrix}1&0\\0&0\end{pmatrix} +\frac14\begin{pmatrix}1&1\\1&1\end{pmatrix} \right] =\begin{pmatrix}1/4&1/8\\1/8&1/8\end{pmatrix}.

Therefore, \(I_{\mathrm{eff}}=1/4-(1/8)^2/(1/8)=1/8\), and the asymptotic variance bound for \(\hat{\alpha}\) is \(8/n\). If the slope were known, the bound would be \(1/(nI_{\alpha\alpha})=4/n\). The nonzero cross-information means that changes in the intercept and slope can partly imitate one another in this design.

Worked Example: A Three-Category Multinomial Model

For one categorical observation, let the category probabilities be \(p_1=\psi\), \(p_2=\lambda\), and \(p_3=1-\psi-\lambda\), with all three probabilities positive. The target is \(\psi\), and \(\lambda\) is nuisance. At \(\psi=0.4\), \(\lambda=0.35\), and \(p_3=0.25\), compute the efficient information.

The score is the derivative of the log category probability. For categories 1, 2, and 3, respectively, its \((\psi,\lambda)\) values are \((1/\psi,0)\), \((0,1/\lambda)\), and \((-1/p_3,-1/p_3)\). Taking the probability-weighted outer products gives

I_{\psi\psi}=\frac1\psi+\frac1{p_3},\qquad I_{\lambda\lambda}=\frac1\lambda+\frac1{p_3},\qquad I_{\psi\lambda}=\frac1{p_3}.

At the specified values, these are \(I_{\psi\psi}=5/2+4=13/2\), \(I_{\lambda\lambda}=20/7+4=48/7\), and \(I_{\psi\lambda}=4\). Consequently,

I_{\mathrm{eff}} =\frac{13}{2}-\frac{4^2}{48/7} =\frac{13}{2}-\frac73 =\frac{25}{6}.

For \(n\) independent categorical observations, the target variance bound is \(6/(25n)=0.24/n\). If \(\lambda\) were known, the bound would instead be \(1/(nI_{\psi\psi})=2/(13n)\), which is smaller. The adjustment recovers the information available about the total probability of category 1 when the division of probability between categories 2 and 3 is not fixed.

Common mistake and practical interpretation

A frequent error is to use \(I_{\psi\psi}^{-1}/n\) as the target variance bound even when nuisance parameters are estimated. That quantity is the bound for the target when the nuisance parameter is known. When it is not known, the relevant bound is \(I_{\mathrm{eff}}^{-1}/n\), equivalently the target block of \(I(\theta)^{-1}/n\). In the logistic example, using \(I_{\alpha\alpha}^{-1}/n=4/n\) ignores the unknown slope; the correct bound is \(8/n\).

The efficient information is no larger than the target block \(I_{\psi\psi}\): the subtracted matrix \(I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\) is positive semidefinite. Thus accounting for nuisance parameters cannot increase the available information about the target. In applications, the Schur complement provides both a calculation and an interpretation: it measures what remains of the target score after the nuisance-score directions have been accounted for.

Key takeaway: Partition the Fisher information by target and nuisance parameters. The information relevant to the target is \(I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\), and its inverse, divided by \(n\), is the asymptotic covariance bound for regular estimation of the target.

Check Your Understanding

Use the definitions and derivations above to answer the following.

In words, what component of the target score is removed to form the efficient score?
Why does \(I_{\mathrm{eff}}^{-1}/n\), rather than \(I_{\psi\psi}^{-1}/n\), give the target bound when nuisance parameters are unknown?
If \(I_{\psi\psi}=6\), \(I_{\lambda\lambda}=2\), and \(I_{\psi\lambda}=2\) in a scalar target, scalar nuisance model, calculate \(I_{\mathrm{eff}}\).
What regularity identities for an influence function are used in the asymptotic variance-bound proof?
In the logistic example, why is the bound with an unknown slope larger than the bound with a known slope?