Efficient Information With Gaussian Nuisance Parameters

Why nuisance parameters change precision

A Gaussian likelihood may involve parameters that are not themselves the scientific target. For example, a mean component may be of interest while another mean component must be estimated as a nuisance parameter. Although the nuisance parameter is not the quantity to report, estimating it from the same data can affect how precisely the target is estimated. The relevant information is therefore not generally the target’s diagonal block of the full Fisher information matrix.

The adjustment has a simple matrix form: a Schur complement. It subtracts the part of the target information that is entangled with estimation of the nuisance parameters. The resulting efficient information determines the asymptotic covariance for estimating the target when the nuisance parameters are unknown.

Partitioning the information matrix

Definition: Write the parameter as $\theta=(\psi^\mathsf{T},\lambda^\mathsf{T})^\mathsf{T}$, where $\psi\in\mathbb{R}^q$ is the target parameter and $\lambda\in\mathbb{R}^r$ is the nuisance parameter. Partition the Fisher information for one observation as $$ I(\theta)= \begin{pmatrix} I_{\psi\psi} & I_{\psi\lambda}\\ I_{\lambda\psi} & I_{\lambda\lambda} \end{pmatrix}. $$ If $I_{\lambda\lambda}$ is nonsingular, the efficient information for $\psi$, with $\lambda$ unknown, is $$ I_{\mathrm{eff}}(\psi)=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}. $$ This is the Schur complement of $I_{\lambda\lambda}$ in $I(\theta)$.

For a sample of $n$ independent observations from the same model, the Fisher information is $nI(\theta)$, so the efficient information for the full sample is $nI_{\mathrm{eff}}(\psi)$. The formula uses the information at the true parameter value, or at a specified parameter value when discussing the model’s local information.

The block $I_{\psi\psi}$ describes information about the target when the nuisance parameter is treated as known. The subtracted term accounts for the loss from having to estimate the nuisance parameter too. In a regular identifiable model, the full information is positive definite; then $I_{\lambda\lambda}$ and its Schur complement are also positive definite.

Why the Schur complement is the efficient information

Let $U_\psi$ and $U_\lambda$ be the score vectors for one observation, written as columns. Under the usual regularity conditions for Fisher information, the scores have mean zero and their covariance blocks are the corresponding information blocks. In particular, $E[U_\psi U_\lambda^\mathsf{T}]=I_{\psi\lambda}$ and $E[U_\lambda U_\lambda^\mathsf{T}]=I_{\lambda\lambda}$.

Theorem: Suppose the score has finite second moments and $I_{\lambda\lambda}$ is positive definite. The residual after linearly projecting the target score onto the nuisance score is $$ U_{\mathrm{eff}}=U_\psi-I_{\psi\lambda}I_{\lambda\lambda}^{-1}U_\lambda. $$ Its covariance is $I_{\mathrm{eff}}(\psi)$.

Proof. For a matrix $A$, the covariance of $U_\psi-AU_\lambda$ is $$ I_{\psi\psi}-AI_{\lambda\psi}-I_{\psi\lambda}A^\mathsf{T} +AI_{\lambda\lambda}A^\mathsf{T}. $$ The projection coefficient must make the residual uncorrelated with the nuisance score. This requires $$ E[(U_\psi-AU_\lambda)U_\lambda^\mathsf{T}] =I_{\psi\lambda}-AI_{\lambda\lambda}=0, $$ so, because $I_{\lambda\lambda}$ is invertible, $A=I_{\psi\lambda}I_{\lambda\lambda}^{-1}$. Substituting this value into the covariance gives $$ \operatorname{Cov}(U_{\mathrm{eff}}) =I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}. $$ Thus removing the component of the target score explained linearly by the nuisance score leaves precisely the efficient information. $\square$

This projection argument also shows why the adjustment is nonnegative in information terms. When the full information matrix is symmetric, $I_{\lambda\psi}=I_{\psi\lambda}^\mathsf{T}$, and $I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}$ is positive semidefinite. Consequently $I_{\mathrm{eff}}(\psi)$ cannot exceed $I_{\psi\psi}$ in the positive-semidefinite ordering.

Efficient information and asymptotic precision

Suppose observations are independent and identically distributed from a regular parametric model, the true parameter is interior, the Fisher information is finite and positive definite, and the maximum likelihood estimator is consistent and asymptotically normal under the model’s usual differentiability and identifiability conditions. The earlier material on asymptotic covariance from Gaussian information gives the full-parameter covariance $I(\theta)^{-1}/n$. The target block of that covariance has a useful expression.

Theorem: Under the preceding conditions, if $\hat\theta=(\hat\psi^\mathsf{T},\hat\lambda^\mathsf{T})^\mathsf{T}$ is the joint maximum likelihood estimator, then $$ \sqrt{n}(\hat\psi-\psi)\xrightarrow{d} N_q\!\left(0,I_{\mathrm{eff}}(\psi)^{-1}\right). $$ Equivalently, its asymptotic covariance is $I_{\mathrm{eff}}(\psi)^{-1}/n$.

Proof. Write the inverse of the full information matrix in blocks, and let $B$ be its upper-left block. The first block row of $I(\theta)I(\theta)^{-1}$ must equal $(I_q,0)$. If $C$ is the lower-left block of the inverse, this gives $$ I_{\psi\psi}B+I_{\psi\lambda}C=I_q, \qquad I_{\lambda\psi}B+I_{\lambda\lambda}C=0. $$ The second equation yields $C=-I_{\lambda\lambda}^{-1}I_{\lambda\psi}B$. Substitution in the first gives $$ \left(I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\right)B=I_q, $$ so $B=I_{\mathrm{eff}}(\psi)^{-1}$. Asymptotic normality of the joint estimator gives limiting covariance $I(\theta)^{-1}$ for the scaled estimator; the target subvector therefore has the upper-left block $B$ as its limiting covariance. This proves the result. $\square$

If $\lambda$ were known, the target information would instead be $I_{\psi\psi}$, with corresponding asymptotic covariance $I_{\psi\psi}^{-1}/n$. Unknown nuisance parameters therefore generally increase target uncertainty. The increase can be zero when the target and nuisance scores are orthogonal, meaning $I_{\psi\lambda}=0$.

Worked Gaussian examples

Worked Example: A Target Mean With a Nuisance Mean

Let $X=(X_1,X_2)^\mathsf{T}\sim N_2(\mu,\Sigma)$, with known covariance $$ \Sigma=\begin{pmatrix}4&3\\3&9\end{pmatrix}, $$ and unknown mean $\mu=(\mu_1,\mu_2)^\mathsf{T}$. Estimate $\mu_1$, treating $\mu_2$ as a nuisance parameter. For a Gaussian mean with known covariance, the information for one observation is $\Sigma^{-1}$. Since $\det(\Sigma)=4(9)-3(3)=27$, $$ I(\mu)=\Sigma^{-1} =\frac{1}{27}\begin{pmatrix}9&-3\\-3&4\end{pmatrix} =\begin{pmatrix}1/3&-1/9\\-1/9&4/27\end{pmatrix}. $$

Here $I_{\psi\psi}=1/3$, $I_{\psi\lambda}=-1/9$, and $I_{\lambda\lambda}=4/27$. The efficient information for $\mu_1$ is $$ I_{\mathrm{eff}}(\mu_1) =\frac{1}{3}-\left(-\frac{1}{9}\right) \left(\frac{27}{4}\right)\left(-\frac{1}{9}\right) =\frac{1}{3}-\frac{1}{12} =\frac{1}{4}. $$ For $n$ observations, the asymptotic variance is $1/(n/4)=4/n$. If $\mu_2$ were known, the variance bound would be $1/(n/3)=3/n$. Estimating $\mu_2$ costs information about $\mu_1$, because the two mean scores are correlated.

Worked Example: A Regression Slope With a Nuisance Intercept

Consider independent observations $Y_i=\beta_0+\beta_1x_i+\varepsilon_i$, with fixed design values $x=(0,1,3)$ and $\varepsilon_i\sim N(0,2)$. The target is the slope $\beta_1$; the intercept $\beta_0$ is nuisance. The information for $(\beta_0,\beta_1)$ is $X^\mathsf{T}X/2$, where the two columns of $X$ are the intercept column and the $x$-column. Since $n=3$, $\sum x_i=4$, and $\sum x_i^2=10$, $$ I(\beta_0,\beta_1) =\frac{1}{2}\begin{pmatrix}3&4\\4&10\end{pmatrix} =\begin{pmatrix}3/2&2\\2&5\end{pmatrix}. $$

Thus the slope block is $5$, the cross-information is $2$, and the intercept information is $3/2$. The efficient slope information is $$ I_{\mathrm{eff}}(\beta_1) =5-2\left(\frac{2}{3/2}\right) =5-\frac{8}{3} =\frac{7}{3}. $$ The resulting variance bound for this design is $3/7$. As a check, centering the design values gives $\bar{x}=4/3$ and $$ \frac{1}{2}\sum_{i=1}^3(x_i-\bar{x})^2 =\frac{1}{2}\left(\frac{16}{9}+\frac{1}{9}+\frac{25}{9}\right) =\frac{7}{3}. $$ The intercept absorbs information associated with a common shift in all fitted values; slope information comes from variation in the design around its mean.

Worked Example: Unknown Gaussian Variance Causes No Mean-Information Loss

Let $X_1,\ldots,X_n$ be independent $N(\mu,v)$ observations, where both the target mean $\mu$ and nuisance variance $v>0$ are unknown. For one observation $X$, write $Z=X-\mu$. The scores are $$ U_\mu=\frac{Z}{v}, \qquad U_v=-\frac{1}{2v}+\frac{Z^2}{2v^2}. $$ Because a centered Gaussian has $E[Z]=E[Z^3]=0$, the cross-information is $$ I_{\mu v}=E[U_\mu U_v] =-\frac{E[Z]}{2v^2}+\frac{E[Z^3]}{2v^3}=0. $$ Also $I_{\mu\mu}=E[Z^2]/v^2=1/v$. Therefore $$ I_{\mathrm{eff}}(\mu)=I_{\mu\mu}-I_{\mu v}I_{vv}^{-1}I_{v\mu} =\frac{1}{v}. $$ For $n$ observations the efficient information is $n/v$, giving asymptotic variance $v/n$ for the mean estimator. Here estimating the variance does not reduce mean information because the mean and variance scores are orthogonal.

Common pitfall: using the target block alone

A common mistake is to read $I_{\psi\psi}$ from the full information matrix and report $I_{\psi\psi}^{-1}/n$ as the target’s covariance even though the nuisance parameter is being estimated. That calculation corresponds to treating the nuisance parameter as known. When the cross-information is nonzero, the correct information is $I_{\mathrm{eff}}(\psi)$, and its inverse gives the target block of the joint estimator’s asymptotic covariance.

The correction is not to subtract a cross-information block by itself: the adjustment is the full product $I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}$. Its dimensions are $q\times q$, matching $I_{\psi\psi}$, and it accounts for both the relationship between target and nuisance scores and the amount of information available about the nuisance parameter. When the cross-information is zero, there is no penalty; otherwise the efficient information is no greater than the information with the nuisance parameter known.

Key takeaway: With unknown nuisance parameters, target precision is governed by the Schur complement of the nuisance-information block. In regular models, the inverse of this efficient information is the target block of the joint estimator’s asymptotic covariance.

Check Your Understanding

Use the information partition and the distinction between known and unknown nuisance parameters to answer these questions.

For a scalar target and scalar nuisance parameter, write the efficient information in terms of the three distinct information blocks.
Why is the covariance of the efficient score equal to the Schur complement?
If $I_{\psi\lambda}=0$, how does the asymptotic covariance compare with the case where the nuisance parameter is known?
In the regression example, what feature of the design values determines the slope information after accounting for the intercept?
Explain why $I_{\psi\psi}^{-1}/n$ is generally not the correct asymptotic covariance for the target when nuisance parameters are estimated jointly.