Tutorials › Econometrics (Graduate) › Efficient Information and Nuisance Adjustment

Home  /  Tutorials  /  Likelihood and Statistical Inference

Likelihood and Statistical Inference · Tutorial 65 of 952

Efficient Information and Nuisance Adjustment

Learn to obtain a target parameter’s efficient information and asymptotic variance when nuisance parameters are estimated jointly.

Intermediate 9 min read

What You'll Learn

  • Partition Fisher information into target and nuisance parameter blocks
  • Derive the efficient-score adjustment and its Schur-complement formula
  • Connect efficient information to the target block of the inverse information matrix
  • Calculate the asymptotic variance adjustment in a two-parameter likelihood
  • Identify when nuisance estimation strictly reduces information and when it does not

Why Nuisance Parameters Change Precision

A likelihood model may contain a parameter of direct interest and other parameters that must also be estimated. Those additional parameters are called nuisance parameters: they matter for the probability model even when the scientific question concerns a different parameter. Their estimation can affect how precisely the target parameter is determined. The key quantity is not just the target’s information considered by itself, but the information that remains after accounting for overlap between target and nuisance scores.

This tutorial derives that adjustment using the Schur complement of the Fisher information matrix. The result quantifies the asymptotic variance cost of estimating nuisance parameters jointly, while also showing why that cost can be zero when the target and nuisance scores are uncorrelated.

Partitioning the Information Matrix

Let \(\theta=(\psi',\lambda')'\), where \(\psi\) is the target parameter and \(\lambda\) is a nuisance parameter vector. Initially, take \(\psi\) to be scalar. For an independent and identically distributed sample of size \(n\), let \(s_i(\theta)\) be the score vector for observation \(i\). Under the regularity conditions for likelihood inference, the per-observation Fisher information is \(I(\theta)=E[s_i(\theta)s_i(\theta)']\), equivalently the expected negative Hessian for one observation. Partition it at the true parameter value \(\theta_0\) as follows:

$$ I(\theta_0)= \begin{pmatrix} I_{\psi\psi} & I_{\psi\lambda}\\ I_{\lambda\psi} & I_{\lambda\lambda} \end{pmatrix}. $$

Here \(I_{\psi\psi}\) is scalar, \(I_{\lambda\lambda}\) is a square matrix, and \(I_{\psi\lambda}=I_{\lambda\psi}'\). The sample’s expected information is \(nI(\theta_0)\). Assume the model is locally identified and regular, the information matrix is finite and positive definite, and the nuisance block \(I_{\lambda\lambda}\) is invertible. These conditions make the information-based variance calculations well-defined. The usual asymptotic normality result for the maximum likelihood estimator (MLE) then gives covariance approximately \(I(\theta_0)^{-1}/n\) for the full parameter vector.

Definition: The efficient information for \(\psi\) in the presence of nuisance parameter \(\lambda\) is the information remaining about \(\psi\) after accounting for the part of its score that can be linearly explained by the nuisance score. For scalar \(\psi\), it is the Schur complement \(I_{\psi\psi\cdot\lambda}=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\).

If \(\lambda\) were known, the per-observation information for \(\psi\) would be \(I_{\psi\psi}\), giving asymptotic variance \(1/(nI_{\psi\psi})\). When \(\lambda\) is estimated, the corresponding variance is \(1/(nI_{\psi\psi\cdot\lambda})\). The difference between these information quantities is determined by the cross-information blocks.

Deriving the Nuisance Adjustment

Write the one-observation score as \(s_i=(s_{\psi i},s_{\lambda i}')'\). Its covariance blocks are the corresponding information blocks. The linear projection of the scalar target score onto the nuisance score is \(I_{\psi\lambda}I_{\lambda\lambda}^{-1}s_{\lambda i}\): its coefficient is chosen so that the residual has zero covariance with the nuisance score. Subtracting this projection gives the efficient score.

$$ s_{\mathrm{eff},i} = s_{\psi i}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}s_{\lambda i}. $$

To see its information, put \(A=I_{\psi\lambda}I_{\lambda\lambda}^{-1}\). Since the information matrix is symmetric, the covariance of the residual is \(\operatorname{Var}(s_{\psi i}-As_{\lambda i})\). Expanding the covariance gives the following complete calculation:

$$ \begin{aligned} \operatorname{Var}(s_{\mathrm{eff},i}) &= I_{\psi\psi}-A I_{\lambda\psi} - I_{\psi\lambda}A' + A I_{\lambda\lambda}A'\\ &= I_{\psi\psi} - I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi} - I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi} + I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\\ &= I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}. \end{aligned} $$

In the second line, symmetry gives \(A'=I_{\lambda\lambda}^{-1}I_{\lambda\psi}\); the last product in the first line therefore equals \(I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\). The resulting variance is precisely the efficient information defined above. The construction removes from the target score the component that carries no additional information about the target once the nuisance score is taken into account.

From Efficient Information to Asymptotic Variance

The same adjustment appears in the inverse information matrix. To see why, solve the block system \(Iu=v\) for the first component of \(u\), where \(v=(v_\psi',v_\lambda')'\) is partitioned conformably. The second block equation gives \(u_\lambda=I_{\lambda\lambda}^{-1}(v_\lambda-I_{\lambda\psi}u_\psi)\). Substituting this into the first block equation yields \((I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi})u_\psi =v_\psi-I_{\psi\lambda}I_{\lambda\lambda}^{-1}v_\lambda\). Thus, the target-target block of \(I^{-1}\) is the reciprocal of the scalar Schur complement.

Theorem: Under the regularity and nonsingularity conditions stated above, the asymptotic variance of the jointly estimated scalar target is \(\operatorname{Avar}(\hat\psi)=1/(nI_{\psi\psi\cdot\lambda})\). If \(\lambda\) were known, it would instead be \(1/(nI_{\psi\psi})\). Because \(I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\geq 0\), estimating nuisance parameters cannot increase target information or decrease the asymptotic variance. For scalar \(\psi\), the information reduction is strict when \(I_{\psi\lambda}\ne 0\); it is zero when the cross-information is zero.

For a vector target \(\psi\), the same result holds with matrix blocks: the efficient information is \(I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\), and the target block of the inverse information is the inverse of this Schur complement. The information removed is positive semidefinite, so the adjustment cannot improve precision in any target direction.

Worked Example: Computing an Adjustment From Information Blocks

Suppose a hypothetical two-parameter model has per-observation information at its true parameter \(I=\begin{pmatrix}10&3\\3&4\end{pmatrix}\), with scalar target \(\psi\) and scalar nuisance parameter \(\lambda\). The matrix is positive definite: its leading diagonal entry is 10 and its determinant is \(10(4)-3^2=31>0\). Thus the stated information blocks are nonsingular and suitable for this calculation.

The target-only information is \(I_{\psi\psi}=10\). The nuisance block is \(I_{\lambda\lambda}=4\), and the cross-information is \(I_{\psi\lambda}=I_{\lambda\psi}=3\). Therefore,

$$ I_{\psi\psi\cdot\lambda} =10-3(4^{-1})3 =10-\frac{9}{4} =7.75. $$

With \(n=100\) independent observations, the asymptotic variance if \(\lambda\) were known would be \(1/(100\cdot10)=0.001000\). When \(\lambda\) is estimated jointly, it is \(1/(100\cdot7.75)=0.001290\) to four significant figures. The increase is due to nonzero cross-information, not a change in the target parameter’s units. In this hypothetical model the nuisance adjustment lowers target information from 10 to 7.75 per observation.

A Two-Parameter Likelihood Calculation

A simple Gaussian model shows how the formula arises from a likelihood. For each of \(n\) independent hypothetical observation pairs, suppose \(X_i=\psi+\lambda+\varepsilon_i\) and \(Y_i=\lambda+u_i\), where \(\varepsilon_i\) and \(u_i\) are independent \(N(0,\sigma^2)\) errors, and all errors are independent across pairs. Treat \(\sigma^2\) as known. The parameters \(\psi\) and \(\lambda\) are unrestricted real numbers. The two means identify them separately: the mean of \(Y_i\) is \(\lambda\), and the mean of \(X_i\) minus the mean of \(Y_i\) is \(\psi\). The normal model is regular, and its information is positive definite, as verified below.

For one pair, the log likelihood, up to terms not involving the parameters, is \(-[(X_i-\psi-\lambda)^2+(Y_i-\lambda)^2]/(2\sigma^2)\). Differentiating with respect to the parameters gives scores \(s_{\psi i}=(X_i-\psi-\lambda)/\sigma^2\) and \(s_{\lambda i}=[(X_i-\psi-\lambda)+(Y_i-\lambda)]/\sigma^2\). The error variances and independence therefore give the per-pair information

$$ I(\psi,\lambda) =\frac{1}{\sigma^2} \begin{pmatrix} 1&1\\ 1&2 \end{pmatrix}. $$

Its determinant is \(1/\sigma^4>0\), and its first diagonal entry is positive, so it is positive definite. The Schur complement for \(\psi\) is

$$ I_{\psi\psi\cdot\lambda} =\frac{1}{\sigma^2} -\frac{(1/\sigma^2)^2}{2/\sigma^2} =\frac{1}{2\sigma^2}. $$

For a concrete hypothetical sample, take \(n=100\), \(\sigma=2\) measurement units, \(\bar{x}=12.4\) units, and \(\bar{y}=5.1\) units. The likelihood estimates are \(\hat\lambda=\bar{y}=5.1\) units and \(\hat\psi=\bar{x}-\bar{y}=12.4-5.1=7.3\) units. The adjusted information per pair is \(1/(2\cdot 2^2)=0.1250\) per squared unit, so total information is \(100(0.1250)=12.50\) per squared unit. The asymptotic variance for estimating \(\psi\) is \(1/12.50=0.08000\) squared units, with standard error \(\sqrt{0.08000}=0.2828\) units.

If \(\lambda\) were known, the target information per pair would be \(1/\sigma^2=0.2500\), and the variance for \(n=100\) would be \(1/(100\cdot0.2500)=0.04000\) squared units. The jointly estimated variance is twice as large here because the target and nuisance scores share information. In this Gaussian example, the MLE \(\bar{x}-\bar{y}\) has variance \(2\sigma^2/n\) exactly, which agrees with the information calculation.

When There Is No Information Loss

A nonzero nuisance parameter does not automatically reduce target information. The relevant condition is nonzero cross-information. If the target and nuisance scores are orthogonal in the information sense, their cross-information block is zero, so projecting the target score onto the nuisance score removes nothing.

Worked Example: Zero Cross-Information

Consider a hypothetical two-parameter model with per-observation information \(I=\begin{pmatrix}10&0\\0&5\end{pmatrix}\). This matrix is positive definite because both diagonal entries are positive. For target \(\psi\), nuisance parameter \(\lambda\), and \(n=50\), the Schur complement is

$$ I_{\psi\psi\cdot\lambda} =10-0(5^{-1})0 =10. $$

If \(\lambda\) were known, the target variance would be \(1/(50\cdot10)=0.002000\). If it is estimated jointly, the asymptotic variance is also \(1/(50\cdot10)=0.002000\). The nuisance parameter still has to be estimated, but its score is uncorrelated with the target score, so it causes no information reduction for this target in this model.

Common Pitfall and Practical Interpretation

A common mistake is to say that positive definiteness alone means estimating a nuisance parameter must strictly reduce target information. Positive definiteness ensures nonsingularity, but it does not require nonzero cross-information: the diagonal matrix in the example above is positive definite and has no target-information adjustment. The correct statement is that nuisance estimation cannot increase target information or decrease its asymptotic variance; the reduction is strict for a scalar target when the cross-information block is nonzero.

In practice, calculate the Schur complement from the information blocks at the parameter value of interest, then invert it and divide by \(n\) to obtain the scalar target’s asymptotic variance. Check that the nuisance information block is invertible and that the full information is positive definite. The adjustment describes asymptotic precision under the specified likelihood and regularity conditions; it does not by itself establish causal interpretation, identification outside the model, or good finite-sample performance.

Check Your Understanding

Answer each question using the information-block formulas and the stated assumptions.

  1. For \(I=\begin{pmatrix}8&2\\2&5\end{pmatrix}\), calculate the efficient information for the first parameter when the second is nuisance.
  2. For a sample of size \(n\), write the asymptotic variance of a scalar target when its efficient information per observation is \(I_{\psi\psi\cdot\lambda}\).
  3. Why does zero cross-information imply that estimating the nuisance parameter does not alter the target’s asymptotic information in the two-parameter scalar-target case?
  4. State the conditions under which the nuisance adjustment is strictly positive for a scalar target.
  5. Explain why positive definiteness by itself does not imply a strict information reduction.

Previous

Next

Back to Tutorials
Clinical Biostats