Orthogonal And Nonorthogonal Nuisance Parameters

Why Cross-Information Matters

When a model has a target parameter and nuisance parameters, uncertainty about the nuisance quantities can affect how precisely the target can be estimated. Fisher information records this interaction through its cross-information terms. If those terms are zero, the target score is orthogonal to the nuisance scores: at first order, estimating the nuisance parameters does not reduce information about the target. If they are nonzero, some target-score information is shared with the nuisance scores and must be removed when calculating the target’s bound.

The distinction depends on the parameterization, so a nonzero cross-information term does not by itself mean the statistical model contains less information. A change in nuisance coordinates can make the cross-information vanish while leaving the bound for the same target unchanged. The key comparison is between the efficient information and the target’s diagonal information entry, not between diagonal entries from unrelated coordinates.

Definition and First-Order Bound

Definition: Let $\psi$ be a scalar target parameter and let $\lambda\in\mathbb{R}^q$ be a vector of nuisance parameters. For one observation, write the expected Fisher information in blocks as $$ I(\psi,\lambda)= \begin{pmatrix} I_{\psi\psi} & I_{\psi\lambda}\\ I_{\lambda\psi} & I_{\lambda\lambda} \end{pmatrix}. $$ The parameters are information-orthogonal at a parameter value if $I_{\psi\lambda}=0$ there. If $I_{\lambda\lambda}$ is nonsingular, the efficient information for $\psi$, accounting for unknown $\lambda$, is $$ I_{\mathrm{eff}}=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}. $$

Here $I_{\psi\psi}$ is the information about $\psi$ in the $\psi$-score when the nuisance coordinates are held fixed. The subtracted quantity is nonnegative: it is the information in that score component that is linearly explained by the nuisance scores. With $n$ independent, identically distributed observations, the first-order asymptotic variance bound for a regular estimator of $\psi$, on the $\sqrt n$ scale, is $I_{\mathrm{eff}}^{-1}$. Equivalently, its variance at the original scale is bounded to first order by $1/(nI_{\mathrm{eff}})$.

These statements use the usual regular likelihood conditions: differentiability of the model and interchange of differentiation and integration sufficient for the score identities, finite second moments of the scores, and a nonsingular information matrix. In particular, assume $I_{\lambda\lambda}$ and the full information matrix are nonsingular. As in the parametric asymptotic information bound, the variance statement concerns regular estimators with an asymptotic variance; it is not a claim that every estimator has a variance bounded in this way at every fixed parameter value.

Why the Schur Complement Is the Relevant Information

Let $U_\psi$ and $U_\lambda$ denote the score contributions from one observation. The efficient score is what remains of $U_\psi$ after its best linear prediction from the nuisance score has been removed. Its variance is the efficient information.

Result: Under the stated regularity and nonsingularity conditions, the efficient score is $$ U_{\mathrm{eff}}=U_\psi-I_{\psi\lambda}I_{\lambda\lambda}^{-1}U_\lambda, $$ and its variance is $I_{\mathrm{eff}}$. The target entry of the inverse information matrix is $I_{\mathrm{eff}}^{-1}$, giving the first-order bound for $\psi$.

Proof. The score has mean zero, and its covariance blocks are the corresponding Fisher information blocks. Thus $E[U_\psi U_\lambda^{\mathsf T}]=I_{\psi\lambda}$ and $E[U_\lambda U_\lambda^{\mathsf T}]=I_{\lambda\lambda}$. The covariance of the proposed residual with the nuisance score is $$ E[U_{\mathrm{eff}}U_\lambda^{\mathsf T}] =I_{\psi\lambda} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\lambda} =0. $$ Its variance is $$ \begin{aligned} \operatorname{Var}(U_{\mathrm{eff}}) &=I_{\psi\psi} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi} -I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\\ &\quad+ I_{\psi\lambda}I_{\lambda\lambda}^{-1} I_{\lambda\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\\ &=I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi} =I_{\mathrm{eff}}. \end{aligned} $$ To identify the inverse-information entry, solve $I(a,b)^{\mathsf T}=(1,0,\ldots,0)^{\mathsf T}$. The nuisance rows give $I_{\lambda\psi}a+I_{\lambda\lambda}b=0$, hence $b=-I_{\lambda\lambda}^{-1}I_{\lambda\psi}a$. Substitution into the first row gives $I_{\mathrm{eff}}a=1$, so $a=I_{\mathrm{eff}}^{-1}$. The parametric asymptotic information bound therefore has target entry $I_{\mathrm{eff}}^{-1}$. $\square$

When the target and nuisance parameters are orthogonal, $I_{\mathrm{eff}}=I_{\psi\psi}$, so the bound is $1/I_{\psi\psi}$ per observation on the $\sqrt n$ scale. When they are not orthogonal, the correction is positive under nonsingular information, so $I_{\mathrm{eff}}\leq I_{\psi\psi}$ and the variance bound is at least $1/I_{\psi\psi}$. That comparison is for a fixed parameterization: changing coordinates can also change $I_{\psi\psi}$.

Worked Comparisons

Worked Example: Normal Mean and Variance Are Orthogonal

Let $X_1,\ldots,X_n$ be independent with $X_i\sim N(\mu,v)$, where $v>0$ is the variance. Treat $\mu$ as the target and $v$ as a nuisance parameter. For one observation, the log density, up to its displayed normalization, is $\ell(\mu,v)=-\tfrac12\log(2\pi v)-(x-\mu)^2/(2v)$.

The scores are $$ U_\mu=\frac{x-\mu}{v}, \qquad U_v=-\frac{1}{2v}+\frac{(x-\mu)^2}{2v^2}. $$

Using $E[X-\mu]=0$, $E[(X-\mu)^2]=v$, and the centered normal third moment $E[(X-\mu)^3]=0$, the information entries are $$ I_{\mu\mu}=E[U_\mu^2]=\frac{1}{v},\qquad I_{\mu v}=E[U_\mu U_v]=0,\qquad I_{vv}=E[U_v^2]=\frac{1}{2v^2}. $$

Thus the parameters are orthogonal, and $I_{\mathrm{eff}}=1/v$. The asymptotic variance bound for estimating $\mu$ is $v/n$. The sample mean has variance $v/n$, so it attains this bound. In this parameterization, treating $v$ as unknown causes no first-order loss for estimating $\mu$.

Worked Example: A Nonorthogonal Variance Coordinate

Keep the same normal model and the same target $\mu$, but use the nuisance coordinate $\eta=v+a\mu$, where $a$ is a fixed nonzero constant. The model is defined where $v=\eta-a\mu>0$. At fixed $\eta$, changing $\mu$ also changes $v$, so the new target score is $U_\mu-aU_v$, while the nuisance score is $U_\eta=U_v$.

Using the information from the preceding calculation, the information entries in coordinates $(\mu,\eta)$ are $$ I_{\mu\mu}^{\mathrm{new}} =E[(U_\mu-aU_v)^2] =\frac{1}{v}+\frac{a^2}{2v^2}, \qquad I_{\mu\eta}^{\mathrm{new}} =E[(U_\mu-aU_v)U_v] =-\frac{a}{2v^2}, \qquad I_{\eta\eta}^{\mathrm{new}}=\frac{1}{2v^2}. $$

The cross-information is nonzero. Removing its nuisance-score component gives $$ \begin{aligned} I_{\mathrm{eff}} &=I_{\mu\mu}^{\mathrm{new}} -\frac{(I_{\mu\eta}^{\mathrm{new}})^2}{I_{\eta\eta}^{\mathrm{new}}}\\ &=\frac{1}{v}+\frac{a^2}{2v^2} -\frac{a^2/(4v^4)}{1/(2v^2)} =\frac{1}{v}. \end{aligned} $$ The bound remains $v/n$, exactly as before. By contrast, using only $1/I_{\mu\mu}^{\mathrm{new}}$ would give a smaller number than $v$, and would incorrectly ignore the nuisance parameter. The larger diagonal information here is not extra usable information about $\mu$: part of the new $\mu$-score tracks the nuisance score.

Worked Example: Orthogonal Coordinates in a Two-Measurement Model

For each subject, observe independent $X\sim N(\psi+\lambda,1)$ and $Y\sim N(\lambda,1)$. The target is $\psi$; $\lambda$ is nuisance. The scores for one pair are $U_\psi=X-(\psi+\lambda)$ and $U_\lambda=X-(\psi+\lambda)+Y-\lambda$. Their information matrix is $$ I(\psi,\lambda)= \begin{pmatrix} 1&1\\ 1&2 \end{pmatrix}. $$

Consequently, $I_{\mathrm{eff}}=1-1(2^{-1})1=1/2$, and the variance bound for $n$ independent pairs is $2/n$. This bound is attained by $\hat\psi=\bar X-\bar Y$: its expectation is $\psi$, and independence gives $\operatorname{Var}(\bar X-\bar Y)=1/n+1/n=2/n$.

Now let $\eta=\lambda+\psi/2$, keeping $\psi$ as the target. Then $\lambda=\eta-\psi/2$, and the two means become $E[X]=\eta+\psi/2$ and $E[Y]=\eta-\psi/2$. The derivatives of these means with respect to $(\psi,\eta)$ are $(1/2,1)$ for $X$ and $(-1/2,1)$ for $Y$. Since both variances are one and the measurements are independent, the information is $$ I(\psi,\eta)= \begin{pmatrix} (1/2)^2+(-1/2)^2 & (1/2)(1)+(-1/2)(1)\\ (1/2)(1)+(-1/2)(1) & 1^2+1^2 \end{pmatrix} = \begin{pmatrix} 1/2&0\\ 0&2 \end{pmatrix}. $$ The new coordinates are orthogonal, and the target bound is again $1/(n(1/2))=2/n$. The zero cross-information simplifies the calculation; it does not improve the bound for the unchanged target.

Using Orthogonality Correctly

Orthogonality is useful both for interpreting a bound and for choosing coordinates in likelihood calculations. In an orthogonal parameterization, the target’s diagonal information entry is already its efficient information. In a nonorthogonal parameterization, the Schur-complement adjustment is essential. A carefully chosen nuisance coordinate can sometimes eliminate the cross-information locally, but the information bound for a fixed target remains the same when the transformation is a valid smooth, one-to-one change of nuisance coordinates.

The adjustment also explains the geometry: the nuisance score spans directions in the space of score variables. The efficient score is the part of the target score perpendicular to that span. If the target score is already perpendicular, there is nothing to remove; otherwise, its residual has smaller variance than the unadjusted target score.

Common Mistakes

Using $1/I_{\psi\psi}$ when nuisance parameters are unknown. This is generally not the target bound unless the cross-information is zero. Use $1/I_{\mathrm{eff}}$.
Calling every nonzero cross-information term an intrinsic loss. Cross-information depends on coordinates. Compare bounds for the same target, not diagonal entries across parameterizations.
Assuming orthogonality means independence. Zero Fisher cross-information means the scores are uncorrelated at the parameter value; it does not in general imply that the estimators or parameters are independent.
Forgetting the sample-size factor. $I_{\mathrm{eff}}^{-1}$ is the bound for the asymptotic variance of the $\sqrt n$-scaled estimator; the corresponding first-order variance at the original scale is $I_{\mathrm{eff}}^{-1}/n$.

Key takeaway: The target’s first-order variance bound is determined by efficient information, not by the target’s diagonal information alone. Orthogonality makes the efficient-information calculation simpler; reparameterizing nuisance quantities can change cross-information without changing the bound for the same target.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

For a scalar nuisance parameter, write the efficient information in terms of the three distinct entries of the $2\times2$ information matrix.
When does the target’s diagonal information equal its efficient information?
In the normal mean and variance example, why is the cross-information zero?
In the transformed normal example, why is $1/I_{\mu\mu}^{\mathrm{new}}$ not the correct bound when $\eta$ is unknown?
Does changing to orthogonal nuisance coordinates necessarily improve the first-order bound for a target that is unchanged? Explain.