Variance Stabilization With Nuisance Parameters

Why Nuisance Parameters Matter for Stabilization

A variance-stabilizing transformation is often introduced by looking at how an estimator’s asymptotic variance changes with its target parameter. With nuisance parameters, that is not enough: the uncertainty of the target estimator can also depend on parameters that are not themselves of interest. The relevant variance must account for estimating those nuisance parameters, rather than pretending they are known.

This tutorial uses the first-order asymptotic variance of an efficient estimator, such as a regular maximum likelihood estimator. The central question is whether a transformation of the scalar target alone can make that variance constant across the full parameter space, including changes in nuisance parameters.

Definition: Write the parameter as $(\psi,\lambda)$, where $\psi$ is a scalar target and $\lambda$ is a vector of nuisance parameters. Suppose an estimator $\hat{\psi}$ satisfies $$ \sqrt{n}(\hat{\psi}-\psi)\xrightarrow{d}N(0,V(\psi,\lambda)), $$ with $V(\psi,\lambda)>0$. The function $V$ is the estimator’s first-order asymptotic variance function. A differentiable transformation $g(\psi)$, with $g'(\psi)\ne 0$, has first-order asymptotic variance $g'(\psi)^2V(\psi,\lambda)$.

The variance function here is for the target estimated while the nuisance parameter is also estimated. A nuisance parameter that is known in advance can yield a different variance. Throughout, assume a regular, finite-dimensional model with independent and identically distributed observations, a nonsingular Fisher information matrix per observation, and the usual conditions for asymptotic normality of the maximum likelihood estimator and the delta method.

Efficient Variance When Nuisance Parameters Are Estimated

Partition the Fisher information per observation, in coordinates $(\psi,\lambda)$, as

I(\psi,\lambda)= \begin{pmatrix} I_{\psi\psi} & I_{\psi\lambda}\\ I_{\lambda\psi} & I_{\lambda\lambda} \end{pmatrix}.

Assume $I_{\lambda\lambda}$ is invertible. The information about $\psi$ after accounting for estimation of $\lambda$ is the efficient information

I_{\mathrm{eff}}(\psi,\lambda) =I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}.

For a regular maximum likelihood estimator, the first-order asymptotic variance function of $\hat{\psi}$ is $V=I_{\mathrm{eff}}^{-1}$. If $\lambda$ were known, the variance function would instead be $I_{\psi\psi}^{-1}$. The subtraction in efficient information accounts for the information about $\psi$ that is entangled with estimation of the nuisance parameter.

Theorem: Under the regularity conditions above, for a scalar target coordinate $\psi$ and a nuisance vector $\lambda$, the $(\psi,\psi)$ entry of $I^{-1}$ is $$ (I^{-1})_{\psi\psi} =\left(I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\right)^{-1}. $$ Consequently, the asymptotic variance of $\hat{\psi}$ is $I_{\mathrm{eff}}^{-1}/n$.

Solve a block system.
Let the first column of $I^{-1}$ be $(a,b^\mathsf{T})^\mathsf{T}$, where $a$ is scalar. Since $I I^{-1}$ is the identity, this column satisfies $I_{\psi\psi}a+I_{\psi\lambda}b=1$ and $I_{\lambda\psi}a+I_{\lambda\lambda}b=0$.

Eliminate the nuisance block.
The second equation gives $b=-I_{\lambda\lambda}^{-1}I_{\lambda\psi}a$. Substitute this into the first equation to obtain $\left(I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\right)a=1$.

Read off the target variance.
Thus $a=I_{\mathrm{eff}}^{-1}$. The first diagonal entry of $I^{-1}$ is $a$, and the asymptotic covariance matrix of the maximum likelihood estimator is $I^{-1}/n$. This proves the claim. $\square$

Applying the scalar delta method to $g(\hat{\psi})$ gives

\sqrt{n}\{g(\hat{\psi})-g(\psi)\} \xrightarrow{d} N\!\left(0,\;g'(\psi)^2V(\psi,\lambda)\right).

Therefore, if $V$ depends on $\psi$ alone, choosing $g'(\psi)=1/\sqrt{V(\psi)}$ makes the transformed first-order variance equal to $1$. If $V$ also depends on $\lambda$, the multiplier $g'(\psi)^2$ cannot vary with $\lambda$. In general, a target-only transformation cannot remove that dependence.

Worked Examples

Worked Example: A Mean with Unknown Normal Variance

Let $X_1,\ldots,X_n$ be independent $N(\mu,\sigma^2)$ observations, where both parameters are unknown. Take $\mu$ as the target and $v=\sigma^2$ as the nuisance parameter. The Fisher information per observation in coordinates $(\mu,v)$ is

I(\mu,v)= \begin{pmatrix} 1/v & 0\\ 0 & 1/(2v^2) \end{pmatrix}.

The off-diagonal entries are zero, so $I_{\mathrm{eff}}=I_{\mu\mu}=1/v$. The efficient asymptotic variance function for $\hat{\mu}=\bar X$ is $V(\mu,v)=v$, giving $\operatorname{Var}(\hat{\mu})$ approximately $v/n$.

Although $V$ does not depend on $\mu$, it depends on the nuisance parameter $v$. For any differentiable $g(\mu)$, the transformed variance function is $g'(\mu)^2v$. At a fixed $\mu$, this changes with $v$ unless $g'(\mu)=0$, which is excluded for a useful transformation. Thus no nonconstant transformation of the mean alone can make the variance constant over both parameters. The zero information cross-term means estimating $v$ does not inflate the first-order variance of the mean; it does not mean the variance is independent of $v$.

Worked Example: A Variance with Unknown Normal Mean

For the same normal model, now take $v=\sigma^2$ as the target and $\mu$ as the nuisance parameter. In coordinates $(v,\mu)$, the information matrix is

I(v,\mu)= \begin{pmatrix} 1/(2v^2) & 0\\ 0 & 1/v \end{pmatrix}.

Again the cross-information is zero, so $I_{\mathrm{eff}}=1/(2v^2)$, and the efficient variance function for the maximum likelihood estimator $\hat v$ is $V(v,\mu)=2v^2$. Consider $g(v)=\log v$, which is defined for $v>0$. Its derivative is $g'(v)=1/v$, so

g'(v)^2V(v,\mu) =\frac{1}{v^2}\,2v^2 =2.

The transformed estimator therefore has first-order asymptotic variance $2/n$, independent of both the target $v$ and the nuisance mean $\mu$. Here the nuisance parameter contributes no dependence to the variance function, and the target dependence can be removed by a transformation.

Worked Example: A Target Variance That Factors Through a Nuisance Parameter

Suppose $X_1,\ldots,X_n$ are independent observations from $N(\theta,\eta^2\theta^2)$, with $\theta>0$ the target and $\eta>0$ a nuisance parameter. To calculate the information, write the normal mean as $m=\theta$ and variance as $w=\eta^2\theta^2$. For a normal model with differentiable $m,w$, the information entries are

I_{ab}=\frac{m_a m_b}{w}+\frac{w_a w_b}{2w^2},

where subscripts denote derivatives with respect to the parameter coordinates. This identity follows by differentiating the normal log density: the score for coordinate $a$ is $(X-m)m_a/w+\{(X-m)^2-w\}w_a/(2w^2)$. The two terms have covariance zero, since the centered normal third moment is zero, and $\operatorname{Var}\{(X-m)^2\}=2w^2$. Taking score covariances gives the displayed information formula.

Here $m_\theta=1$, $m_\eta=0$, $w_\theta=2\eta^2\theta$, and $w_\eta=2\eta\theta^2$. Substitution gives

I(\theta,\eta)= \begin{pmatrix} (2+1/\eta^2)/\theta^2 & 2/(\theta\eta)\\ 2/(\theta\eta) & 2/\eta^2 \end{pmatrix}.

The efficient information for $\theta$ is

I_{\mathrm{eff}} =\frac{2+1/\eta^2}{\theta^2} -\frac{\{2/(\theta\eta)\}^2}{2/\eta^2} =\frac{2+1/\eta^2}{\theta^2}-\frac{2}{\theta^2} =\frac{1}{\eta^2\theta^2}.

Hence $V(\theta,\eta)=\eta^2\theta^2$. The transformation $g(\theta)=\log\theta$ gives $g'(\theta)=1/\theta$, so the transformed variance function is $\eta^2$. It no longer depends on $\theta$, but it still depends on $\eta$. No target-only transformation can make this variance independent of $\eta$: at fixed $\theta$, it is $g'(\theta)^2\eta^2\theta^2$, which varies with $\eta$ for every nonzero $g'(\theta)$. This is a case where the variance factors as a function of the target times a function of the nuisance parameter, and the target factor can be removed without removing the nuisance factor.

When Can a Target-Only Transformation Stabilize?

The examples distinguish three useful situations. If $V(\psi,\lambda)$ is a function of $\psi$ alone, integrating $g'(\psi)=1/\sqrt{V(\psi)}$ gives a transformation with constant first-order variance. If $V(\psi,\lambda)=A(\psi)B(\lambda)$, the choice $g'(\psi)=1/\sqrt{A(\psi)}$ leaves variance $B(\lambda)$: it stabilizes changes in the target but not changes in the nuisance parameter. If the dependence on $\psi$ and $\lambda$ does not separate in this way, even this partial stabilization may not be possible across the parameter space.

These statements concern first-order asymptotic variance, not exact finite-sample variance or the shape of the estimator’s distribution. A constant first-order variance does not by itself imply normality in a small sample, nor does it guarantee that every other feature of inference is unchanged.

Common Mistake: Treating the Nuisance Parameter as Known

A frequent error is to use $I_{\psi\psi}^{-1}$ as the target variance even when the nuisance parameter is estimated. That is the known-nuisance calculation. With an estimated nuisance parameter, use the Schur complement $I_{\mathrm{eff}}$, or equivalently the $(\psi,\psi)$ entry of the full inverse information matrix.

In the last example, at $\eta=1$, using $I_{\theta\theta}^{-1}$ gives $\theta^2/3$, because $I_{\theta\theta}=3/\theta^2$. The correct efficient variance function is $\eta^2\theta^2=\theta^2$. The discrepancy arises because the information cross-term is nonzero. Ignoring it understates uncertainty and can lead to a mistaken conclusion about whether a transformation stabilizes the target’s variance.

Key takeaway: First compute the efficient asymptotic variance with nuisance parameters estimated. A transformation of the target multiplies that variance by a factor depending only on the target. It can remove target dependence, but nuisance dependence remains unless the variance structure allows it to disappear as well.

Check Your Understanding

Use the efficient variance, rather than the known-nuisance variance, in each answer.

In a model with scalar target $\psi$ and nuisance vector $\lambda$, write the Schur-complement formula for the efficient information about $\psi$.
If $V(\psi,\lambda)=A(\psi)B(\lambda)$, what first-order variance results from choosing $g'(\psi)=1/\sqrt{A(\psi)}$?
Why can no differentiable transformation of the target alone remove nuisance dependence in the normal mean example when the variance is unknown?
In the normal variance example, verify that the logarithm transforms the variance function $2v^2$ into a constant.
Explain why the inverse of the target’s diagonal information entry is generally not the efficient variance when nuisance parameters are estimated.