Worked Nuisance-Parameter Efficiency Calculation

Why Nuisance Parameters Matter in a Correlation Calculation

A correlation is a feature of the joint distribution, but the data used to estimate it also reflect the two means and two marginal variances. If those quantities are unknown, the information available for estimating correlation is not generally the same as it would be if they were fixed and known. The relevant comparison is between the information in the correlation score alone and the information left after accounting for its association with nuisance-parameter scores.

We calculate that comparison for independent observations from a bivariate normal distribution. The calculation illustrates both a useful simplification and an important loss: unknown means are orthogonal to the covariance parameters in this model, whereas unknown marginal variances can reduce the information about correlation.

Model and target: Let \((X_i,Y_i)\), \(i=1,\ldots,n\), be independent and identically distributed bivariate normal observations with means \(\mu_x,\mu_y\), standard deviations \(\sigma_x,\sigma_y>0\), and correlation \(\rho\), where \(-1<\rho<1\). The parameter of interest is \(\rho\). The nuisance parameters are \(\mu_x,\mu_y,\sigma_x,\sigma_y\).

Scores and Information for One Observation

Write \(U=(X-\mu_x)/\sigma_x\), \(V=(Y-\mu_y)/\sigma_y\), \(d=1-\rho^2\), \(\alpha=\log\sigma_x\), and \(\beta=\log\sigma_y\). Using log standard deviations makes the scale scores and information block particularly simple. The log density, up to the displayed terms, is

\ell=-\alpha-\beta-\frac12\log d -\frac{U^2-2\rho UV+V^2}{2d}.

Differentiating with respect to \(\rho,\alpha,\beta\) gives

s_\rho=\frac{\rho}{d} +\frac{(1+\rho^2)UV-\rho(U^2+V^2)}{d^2}, \qquad s_\alpha=-1+\frac{U^2-\rho UV}{d}, \qquad s_\beta=-1+\frac{V^2-\rho UV}{d}.

The mean scores are \(s_{\mu_x}=(U-\rho V)/(\sigma_xd)\) and \(s_{\mu_y}=(V-\rho U)/(\sigma_yd)\). They are linear in the centered normal variables, while the covariance scores are centered quadratic functions. The centered bivariate normal distribution is unchanged by \((U,V)\mapsto(-U,-V)\); consequently, expectations of products of a mean score and a covariance score are zero.

For the covariance scores, the needed moments include \(E[U^2]=E[V^2]=1\), \(E[UV]=\rho\), \(E[U^4]=E[V^4]=3\), \(E[U^3V]=E[UV^3]=3\rho\), and \(E[U^2V^2]=1+2\rho^2\). Substitution into \(E[s_as_b]\) gives the covariance-parameter information matrix per observation:

I_{\rho,\alpha,\beta}= \begin{pmatrix} \dfrac{1+\rho^2}{d^2} & -\dfrac{\rho}{d} & -\dfrac{\rho}{d}\\[6pt] -\dfrac{\rho}{d} & \dfrac{2-\rho^2}{d} & -\dfrac{\rho^2}{d}\\[6pt] -\dfrac{\rho}{d} & -\dfrac{\rho^2}{d} & \dfrac{2-\rho^2}{d} \end{pmatrix}.

For example, the scale-score variances follow by centering \(U^2-\rho UV\), whose mean is \(d\). Its variance is \((2-\rho^2)d\), so \(\operatorname{Var}(s_\alpha)=(2-\rho^2)/d\). The covariance of the centered quantities \(U^2-\rho UV\) and \(V^2-\rho UV\) is \(-\rho^2d\), giving \(E[s_\alpha s_\beta]=-\rho^2/d\). Expanding the correlation score with the same moments gives \(E[s_\rho^2]=(1+\rho^2)/d^2\) and \(E[s_\rho s_\alpha]=E[s_\rho s_\beta]=-\rho/d\).

The mean-parameter information block is

I_{\mu_x,\mu_y} =\frac{1}{d} \begin{pmatrix} 1/\sigma_x^2 & -\rho/(\sigma_x\sigma_y)\\ -\rho/(\sigma_x\sigma_y) & 1/\sigma_y^2 \end{pmatrix}.

It is positive definite for the stated parameter values, and its information cross-block with \((\rho,\alpha,\beta)\) is zero. Thus estimating the means does not change the efficient information for \(\rho\) in this model. The remaining nuisance adjustment is for the two scales.

Efficient Information with Unknown Scales

Definition: For a scalar target parameter and a vector of nuisance parameters, partition the per-observation Fisher information as \(I=\begin{pmatrix}I_{\rho\rho}&I_{\rho\eta}\\I_{\eta\rho}&I_{\eta\eta}\end{pmatrix}\). When \(I_{\eta\eta}\) is nonsingular, the efficient information for the target after accounting for nuisance parameters is the Schur complement \(I_{\mathrm{eff}}=I_{\rho\rho}-I_{\rho\eta}I_{\eta\eta}^{-1}I_{\eta\rho}\).

Here \(\eta=(\alpha,\beta)\). The scale block has eigenvector \((1,1)^\mathsf{T}\) with eigenvalue \(2\): each row of that block sums to \(2\). The vector \(I_{\eta\rho}=-(\rho/d)(1,1)^\mathsf{T}\) lies in precisely that direction. Therefore

I_{\eta\eta}^{-1}I_{\eta\rho} =-\frac{\rho}{2d} \begin{pmatrix}1\\1\end{pmatrix}, \qquad I_{\rho\eta}I_{\eta\eta}^{-1}I_{\eta\rho} =\frac{\rho^2}{d^2}.

Theorem: In the bivariate normal model with \(-1<\rho<1\) and positive marginal standard deviations, the efficient information per observation for \(\rho\), when both means and both variances are unknown, is \(I_{\mathrm{eff}}=1/(1-\rho^2)^2\). The corresponding first-order asymptotic variance bound for a regular estimator based on \(n\) observations is \((1-\rho^2)^2/n\).

Proof. The mean scores have zero information cross-block with the covariance scores, as shown by symmetry of the centered normal distribution. Thus including unknown means adds no term to the Schur-complement adjustment. For the scale nuisance parameters, the preceding matrix calculation gives

I_{\mathrm{eff}} =I_{\rho\rho}-I_{\rho\eta}I_{\eta\eta}^{-1}I_{\eta\rho} =\frac{1+\rho^2}{d^2}-\frac{\rho^2}{d^2} =\frac{1}{d^2}.

The information in \(n\) independent observations is \(nI_{\mathrm{eff}}\). The usual first-order information bound for regular estimators in this smooth, finite-dimensional model is its reciprocal, \(1/(nI_{\mathrm{eff}})=d^2/n\). This is an asymptotic bound, not a claim that every finite-sample estimator has variance at least that amount. \(\square\)

The same projection gives an explicit efficient score. The coefficient for projecting \(s_\rho\) onto the scale scores is \(I_{\rho\eta}I_{\eta\eta}^{-1}=-(\rho/(2d))(1,1)\). Subtracting that linear projection and simplifying yields

s_{\mathrm{eff}} =s_\rho+\frac{\rho}{2d}(s_\alpha+s_\beta) =\frac{UV-\frac{\rho}{2}(U^2+V^2)}{d^2}.

Its expectation is zero because \(E[UV]=\rho\) and \(E[U^2+V^2]=2\). Its variance is \(I_{\mathrm{eff}}\), as also follows from the Schur-complement calculation. The efficient score makes the adjustment concrete: the raw correlation score is modified to remove its component predictable from the scale scores.

Worked Comparisons

Worked Example: Correlation of 0.6 with Unknown Scales

Suppose the true correlation is \(\rho=0.6\), so \(d=1-0.6^2=0.64\). With unknown means and variances, the efficient information per observation is

I_{\mathrm{eff}}=\frac{1}{0.64^2} =\frac{1}{0.4096}=2.44140625.

For \(n=200\), the asymptotic variance bound is \(0.4096/200=0.002048\), and the corresponding standard deviation is \(\sqrt{0.002048}\approx0.04525\).

If the means and variances were known, the information per observation would instead be \(I_{\rho\rho}=(1+\rho^2)/d^2=1.36/0.4096=3.3203125\). Its variance bound for 200 observations is \(d^2/[200(1+\rho^2)]=0.4096/272\approx0.001506\). Knowing the nuisance parameters lowers the bound, as it should: it supplies information that otherwise must be estimated from the data.

Worked Example: Zero Correlation

At \(\rho=0\), \(d=1\). The correlation-score information is \(I_{\rho\rho}=1\), the correlation-scale cross-information entries are both zero, and the scale block is

I_{\eta\eta}= \begin{pmatrix}2&0\\0&2\end{pmatrix}.

The Schur-complement subtraction is therefore zero, so \(I_{\mathrm{eff}}=1\), whether or not the scales are known. For \(n=100\), both variance bounds are \(1/100=0.01\). This equality at zero correlation is a special feature of the information calculation at that parameter value; it does not mean the scales are irrelevant at nonzero correlations.

Worked Example: Evaluating the Efficient Score

Take \(\rho=0.5\) and an observation with standardized values \(U=1\) and \(V=2\). Then \(d=0.75\). The raw correlation score is

s_\rho=\frac{0.5}{0.75} +\frac{1.25(2)-0.5(1+4)}{0.75^2} =\frac{2}{3}.

The scale scores are \(s_\alpha=-1+(1-0.5(2))/0.75=-1\) and \(s_\beta=-1+(4-0.5(2))/0.75=3\). The nuisance-adjusted score is

s_{\mathrm{eff}} =s_\rho+\frac{0.5}{2(0.75)}(s_\alpha+s_\beta) =\frac23+\frac13(2)=\frac43.

Checking with the simplified formula gives \((2-0.25(1+4))/0.75^2=0.75/0.5625=4/3\), the same result. At this parameter value, the unknown-scale variance bound is \(0.75^2/n=0.5625/n\); if all nuisance parameters are known, it is \(0.75^2/[1.25n]=0.45/n\).

Interpreting the Cost of Unknown Nuisance Parameters

When the means and scales are known, the information per observation for \(\rho\) is \((1+\rho^2)/(1-\rho^2)^2\). When the means and scales are unknown, it is \(1/(1-\rho^2)^2\). Thus the known-nuisance information exceeds the unknown-nuisance information by \(\rho^2/(1-\rho^2)^2\), which is nonnegative and is strictly positive when \(\rho\ne0\). Equivalently, the ratio of the unknown-nuisance variance bound to the known-nuisance variance bound is \(1+\rho^2\).

This comparison also explains why merely using the raw score information \(I_{\rho\rho}\) would give the wrong bound when the scales are unknown: it ignores the score components shared with the nuisance parameters. The Schur complement removes those components. Estimating the means causes no additional first-order loss here because their scores are orthogonal to the covariance scores; that conclusion depends on the model and should not be assumed in other settings.

Common Mistake

Wrong reasoning: “The correlation score has information \((1+\rho^2)/(1-\rho^2)^2\), so that is the information available even when the marginal variances are estimated.” This treats the nuisance parameters as known.
Correction: With unknown nuisance parameters, subtract \(I_{\rho\eta}I_{\eta\eta}^{-1}I_{\eta\rho}\). Here the subtraction is \(\rho^2/(1-\rho^2)^2\), leaving efficient information \(1/(1-\rho^2)^2\).

Key takeaway: In the bivariate normal model, unknown means do not reduce first-order information about correlation, but unknown marginal variances generally do. The efficient information is the correlation-score information minus the nuisance-score contribution, not the correlation-score entry by itself.

Check Your Understanding

Use the model and notation above to answer the following.

At \(\rho=0.8\), calculate the efficient information per observation when the means and variances are unknown.
At the same correlation, what is the ratio of the unknown-nuisance variance bound to the known-nuisance variance bound?
Why does the mean-score block not enter the Schur-complement adjustment for correlation in this model?
At \(\rho=0\), what are the correlation-scale cross-information entries, and what consequence do they have for the efficient information?
In one sentence, explain why using \(I_{\rho\rho}\) alone gives an incorrect bound when the scales are unknown and \(\rho\ne0\).