Why Joint Estimation Changes a Variance
An interval for a parameter of interest is often reported even though the fitted model also contains parameters that are not the focus of the question. These nuisance parameters still affect the target estimate: a change in a nuisance estimate may be partly offset by a change in the target estimate, or may add uncertainty to it. The relevant standard error therefore comes from the joint estimation problem, not from pretending the nuisance parameters were known.
The central bookkeeping rule is simple: if a joint estimator has an asymptotic covariance matrix, use the block corresponding to the target component. In a regular likelihood model, that block comes from the inverse of the full Fisher information matrix. It is generally not the inverse of the target’s information entry alone.
For a scalar target, \(\Sigma_{\psi\psi}\) is a single number. For a vector target, it is a matrix that describes the joint uncertainty among the target components. If an estimator is asymptotically normal with a sandwich covariance rather than an inverse-information covariance, the same rule applies: take the target block of that full covariance matrix.
The Information-Matrix Adjustment
For a regular, correctly specified likelihood model, let \(I(\theta)\) denote the per-observation Fisher information matrix. Under standard maximum-likelihood regularity conditions, including an interior true parameter, identifiability, differentiability, and nonsingular information, the maximum likelihood estimator satisfies \(\sqrt{n}(\hat{\theta}-\theta)\xrightarrow{d}N(0,I(\theta)^{-1})\). Thus, the target variance is the target block of \(I(\theta)^{-1}/n\).
The block-inverse identity makes the adjustment explicit. For simplicity, take \(\psi\) to be scalar and write the information matrix as \(I=\begin{pmatrix}I_{\psi\psi}&I_{\psi\lambda}\\I_{\lambda\psi}&I_{\lambda\lambda}\end{pmatrix}\). Assume \(I\) is positive definite. Then \(I_{\lambda\lambda}\) is invertible, and the target entry of \(I^{-1}\) is the inverse of a Schur complement.
To prove the formula, solve \(I\binom{x}{y}=\binom{1}{0}\), where \(x\) is scalar and \(y\) has the dimension of \(\lambda\). The first block equation gives \(I_{\psi\psi}x+I_{\psi\lambda}y=1\); the second gives \(I_{\lambda\psi}x+I_{\lambda\lambda}y=0\). Since \(I_{\lambda\lambda}\) is invertible, the second equation yields \(y=-I_{\lambda\lambda}^{-1}I_{\lambda\psi}x\). Substituting into the first gives \[ \left(I_{\psi\psi}-I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\right)x=1. \] The solution \(x\) is the target entry of \(I^{-1}\), which proves the identity. Positive definiteness ensures the scalar Schur complement is positive, so the reciprocal is defined. \(\square\)
If \(\psi\) were estimated with \(\lambda\) known, its information would be \(I_{\psi\psi}\), giving variance \(1/(nI_{\psi\psi})\). With jointly estimated nuisance parameters, the subtracted term \(I_{\psi\lambda}I_{\lambda\lambda}^{-1}I_{\lambda\psi}\) is nonnegative. It is zero when the target and nuisance parameters are information-orthogonal, but otherwise the effective information is smaller and the target variance larger.
Worked Examples
Worked Example: A Gaussian Model With Two Unknown Means
For each of \(n\) independent subjects, observe independent variables \(U_i\sim N(\psi+\lambda,1)\) and \(V_i\sim N(\lambda,1)\). The target is \(\psi\), and \(\lambda\) is a nuisance parameter. Find the asymptotic variance of the joint maximum likelihood estimator of \(\psi\), and compare it with the variance if \(\lambda\) were known.
For one subject, the log-likelihood, apart from a constant, is \(-\tfrac12(U-\psi-\lambda)^2-\tfrac12(V-\lambda)^2\). Its expected information can be obtained from the derivatives of the two means. The derivative vectors with respect to \((\psi,\lambda)\) are \((1,1)\) for the mean of \(U\) and \((0,1)\) for the mean of \(V\). Since both variances are one, the information is \[ I= \begin{pmatrix}1&1\\1&2\end{pmatrix}. \]
The nuisance-adjusted information for \(\psi\) is \(1-1(2^{-1})1=1/2\). Therefore \(\operatorname{Var}(\hat{\psi})\) is asymptotically \(2/n\). The same result follows directly: the likelihood estimates are \(\hat{\lambda}=\bar V\) and \(\hat{\psi}=\bar U-\bar V\), so independence gives \(\operatorname{Var}(\hat{\psi})=1/n+1/n=2/n\).
If \(\lambda\) were known, the estimator \(\bar U-\lambda\) would have variance \(1/n\). Estimating \(\lambda\) from the data adds uncertainty; treating it as fixed would miss the contribution from \(\bar V\).
Worked Example: Slope With an Unknown Regression Intercept
Suppose \(Y_i=\alpha+\beta x_i+\varepsilon_i\), with independent \(\varepsilon_i\sim N(0,4)\). The target is the slope \(\beta\), and the intercept \(\alpha\) is nuisance. There are six observations: two at each of \(x=0,1,2\). Calculate the asymptotic variance of the slope estimator and compare it with the calculation that would assume the intercept known.
Here \(n=6\), \(\sum_i x_i=6\), and \(\sum_i x_i^2=10\). The information for \((\alpha,\beta)\) is \[ I=\frac{1}{4} \begin{pmatrix}6&6\\6&10\end{pmatrix}. \] The slope’s nuisance-adjusted information is its diagonal entry minus the intercept adjustment: \[ I_{\beta\beta}-I_{\beta\alpha}I_{\alpha\alpha}^{-1}I_{\alpha\beta} =\frac{10}{4}-\frac{6}{4}\left(\frac{4}{6}\right)\frac{6}{4} =1. \] Thus the slope variance is asymptotically \(1\). Equivalently, the usual regression expression is \(4/S_{xx}\), where \(S_{xx}=\sum_i(x_i-\bar{x})^2=10-6^2/6=4\), giving \(4/4=1\).
If the intercept were known, the slope information would instead be \(\sum_i x_i^2/4=10/4\), giving variance \(4/10=0.4\). The centered spread \(S_{xx}\), rather than the uncentered sum of squares, reflects the loss of slope information when the intercept must also be estimated.
Worked Example: A Logistic Slope With an Unknown Intercept
In a logistic regression, let \(\operatorname{logit}(P(Y=1\mid x))=\alpha+\beta x\). Observe 100 independent outcomes at \(x=0\) and 100 at \(x=1\). At the parameter values considered, let \(p_0=0.2\) and \(p_1=0.5\). Find the asymptotic variance of the slope estimator when the intercept is estimated jointly.
For a Bernoulli observation with success probability \(p\), the information weight for its linear predictor is \(p(1-p)\). Thus \(w_0=0.2(0.8)=0.16\) and \(w_1=0.5(0.5)=0.25\). The information matrix for \((\alpha,\beta)\) is \[ I= \begin{pmatrix} 100w_0+100w_1&100w_1\\ 100w_1&100w_1 \end{pmatrix} = \begin{pmatrix}41&25\\25&25\end{pmatrix}. \]
The adjusted information for \(\beta\) is \(25-25^2/41=400/41\). Consequently, \(\operatorname{Var}(\hat{\beta})\) is asymptotically \(41/400=0.1025\), and its standard error is approximately \(\sqrt{0.1025}=0.320\). If the intercept were fixed, the variance would be \(1/25=0.04\), because that calculation uses only the \(x=1\) group’s slope information. It understates uncertainty when the baseline probability must also be estimated.
Using the Variance in an Interval
In practice, the true information is unknown. Estimate it at the fitted parameter, or use a consistent estimate of the joint asymptotic covariance when a sandwich variance is appropriate. Then extract the target block. For a scalar target, if \(\widehat V_{\psi}\) estimates the per-observation asymptotic variance of \(\hat{\psi}\), a two-sided large-sample interval is \[ \hat{\psi}\ \pm\ z_{1-\alpha/2}\sqrt{\widehat V_{\psi}/n}. \]
The coverage justification is a direct application of the large-sample normal approximation. Suppose \(\sqrt{n}(\hat{\psi}-\psi)\xrightarrow{d}N(0,V_{\psi})\) and \(\widehat V_{\psi}\xrightarrow{p}V_{\psi}>0\). By Slutsky’s theorem, \[ \frac{\sqrt{n}(\hat{\psi}-\psi)}{\sqrt{\widehat V_{\psi}}} \xrightarrow{d}N(0,1). \] Therefore, for \(z_{1-\alpha/2}\) the standard normal quantile, \[ P\left( -z_{1-\alpha/2}\le \frac{\sqrt{n}(\hat{\psi}-\psi)}{\sqrt{\widehat V_{\psi}}} \le z_{1-\alpha/2} \right)\longrightarrow 1-\alpha. \] Rearranging the event gives the stated interval and its first-order coverage. This argument requires consistency of the variance estimate as well as asymptotic normality of the estimator.
Common Mistake: Inverting Only the Target Entry
A tempting calculation is to read \(I_{\psi\psi}\) from the information matrix and report \(1/(nI_{\psi\psi})\). That is the variance calculation for a model in which the nuisance parameters are known, not the usual joint-estimation problem. The correction is to invert the full information matrix and then extract the target block, or equivalently use the Schur-complement formula.
There is also a useful exception: if \(I_{\psi\lambda}=0\), then the target and nuisance parameters are information-orthogonal, and the adjustment term vanishes. The target variance then agrees, to first order, with the known-nuisance calculation. Orthogonality should be checked from the model; it should not be assumed simply because the target and nuisance parameters have different names.
Check Your Understanding
Use the full joint covariance or information structure in each answer.
- For a scalar target and vector nuisance parameter, state the Schur-complement expression for the target block of the inverse information matrix.
- In the Gaussian two-mean example, why is the variance larger when \(\lambda\) is estimated than when it is known?
- For a regression slope with an unknown intercept, explain why the relevant design quantity is \(\sum_i(x_i-\bar{x})^2\) rather than \(\sum_i x_i^2\).
- Under what information-matrix condition does estimating the nuisance parameter have no first-order effect on the target variance?
- What consistency condition on the estimated variance is needed, in addition to asymptotic normality, to justify the plug-in Wald interval?