Nuisance Dispersion and Joint Information

Why Dispersion Belongs in the Information Matrix

In a generalized linear model, regression coefficients describe how the mean changes with the predictors, while a dispersion parameter describes variability around that mean. When dispersion is unknown, it is estimated from the same data. Likelihood inference should therefore consider the coefficients and dispersion jointly, rather than treating the dispersion estimate as if it were fixed.

The key calculation is a partition of the joint information matrix. In a general model, estimating a nuisance parameter can reduce the information available for the coefficients. In a regular exponential-dispersion GLM, however, the expected coefficient–dispersion information is zero. This orthogonality means that the usual coefficient covariance block is unchanged at the Fisher-information level, even though dispersion remains an estimated parameter.

Definition: Let $\beta\in\mathbb{R}^p$ be the regression-coefficient vector and let $\phi$ be a scalar dispersion parameter. The joint parameter is $\theta=(\beta^\mathsf{T},\phi)^\mathsf{T}$. Its expected information is partitioned as $$ I(\theta)= \begin{pmatrix} I_{\beta\beta} & I_{\beta\phi}\\ I_{\phi\beta} & I_{\phi\phi} \end{pmatrix}, $$ where $I_{\beta\beta}$ is the coefficient block, $I_{\phi\phi}$ is the dispersion block, and $I_{\beta\phi}=I_{\phi\beta}^{\mathsf T}$ is the cross-information block. Dispersion is a nuisance parameter when the inferential target is $\beta$, not $\phi$.

The Nuisance-Adjusted Covariance Block

Suppose the joint expected information is nonsingular, and suppose $I_{\phi\phi}$ is nonsingular. The coefficient block of the inverse joint information is not generally just $I_{\beta\beta}^{-1}$. The cross-information must first be accounted for.

Theorem: Define the Schur complement of $I_{\phi\phi}$ in the joint information matrix by $$ S_\beta=I_{\beta\beta}-I_{\beta\phi}I_{\phi\phi}^{-1}I_{\phi\beta}. $$ If $S_\beta$ is nonsingular, the coefficient block of $I(\theta)^{-1}$ is $S_\beta^{-1}$. Under the usual regularity conditions for joint maximum-likelihood asymptotics—an interior true parameter, an identifiable model, differentiable likelihood, and nonsingular information—this is the first-order asymptotic covariance block for $\hat\beta$.

The block formula follows directly by solving a linear system. Write $A=I_{\beta\beta}$, $B=I_{\beta\phi}$, and $C=I_{\phi\phi}$. To find the first block column of $I(\theta)^{-1}$, solve $$ \begin{pmatrix}A&B\\B^\mathsf{T}&C\end{pmatrix} \begin{pmatrix}U\\V\end{pmatrix} = \begin{pmatrix}I_p\\0\end{pmatrix}. $$ The second block row gives $B^\mathsf{T}U+CV=0$, hence $V=-C^{-1}B^\mathsf{T}U$. Substituting this expression into the first block row gives $$ (A-BC^{-1}B^\mathsf{T})U=I_p. $$ Therefore $U=(A-BC^{-1}B^\mathsf{T})^{-1}=S_\beta^{-1}$, as claimed. $\square$

This result explains the direction of the adjustment. When the joint information matrix is positive definite, $BC^{-1}B^\mathsf{T}$ is positive semidefinite: for any vector $a$, $$ a^\mathsf{T}BC^{-1}B^\mathsf{T}a =(B^\mathsf{T}a)^\mathsf{T}C^{-1}(B^\mathsf{T}a)\geq 0. $$ Thus $S_\beta$ is no larger than $A$ in the positive-semidefinite ordering, and accounting for nuisance uncertainty cannot increase coefficient information. If the cross block is zero, the adjustment vanishes.

Worked Example: Reading the Nuisance Adjustment

Consider a two-parameter likelihood with coefficient $\beta$, nuisance dispersion $\phi$, and joint expected information $$ I= \begin{pmatrix} 10&2\\ 2&4 \end{pmatrix}. $$ The matrix is positive definite because its first diagonal entry is positive and its determinant is $10(4)-2^2=36>0$.

Here $I_{\beta\beta}=10$, $I_{\beta\phi}=2$, and $I_{\phi\phi}=4$. The nuisance-adjusted information for $\beta$ is $$ S_\beta=10-2(4^{-1})2=10-1=9. $$ Consequently, the coefficient variance approximation is $S_\beta^{-1}=1/9$. If dispersion were incorrectly treated as known, the variance approximation would instead be $I_{\beta\beta}^{-1}=1/10$, which is too small. Direct inversion confirms the calculation: $$ I^{-1}=\frac{1}{36} \begin{pmatrix} 4&-2\\ -2&10 \end{pmatrix}, $$ whose coefficient entry is $4/36=1/9$.

This is a general information-matrix illustration: a nonzero cross block makes the nuisance adjustment positive. In the usual regular GLM parameterization with free dispersion, the expected cross block has a special property, derived next.

Expected Information in an Exponential-Dispersion GLM

For the orthogonality calculation, assume independent observations with fixed design vectors $x_i$, and a regular exponential-dispersion model with log likelihood $$ \ell(\beta,\phi)=\sum_{i=1}^n\left\{ \frac{w_i}{\phi}\big[y_i\vartheta_i-b(\vartheta_i)\big] +c(y_i,\phi,w_i)\right\}. $$ Here $w_i>0$ are fixed weights, $\vartheta_i$ is the canonical parameter determined by the mean model, $E[Y_i]=\mu_i=b'(\vartheta_i)$, and $\operatorname{Var}(Y_i)=\phi V(\mu_i)/w_i$. Assume the model is differentiable in both parameter blocks, the relevant expectations and derivatives exist, and the score has mean zero. The dispersion parameter must be identifiable and interior to its parameter space. Some GLMs, including the usual Bernoulli and Poisson models, have dispersion fixed by the family and do not estimate a free $\phi$.

Theorem: Under the stated exponential-dispersion model and regularity conditions, the expected coefficient–dispersion information is zero: $$ I_{\beta\phi}=0. $$ Thus, when the expected information blocks are nonsingular, the coefficient block of the inverse joint information is $I_{\beta\beta}^{-1}$.

To prove the result, write $q_i=d\vartheta_i/d\eta_i$, where $\eta_i=x_i^\mathsf{T}\beta$. Differentiating the log likelihood with respect to $\beta$ gives $$ U_\beta=\frac{\partial\ell}{\partial\beta} =\frac{1}{\phi}\sum_{i=1}^n w_i(y_i-\mu_i)q_i x_i. $$ The sum apart from $1/\phi$ does not depend on $\phi$. Therefore $$ \frac{\partial^2\ell}{\partial\beta\,\partial\phi} =-\frac{1}{\phi}U_\beta. $$ Expected information is the negative expected Hessian under the regularity conditions, so $$ I_{\beta\phi} =-E\left[\frac{\partial^2\ell}{\partial\beta\,\partial\phi}\right] =\frac{1}{\phi}E[U_\beta]=0. $$ The final equality uses the zero-mean score condition. Applying the block formula with $I_{\beta\phi}=0$ gives $S_\beta=I_{\beta\beta}$, and hence the stated covariance block. $\square$

For this model, the coefficient information can also be written explicitly. Since $q_i=(d\mu_i/d\eta_i)/V(\mu_i)$, independence and the variance formula yield $$ I_{\beta\beta} =\sum_{i=1}^n \frac{w_i}{\phi V(\mu_i)} \left(\frac{d\mu_i}{d\eta_i}\right)^2x_ix_i^\mathsf{T}. $$ This is expected information. The observed mixed derivative need not be zero for a particular data set: the proof establishes that its expectation is zero, not that each realized value vanishes.

Worked Example: Gaussian GLM With an Unknown Variance

Let $Y_i$ be independent with $Y_i\sim N(x_i^\mathsf{T}\beta,\phi)$, where $\phi>0$ is the unknown variance. Take an intercept and a slope, with predictor values $x=(-1,0,1)$, so the design matrix has rows $(1,-1),(1,0),(1,1)$. Then $$ X^\mathsf{T}X= \begin{pmatrix} 3&0\\ 0&2 \end{pmatrix}. $$

The log likelihood, up to a constant not depending on the parameters, is $$ \ell(\beta,\phi)=-\frac{3}{2}\log\phi-\frac{1}{2\phi}(y-X\beta)^\mathsf{T}(y-X\beta). $$ The coefficient information is $I_{\beta\beta}=X^\mathsf{T}X/\phi$, so at $\phi=4$, $$ I_{\beta\beta}= \begin{pmatrix} 3/4&0\\ 0&1/2 \end{pmatrix}. $$ The dispersion information is $I_{\phi\phi}=3/(2\phi^2)=3/32$, and the expected cross block is zero. The inverse joint information therefore has coefficient block $$ I_{\beta\beta}^{-1}= \begin{pmatrix} 4/3&0\\ 0&2 \end{pmatrix}. $$

Thus the approximate standard errors are $\sqrt{4/3}$ for the intercept and $\sqrt{2}$ for the slope. The calculation accounts for dispersion as an estimated nuisance parameter; orthogonality explains why the coefficient block is still the inverse of $X^\mathsf{T}X/\phi$.

Examples of Estimation and Observed Information

Worked Example: Fitting an Intercept With Unknown Dispersion

Suppose $Y_1,\ldots,Y_4$ are independent $N(\beta,\phi)$ observations, and the observed values are $2,3,5,6$. The maximum-likelihood estimate of the mean is $$ \hat\beta=\bar y=\frac{2+3+5+6}{4}=4. $$ The residuals are $-2,-1,1,2$, so their squared sum is $4+1+1+4=10$. The Gaussian maximum-likelihood estimate of the variance divides by $n$, giving $\hat\phi=10/4=2.5$.

For this intercept-only design, $I_{\beta\beta}=n/\phi$, $I_{\phi\phi}=n/(2\phi^2)$, and $I_{\beta\phi}=0$. Evaluating the coefficient covariance approximation at the estimates gives $$ \widehat{\operatorname{Var}}(\hat\beta)=\frac{\hat\phi}{n} =\frac{2.5}{4}=0.625, \qquad \widehat{\operatorname{SE}}(\hat\beta)=\sqrt{0.625}\approx0.791. $$ The unknown variance is estimated from the residuals, but its expected-information cross block with the mean is zero. This example also illustrates why the variance maximum-likelihood estimate uses $n$, not $n-1$; the latter is the unbiased sample-variance denominator, not the likelihood maximizer.

Worked Example: A Nonzero Observed Cross Block

Use the three-point Gaussian regression design with predictor values $-1,0,1$ and responses $1,2,4$. At the trial coefficient values $\beta=(2,1)^\mathsf{T}$, the fitted values are $1,2,3$, and the residual vector is $r=(0,0,1)^\mathsf{T}$. Therefore $$ X^\mathsf{T}r= \begin{pmatrix}1\\1\end{pmatrix}. $$ For Gaussian regression, the coefficient score is $U_\beta=X^\mathsf{T}r/\phi$. Differentiating it with respect to $\phi$ shows that the observed information cross block, which is the negative mixed second derivative, is $$ J_{\beta\phi}=-\frac{\partial^2\ell}{\partial\beta\,\partial\phi} =\frac{X^\mathsf{T}r}{\phi^2}. $$ At $\phi=2$, this is $(1/4,1/4)^\mathsf{T}$, not zero.

There is no contradiction with the orthogonality result: the realized residual vector need not make $X^\mathsf{T}r$ zero at arbitrary trial coefficients. At the maximum-likelihood coefficient estimate, the least-squares score equation gives $X^\mathsf{T}r=0$, so this observed cross block is zero in this Gaussian example. In general, the theorem concerns expected information and does not require observed cross-information to vanish away from the fitted estimate.

Interpretation and a Common Pitfall

The joint-information calculation is useful because it separates two questions: how much information the data provide about $\beta$ when $\phi$ is fixed, and how much remains when $\phi$ must also be estimated. In a general parameterization, the Schur complement answers the second question. In the regular exponential-dispersion GLM setup above, expected orthogonality makes the nuisance adjustment zero, so estimating dispersion does not change the first-order coefficient covariance block.

Common mistake: “Dispersion is estimated, so I must always subtract a nuisance adjustment from the coefficient information.” That reasoning applies only when the cross-information is nonzero. The correction is to inspect the joint expected information: calculate $I_{\beta\phi}$, then use the Schur complement. For a regular exponential-dispersion GLM with the stated parameterization, the cross block is zero. Conversely, it is also incorrect to claim that observed mixed information must be zero merely because expected mixed information is zero.

Key takeaway: The coefficient covariance block from joint likelihood inference is the inverse Schur complement $ (I_{\beta\beta}-I_{\beta\phi}I_{\phi\phi}^{-1}I_{\phi\beta})^{-1}$. In a regular GLM with free exponential-family dispersion, expected coefficient–dispersion information is zero, reducing this to $I_{\beta\beta}^{-1}$.

Check Your Understanding

Answer each question by identifying the relevant information blocks and stating the conditions you use.

Write the Schur complement that determines the coefficient covariance block when the nuisance parameter is scalar dispersion.
For the information matrix $\begin{pmatrix}12&3\\3&6\end{pmatrix}$, calculate the nuisance-adjusted coefficient information and its inverse.
Why does a zero expected coefficient–dispersion information block not imply that the observed mixed derivative is zero for every data set?
In Gaussian regression with fixed design matrix $X$ and variance $\phi$, state the coefficient and dispersion expected information blocks.
Why is the usual Bernoulli GLM not an example of estimating a free dispersion parameter in the standard model?