Expected Information from the GLM Mean

Why the Mean Derivative Matters

The expected information for GLM coefficients depends on two features of each observation: how variable its response is, and how strongly its mean changes when its linear predictor changes. A response with high variance generally gives less information about the coefficients. A mean that changes rapidly with the linear predictor can, in contrast, make the response more sensitive to coefficient changes.

These two effects combine in the diagonal matrix $W$ in the expression $X'WX$. The expression is not a new score equation: it is the expected information matrix that determines the curvature of the likelihood on average. We derive it from the response variance and the derivative of the mean, then calculate it for three different GLMs.

Model Setup and Definition

Assume $Y_1,\ldots,Y_n$ are independent responses with fixed design vectors $\mathbf{x}_i$, and let $\mathbf{X}$ be the design matrix whose $i$th row is $\mathbf{x}_i'$. The coefficient vector is $\boldsymbol\beta$, and the linear predictor and mean for observation $i$ are

\eta_i=\mathbf{x}_i'\boldsymbol\beta,\qquad \mu_i=E[Y_i]=g^{-1}(\eta_i).

Here $g$ is a differentiable link, and we write $\mu_{\eta,i}=d\mu_i/d\eta_i$. Assume the responses have variance $\operatorname{Var}(Y_i)=\phi V(\mu_i)$, where $\phi>0$ is fixed for the coefficient calculation and $V(\mu_i)>0$. We use the exponential-dispersion likelihood parameterization

\ell_i(\theta_i,\phi) = \frac{y_i\theta_i-b(\theta_i)}{\phi} +c(y_i,\phi), \qquad \mu_i=b'(\theta_i),\quad V(\mu_i)=b''(\theta_i).

The response distribution is assumed to have a support that does not depend on $\boldsymbol\beta$. For the information identities below, assume differentiation under the integral or sum defining the likelihood is valid through second derivatives. These regularity conditions hold for the usual regular GLM likelihoods in their parameter interiors.

Definition: The expected Fisher information for $\boldsymbol\beta$ is $I_{\boldsymbol\beta}=E[UU']$, where $U=\partial\ell/\partial\boldsymbol\beta$ is the score vector. Under the regularity conditions stated above, it is also $I_{\boldsymbol\beta}=-E[\partial^2\ell/\partial\boldsymbol\beta\partial\boldsymbol\beta']$. In a GLM, it has the form $X'WX$, where $W$ is diagonal and its $i$th diagonal entry is $w_i=\mu_{\eta,i}^2/\operatorname{Var}(Y_i)=\mu_{\eta,i}^2/\{\phi V(\mu_i)\}$.

Deriving the Information from the Mean

The exponential-dispersion form gives the derivative of the individual log-likelihood with respect to the natural parameter as $(y_i-\mu_i)/\phi$. Also, differentiating $\mu_i=b'(\theta_i)$ with respect to $\theta_i$ gives $d\mu_i/d\theta_i=V(\mu_i)$. By the chain rule, $d\theta_i/d\eta_i=\mu_{\eta,i}/V(\mu_i)$. Thus the score contribution with respect to $\eta_i$ is

\frac{\partial\ell_i}{\partial\eta_i} = \frac{y_i-\mu_i}{\phi}\frac{\mu_{\eta,i}}{V(\mu_i)} = \frac{(y_i-\mu_i)\mu_{\eta,i}}{\operatorname{Var}(Y_i)}.

Since $\partial\eta_i/\partial\boldsymbol\beta=\mathbf{x}_i$, the coefficient score is

U = \sum_{i=1}^n \mathbf{x}_i \frac{(Y_i-\mu_i)\mu_{\eta,i}}{\operatorname{Var}(Y_i)}.

Theorem: For independent GLM responses under the stated model and regularity conditions, $$ I_{\boldsymbol\beta} = \sum_{i=1}^n \mathbf{x}_i\mathbf{x}_i' \frac{\mu_{\eta,i}^2}{\operatorname{Var}(Y_i)} = X'WX,\qquad W=\operatorname{diag}(w_1,\ldots,w_n), \quad w_i=\frac{\mu_{\eta,i}^2}{\phi V(\mu_i)}. $$

Proof. Write the score as $U=\sum_i\mathbf{x}_i A_i$, where $A_i=(Y_i-\mu_i)\mu_{\eta,i}/\operatorname{Var}(Y_i)$. The means and other factors in $A_i$ are fixed when taking expectation at the current parameter value. Because $E[Y_i-\mu_i]=0$, $E[A_i]=0$. Independence then implies $\operatorname{Cov}(A_i,A_j)=0$ for $i\ne j$. Therefore

E[UU'] = \sum_{i=1}^n \mathbf{x}_i\mathbf{x}_i'E[A_i^2] = \sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i' \frac{\mu_{\eta,i}^2 E[(Y_i-\mu_i)^2]} {\operatorname{Var}(Y_i)^2} = \sum_{i=1}^n\mathbf{x}_i\mathbf{x}_i' \frac{\mu_{\eta,i}^2}{\operatorname{Var}(Y_i)}.

The last equality uses $E[(Y_i-\mu_i)^2]=\operatorname{Var}(Y_i)$. Substituting $\operatorname{Var}(Y_i)=\phi V(\mu_i)$ gives the stated $w_i$. Finally, the $i$th summand is $w_i\mathbf{x}_i\mathbf{x}_i'$; summing these outer products is exactly $X'WX$. $\square$

Why This Is Also Expected Negative Curvature

The definition using $E[UU']$ agrees with the expected negative Hessian under regularity conditions. This matters because the Hessian describes local likelihood curvature, while the score-covariance derivation makes the roles of response variance and mean sensitivity especially visible.

Result: If the support of the response distribution does not depend on $\boldsymbol\beta$, and the likelihood can be differentiated twice under its normalizing integral or sum, then $$ E\!\left[\frac{\partial^2\ell}{\partial\boldsymbol\beta\partial\boldsymbol\beta'}\right] =-E[UU']. $$

Proof. Let $f(\mathbf y;\boldsymbol\beta)$ be the joint density or mass function and let $U=\partial\log f/\partial\boldsymbol\beta$. Differentiating $\int f(\mathbf y;\boldsymbol\beta)\,d\mathbf y=1$ once gives $E[U]=0$. For coordinates $j,k$, the product and chain rules give

\frac{\partial^2 f}{\partial\beta_j\partial\beta_k} = f\left( \frac{\partial^2\ell}{\partial\beta_j\partial\beta_k} + \frac{\partial\ell}{\partial\beta_j} \frac{\partial\ell}{\partial\beta_k} \right).

Integrating the left side gives zero, because it is the second derivative of the normalized integral. Integrating the right side therefore gives $E[\partial^2\ell/(\partial\beta_j\partial\beta_k)]+E[U_jU_k]=0$. This holds for every pair of coordinates, proving the matrix identity. $\square$

Worked Examples

Worked Example: Bernoulli Responses with a Logit Link

Suppose $Y_i$ is Bernoulli with success probability $p_i$, and use the logit link. Then $\mu_i=p_i$, $\operatorname{Var}(Y_i)=p_i(1-p_i)$, and $dp_i/d\eta_i=p_i(1-p_i)$. Hence

w_i=\frac{\{p_i(1-p_i)\}^2}{p_i(1-p_i)} =p_i(1-p_i).

For a numerical calculation, take $\mathbf{x}_i'=(1,x_i)$ with $x_i=0,1,2$, and evaluate at $\boldsymbol\beta'=(-\log 4,\log 4)$. The linear predictors are $-\log 4,0,\log 4$, so the probabilities are $1/5,1/2,4/5$. The weights are $4/25,1/4,4/25$. Therefore

X'WX = \begin{pmatrix} \frac{4}{25}+\frac14+\frac{4}{25} & 0\frac{4}{25}+1\frac14+2\frac{4}{25} \\ 0\frac{4}{25}+1\frac14+2\frac{4}{25} & 0^2\frac{4}{25}+1^2\frac14+2^2\frac{4}{25} \end{pmatrix} = \begin{pmatrix} 0.57&0.57\\ 0.57&0.89 \end{pmatrix}.

The diagonal weights are largest at $p_i=1/2$ here; probabilities close to zero or one yield smaller weights for the same design vector.

Worked Example: A Bernoulli Response with a Probit Link

For a probit model, $p_i=\Phi(\eta_i)$, where $\Phi$ is the standard normal distribution function and $\varphi$ its density. The derivative is $dp_i/d\eta_i=\varphi(\eta_i)$, while the Bernoulli variance remains $p_i(1-p_i)$. Thus

w_i=\frac{\varphi(\eta_i)^2}{\Phi(\eta_i)\{1-\Phi(\eta_i)\}}.

Consider one observation with design row $(1,2)$ at $\eta_i=0$. Since $\Phi(0)=1/2$ and $\varphi(0)=1/\sqrt{2\pi}$, its weight is

w_i = \frac{1/(2\pi)}{(1/2)(1/2)} = \frac{2}{\pi}.

The resulting two-coefficient information contribution is

w_i\mathbf{x}_i\mathbf{x}_i' = \frac{2}{\pi} \begin{pmatrix}1\\2\end{pmatrix} \begin{pmatrix}1&2\end{pmatrix} = \frac{2}{\pi} \begin{pmatrix}1&2\\2&4\end{pmatrix}.

A single observation contributes a rank-one matrix, even though it informs a model with two coefficients. Information from additional design rows is needed to obtain full rank.

Worked Example: Poisson Responses with a Log Link

For a Poisson response with mean $\mu_i=e^{\eta_i}$, the variance is $\mu_i$ and $d\mu_i/d\eta_i=\mu_i$. Consequently,

w_i=\frac{\mu_i^2}{\mu_i}=\mu_i.

Take two design rows $(1,0)$ and $(1,1)$, with fitted means $2$ and $3$. These are produced, for example, by $\boldsymbol\beta'=(\log 2,\log(3/2))$. The two weights are $2$ and $3$, so

X'WX = 2\begin{pmatrix}1\\0\end{pmatrix}\begin{pmatrix}1&0\end{pmatrix} + 3\begin{pmatrix}1\\1\end{pmatrix}\begin{pmatrix}1&1\end{pmatrix} = \begin{pmatrix}2&0\\0&0\end{pmatrix} + \begin{pmatrix}3&3\\3&3\end{pmatrix} = \begin{pmatrix}5&3\\3&3\end{pmatrix}.

This calculation uses the same matrix construction as the Bernoulli examples, but the link derivative and response variance produce different weights.

Interpreting the Matrix and Avoiding a Common Error

Each weight is nonnegative, so for any coefficient-direction vector $\mathbf{a}$,

\mathbf{a}'X'WX\mathbf{a} = \sum_{i=1}^n w_i(\mathbf{x}_i'\mathbf{a})^2 \ge 0.

Thus expected information is positive semidefinite. If every $w_i>0$, it is positive definite exactly when $X$ has full column rank: the displayed quadratic form is zero only if $X\mathbf{a}=0$, which, under full column rank, implies $\mathbf{a}=0$. If some weights vanish, the weighted design may fail to have full rank even when $X$ does.

A common mistake is to use $1/\operatorname{Var}(Y_i)$ alone as the weight. That omits the change in the mean per unit change in the linear predictor. The correct weight is $\mu_{\eta,i}^2/\operatorname{Var}(Y_i)$: the derivative is squared because the score covariance contains the square of the mean sensitivity. Another mistake is to treat the observed negative Hessian as identical to $X'WX$ for every realized data set. The equality established here is for the expected negative Hessian; the observed Hessian can depend on the data.

The result also makes the model’s local information transparent. A small response variance increases information, while a small absolute value of $\mu_{\eta,i}$ decreases it. The design vectors determine which coefficient directions receive that information. Together, these factors give a direct interpretation of the expected curvature without requiring a separate Hessian calculation for every link.

Check Your Understanding

Use the mean-derivative formula and the assumptions stated above.

For a GLM with $\operatorname{Var}(Y_i)=\phi V(\mu_i)$, write the general expression for $w_i$.
Why does the information formula contain the square of $d\mu_i/d\eta_i$, rather than just its first power?
For a Bernoulli logit model, what is $w_i$ as a function of $p_i$, and at what probability is this weight largest?
Explain why one observation cannot give a positive-definite information matrix for a model with two coefficients.
State the regularity condition that lets expected score covariance equal expected negative Hessian.