Expected Information for Regression Coefficients

Why Expected Information Has a Weighted Design Form

In a generalized linear model, the regression coefficients affect each observation through its linear predictor, while the response distribution determines how informative that observation is about its mean. Expected Fisher information combines these two ingredients: the design specifies the directions in coefficient space, and a scalar weight specifies the information supplied along each design direction.

The link and variance function both matter. Two models with the same design matrix can therefore have different coefficient information, even when their fitted means happen to agree. The derivation below makes that dependence explicit and establishes when the resulting information matrix identifies all coefficient directions.

Model and Information Definition

Let \(Y_1,\ldots,Y_n\) be independent observations from a regular one-parameter exponential-dispersion family, with fixed design vectors \(x_i\in\mathbb{R}^p\). Suppose the density or mass function for observation \(i\) has the form

where the dispersion \(\phi>0\) is known and common to the observations. Write \(\mu_i=E[Y_i]\) and \(V(\mu_i)=\operatorname{Var}(Y_i)/\phi\). Thus \(E[Y_i]=b'(\theta_i)=\mu_i\), \(V(\mu_i)=b''(\theta_i)>0\), and \(\operatorname{Var}(Y_i)=\phi V(\mu_i)\). Let \(\beta\in\mathbb{R}^p\) be in an open parameter set, set \(\eta_i=x_i^T\beta\), and use a differentiable link \(\eta_i=g(\mu_i)\). Assume the mean is in the interior of its allowable range, the link has a nonzero derivative there, and the likelihood is regular so that differentiation and expectation can be interchanged where needed.

Let \(X\) be the \(n\times p\) matrix whose \(i\)th row is \(x_i^T\). The diagonal matrix of information weights will be denoted by \(W\); its entries will be derived rather than assumed.

Deriving the Score and Expected Information

For one observation, differentiating the exponential-dispersion log-likelihood with respect to \(\theta_i\) gives \((y_i-\mu_i)/\phi\). Also \(d\theta_i/d\mu_i=1/V(\mu_i)\), and \(d\mu_i/d\eta_i\) describes how the link converts a change in the linear predictor into a change in the mean. Define \(a_i=d\mu_i/d\eta_i\). By the chain rule, the contribution to the coefficient score is

Because \(\eta_i=g(\mu_i)\), the link derivative gives \(a_i=1/g'(\mu_i)\). The response variance then determines the covariance of each score contribution. Independence makes the covariance between distinct contributions zero, and each contribution has mean zero.

Proof. First derive the expected negative Hessian. Put \(c_i(\mu_i)=a_i/\{\phi V(\mu_i)\}\). The score contribution is \(c_i(\mu_i)(Y_i-\mu_i)x_i\). Since \(d\mu_i/d\beta=a_i x_i\), differentiating this score gives

Taking expectation at the model parameter eliminates the first term because \(E[Y_i-\mu_i]=0\). The remaining expected negative Hessian contribution is \(c_i a_i x_ix_i^T=a_i^2x_ix_i^T/\{\phi V(\mu_i)\}\). Summing over observations yields \(X^TWX\).

For the covariance definition, the score contributions are independent and mean zero. Therefore

Their sum is again \(X^TWX\). This also verifies the information equality directly for this model. \(\square\)

The formula separates the roles of the model components. The design enters through \(x_ix_i^T\); the variance function and dispersion determine response variability; and the link enters through \(a_i\), the derivative of the mean with respect to the linear predictor. When the mean link is differentiable with nonzero derivative, a steep change in the link scale can alter how much coefficient information a given mean supplies.

Worked Examples

When Does the Information Identify Every Coefficient Direction?

A positive information weight cannot compensate for a design that fails to distinguish some coefficient combinations. The following condition makes this precise.

Proof. For any \(v\in\mathbb{R}^p\),

If \(X\) has full column rank, \(Xv=0\) implies \(v=0\). Thus for each nonzero \(v\), at least one \(x_i^Tv\neq0\); since every \(w_i>0\), the displayed sum is strictly positive. Hence \(I(\beta)\) is positive definite. Conversely, if \(X\) does not have full column rank, there is a nonzero \(v\) with \(Xv=0\). The displayed quadratic form is then zero, so \(I(\beta)\) is not positive definite. \(\square\)

Under the stated interior-mean and positive-variance conditions, the formula gives \(w_i>0\). If some observations have zero weight because a model is at a boundary or violates those conditions, the relevant condition is instead that the positively weighted design rows span \(\mathbb{R}^p\).

Common Pitfall and Practical Interpretation

Common mistake: For a Bernoulli log-link model, one might substitute the Bernoulli variance \(\mu(1-\mu)\) into the denominator but forget to square the derivative \(d\mu/d\eta=\mu\), or incorrectly treat the resulting weight as \(\mu(1-\mu)\). The correction is to use the full formula: \(w=\{d\mu/d\eta\}^2/\{\phi V(\mu)\}\). At \(\mu=1/2\), this is \((1/2)^2/\{(1/2)(1/2)\}=1\); at \(\mu=3/4\), it is \((3/4)^2/\{(3/4)(1/4)\}=3\). The link derivative is essential.

The expected weights depend on the model parameter through the means, but not on the observed response values. This distinguishes expected information from the observed negative Hessian, which can depend on the data. The design-weight decomposition is also useful for understanding local precision: information accumulates across observations in the directions represented by their design vectors, with the model assigning each direction its own weight.