Likelihood Sensitivity to Design and Information Loss

Why the Design Controls Likelihood Precision

A generalized linear model can have a well-defined likelihood and still estimate some combinations of its coefficients much less precisely than others. This is often a property of the design: the observed predictor values may provide ample information about one coefficient direction but little information about another. Eigenvalues of the information matrix summarize this local geometry. A small eigenvalue signals a direction in coefficient space along which the likelihood changes relatively little.

This tutorial focuses on that sensitivity. It does not treat an eigenvalue as a universal measure of model quality: its interpretation depends on the units and scaling of the coefficients, and it describes local curvature rather than every feature of the likelihood. The distinction between expected and observed information is also essential. Their eigenvectors need not agree in a particular sample.

Information in a Generalized Linear Model

Consider independent responses \(Y_1,\ldots,Y_n\) in a generalized linear model with fixed dispersion parameter \(\phi\), linear predictors \(\eta_i=x_i^T\beta\), and mean \(\mu_i\). Here \(x_i\) is the \(p\)-component design vector for observation \(i\), and \(\beta\) is the coefficient vector. Suppose \(\operatorname{Var}(Y_i)=\phi V(\mu_i)\), and the link is \(\eta_i=g(\mu_i)\). Under the usual differentiability and regularity conditions for likelihood differentiation, the expected Fisher information is

The formula shows how information combines the design rows and the model-dependent weights. For any coefficient direction \(v\), its expected curvature is

Thus a direction is weakly informed when the weighted predictor changes \(x_i^Tv\) are small across the observed design. This can happen because a predictor barely varies, because predictors are nearly redundant, or because the weights are small in observations that would otherwise distinguish coefficient combinations.

Eigenvalues and Local Likelihood Curvature

For a symmetric information matrix, the spectral theorem gives orthonormal eigenvectors \(v_1,\ldots,v_p\) and eigenvalues \(\lambda_1,\ldots,\lambda_p\). If these are for \(\mathcal I(\beta)\), then for a unit vector \(v_j\),

This is a statement about expected information. To connect it to a realized log-likelihood, let \(\hat\beta\) be an interior maximum and suppose \(\ell\) is twice continuously differentiable in a neighborhood of \(\hat\beta\). Taylor expansion gives

because the score is zero at an interior maximum. If \(v_j\) is an eigenvector of \(\mathcal I(\hat\beta)\), substituting \(h=t v_j\) gives the quadratic term \(-t^2v_j^TJ(\hat\beta)v_j/2\), not automatically \(-\lambda_jt^2/2\). The latter is exact when \(v_j\) and \(\lambda_j\) are an eigenvector and eigenvalue of \(J(\hat\beta)\). When observed information is close to expected information, it is an approximation for the expected-information eigenvector.

This approximation explains the precision interpretation: a small information eigenvalue produces a large approximate variance in the corresponding direction. It also explains why a particular coefficient can be imprecise even when other coefficients are estimated precisely. For a linear contrast \(c^T\beta\), the approximate variance is \(c^T\mathcal I^{-1}c\); it depends on how the contrast aligns with the eigenvectors, not only on the smallest eigenvalue.

How Design Changes Information

The identity \(\mathcal I=X^TWX\) makes comparisons between designs concrete. If the weights are fixed and a new observation with design vector \(x\) and positive weight \(w\) is added, its contribution is \(wxx^T\). For every vector \(u\),

So adding an observation cannot reduce information in the positive-semidefinite matrix order when its weight is positive and the other weights are held fixed. It may add no information in a particular direction if \(x^Tu=0\). In an actual GLM fit, adding an observation can also change fitted means and hence all the weights; the fixed-weight comparison isolates the direct design contribution.

To verify the identity, set \(A=\mathcal I\) and \(q=x^TA^{-1}x\). Multiply the proposed right-hand side by \(A+wxx^T\). The product of \(A\) with the right-hand side is \(I-wxx^TA^{-1}/(1+wq)\). The product of \(wxx^T\) with it is \(wxx^TA^{-1}-w^2qxx^TA^{-1}/(1+wq)\). The coefficient of \(wxx^TA^{-1}\), after combining terms, is

The product is the identity, proving the formula. For any contrast \(c\), the variance reduction is \[ \frac{w(c^T\mathcal I^{-1}x)^2}{1+w x^T\mathcal I^{-1}x}\geq 0. \] This also shows that an observation is most helpful for a contrast when its design vector aligns with that contrast after accounting for current information.

Collinearity and Weakly Identified Directions

When predictor columns are nearly redundant, the model may estimate their separate coefficients poorly even when their combined effect is well determined. The relevant weak direction is often a contrast between coefficients. The next example calculates this effect directly in a logistic model.

Scaling, Interpretation, and a Common Pitfall

Eigenvalues depend on coefficient coordinates. Changing a predictor from meters to millimeters rescales its coefficient and changes the numerical eigenvalues, even though the fitted model can represent the same predictions. Therefore, compare information eigenvalues only after considering the units and scaling of the coefficient vector. Standardizing predictors can make a numerical diagnosis easier to interpret, but it changes the coordinate system; it does not create new information in the data.

A related pitfall is to treat the expected-information eigenvector as an exact direction of curvature for the observed likelihood. The incorrect reasoning is: “\(v_j\) is an eigenvector of \(\mathcal I\), so \(v_j^TJv_j=\lambda_j\).” That conclusion does not follow, since generally \(Jv_j\ne\lambda_jv_j\). The correction is to use \(v_j^TJv_j\) in the observed Taylor expansion, or to use eigenvectors and eigenvalues of \(J\) when an exact observed-curvature statement is wanted. Expected-information eigenvalues give expected curvature and, under regular asymptotic conditions, an approximation to local precision.

In practice, a small eigenvalue is a diagnostic prompt rather than a complete verdict. Check the corresponding coefficient combination, the predictor scales, the fitted weights, and whether the likelihood is locally well approximated by a quadratic. If the information is singular, the usual inverse-information covariance approximation is unavailable, and one or more coefficient directions are not locally identified by that design.