Ill-Conditioned Information and Slow Convergence

Why the Information Matrix Can Make Updates Unstable

A Newton update uses the information matrix to convert a score into a coefficient change. This conversion depends not just on the size of the score, but on the curvature in the direction the score points. If the likelihood is nearly flat along some coefficient combination, even a modest score in that direction can produce a large update. The resulting step may be sensitive to small changes in the data, score, or computed curvature.

Eigenvalues and eigenvectors provide a precise way to describe this geometry. They help distinguish a direction that the data determine strongly from one that the likelihood determines only weakly. They also clarify why a large coefficient update can occur even when the score components do not look large.

Information Eigenvalues and Weak Directions

Definition: At a coefficient vector $\beta$, the observed information is $J(\beta)=-\nabla^2\ell(\beta)$. When it exists, the expected information is $I(\beta)=E[J(\beta)]$. If one of these matrices is symmetric positive definite, its eigenvectors give coefficient directions and its corresponding positive eigenvalues measure the likelihood curvature in those directions. A direction with a small eigenvalue has little curvature locally and is called a weakly identified direction.

For maximization, a Newton step $d$ solves $J(\beta)d=U(\beta)$, where $U(\beta)=\nabla\ell(\beta)$; Fisher scoring instead solves $I(\beta)d=U(\beta)$. The information matrix is the curvature used in that update. The discussion below applies to whichever positive-definite information matrix is being used.

A small eigenvalue means that changing the coefficients along its eigenvector changes the local quadratic approximation to the log-likelihood only slightly. The direction may be a combination of several coefficients rather than one coefficient alone. Near-collinear predictors are a familiar source: the model can distinguish their combined effect well while having difficulty distinguishing their separate effects.

The condition number of a symmetric positive-definite matrix $A$, in the Euclidean norm, is $\kappa_2(A)=\lambda_{\max}(A)/\lambda_{\min}(A)$. A large condition number says that curvature differs substantially across directions. A small minimum eigenvalue indicates weak curvature in absolute terms; a large condition number indicates a large imbalance in curvature. These are related but not identical observations.

Worked Example: Near-Collinear Predictors in a Gaussian GLM

Consider a Gaussian linear model with known error variance $\sigma^2=1$ and two predictor columns, each with squared length 100 and correlation 0.99. Its information matrix is $X^\mathsf{T}X$. The diagonal entries are 100 and the off-diagonal entries are $100(0.99)=99$, so

J= \begin{pmatrix} 100&99\\ 99&100 \end{pmatrix}.

The vectors $(1,1)^\mathsf{T}$ and $(1,-1)^\mathsf{T}$ are eigenvectors. Multiplication by $J$ gives eigenvalues $199$ and $1$, respectively. Thus the common-effect direction is much more strongly informed than the contrast between the two coefficients. The condition number is $199/1=199$. The model contains information about the sum of the coefficients, but comparatively little information for separating their contributions.

What a Small Eigenvalue Does to a Newton Step

Theorem: Let $J$ be a symmetric positive-definite observed information matrix, with orthonormal eigenvectors $v_1,\ldots,v_p$ and eigenvalues $\lambda_1,\ldots,\lambda_p>0$. Write the score as $U=\sum_{j=1}^p u_jv_j$. The Newton step $d=J^{-1}U$ is $$ d=\sum_{j=1}^p\frac{u_j}{\lambda_j}v_j. $$ Consequently, the score component in a direction is divided by the curvature in that direction.

Proof. By the spectral decomposition, $Jv_j=\lambda_jv_j$, so $J^{-1}v_j=v_j/\lambda_j$. Applying $J^{-1}$ to the score expansion gives

J^{-1}U =J^{-1}\sum_{j=1}^p u_jv_j =\sum_{j=1}^p\frac{u_j}{\lambda_j}v_j.

In particular, $\|d\|_2^2=\sum_j u_j^2/\lambda_j^2$, so $\|d\|_2\leq \|U\|_2/\lambda_{\min}(J)$. The bound is attained when the score lies entirely in a minimum-eigenvalue direction. Thus even a small score can yield a large step if it has a component along a weak direction. $\square$

Worked Example: A Modest Score Produces a Large Coefficient Change

Suppose the local observed information and score are

J= \begin{pmatrix} 100&0\\ 0&0.01 \end{pmatrix}, \qquad U= \begin{pmatrix} 10\\ 0.1 \end{pmatrix}.

The first coefficient direction has curvature 100 and the second has curvature 0.01. Solving $Jd=U$ component by component gives

d_1=\frac{10}{100}=0.1, \qquad d_2=\frac{0.1}{0.01}=10, \qquad d= \begin{pmatrix} 0.1\\ 10 \end{pmatrix}.

The score component for the second coefficient is only 0.1, yet it produces a change of 10 because the information in that direction is tiny. In the local quadratic approximation, the corresponding predicted log-likelihood increase is $\tfrac12 U^\mathsf{T}J^{-1}U=\tfrac12(10(0.1)+0.1(10))=1$. The large coefficient change should therefore be read as a consequence of weak curvature, not as evidence that the data precisely determine a large change.

Why Ill-Conditioning Makes Computation Sensitive

A computed update can change when either the score or the information matrix changes slightly. The following result makes the effect of a perturbation to the information matrix explicit. The norm $\|\cdot\|_2$ below is the Euclidean norm for vectors and its induced operator norm for matrices.

Theorem: Let $A$ be symmetric positive definite with minimum eigenvalue $m>0$. Let $E$ be symmetric and satisfy $\|E\|_2\leq\varepsilon<m$. Then $A+E$ is positive definite, and for any score vector $U$, $$ \|(A+E)^{-1}U-A^{-1}U\|_2 \leq \frac{\varepsilon}{m(m-\varepsilon)}\|U\|_2. $$

Proof. For any unit vector $x$, the Rayleigh quotient satisfies

x^\mathsf{T}(A+E)x \geq x^\mathsf{T}Ax-\lvert x^\mathsf{T}Ex\rvert \geq m-\|E\|_2 \geq m-\varepsilon>0.

Thus $A+E$ is positive definite and $\|(A+E)^{-1}\|_2\leq1/(m-\varepsilon)$. Also $\|A^{-1}\|_2=1/m$. The inverse identity $(A+E)^{-1}-A^{-1}=-(A+E)^{-1}EA^{-1}$ follows by multiplying out the right-hand side. Therefore

\begin{aligned} \|(A+E)^{-1}U-A^{-1}U\|_2 &\leq \|(A+E)^{-1}\|_2\|E\|_2\|A^{-1}\|_2\|U\|_2\\ &\leq \frac{\varepsilon}{m(m-\varepsilon)}\|U\|_2. \end{aligned}

This bound becomes large as the smallest eigenvalue $m$ becomes small. It explains why a weak direction can make the solution of the information system sensitive to modest curvature error. $\square$

Worked Example: A Logistic Information Matrix with a Weak Contrast

Take two Bernoulli observations in a logistic regression with two predictors and no intercept. Let their design rows be $(1,1)$ and $(1,1.02)$. At $\beta=(0,0)^\mathsf{T}$, both fitted probabilities are $0.5$, so each logistic variance weight is $p(1-p)=0.25$. The observed information for this model is $X^\mathsf{T}WX$, with $W=0.25I$, and hence

J=0.25X^\mathsf{T}X =0.25 \begin{pmatrix} 2&2.02\\ 2.02&2.0404 \end{pmatrix}.

The trace of $X^\mathsf{T}X$ is $4.0404$. Its determinant is $0.0004$: indeed, the determinant of $X$ is $1.02-1=0.02$, and $\det(X^\mathsf{T}X)=\det(X)^2=0.0004$. The eigenvalues of $X^\mathsf{T}X$ are approximately $4.040301$ and $0.0000990$, since their sum is the trace and their product is the determinant. Multiplying by 0.25 gives eigenvalues of $J$ approximately $1.010075$ and $0.00002475$, with condition number about $40{,}800$.

The design columns are nearly the same, so the information is strong for a shared predictor effect and weak for a contrast between the two effects. This calculation concerns local information at $\beta=(0,0)^\mathsf{T}$; the observed outcomes do not change the logistic information there because the fitted weights are fixed by the fitted probabilities.

Slow Convergence: What Ill-Conditioning Does and Does Not Mean

A large condition number does not, by itself, prove that Newton’s method must converge slowly. For a strictly concave quadratic log-likelihood with constant positive-definite information, the Newton update reaches its maximizer in one exact step, regardless of the condition number. Ill-conditioning instead warns that the step can be very large in weak directions and sensitive to small errors. In a nonquadratic likelihood, curvature changes as the coefficients move; a step that is unreliable or too large may be shortened by an acceptance procedure, so progress in other directions can also be slow.

The distinction is important: small eigenvalues describe weak local curvature, while slow convergence describes the behavior of a sequence of updates. The first can contribute to the second, but the details of the likelihood and the algorithm matter. A solver may also report small coefficient changes when its accepted steps are restricted, even though the underlying model remains weakly informative in a particular direction.

The information matrix also depends on how coefficients are represented. A change of units can alter the numerical eigenvalues and condition number. Thus a large condition number is a useful warning about numerical sensitivity, but it should be interpreted alongside the design, coefficient definitions, and the fitted likelihood. A weak direction is a statement about a coefficient combination in the specified model, not automatically a diagnosis of a particular scientific cause.

Common Mistake

Wrong reasoning: “The score is small, so the Newton step must be small.”
Correction: The step is $J^{-1}U$, not $U$ alone. A score component $u_j$ in an eigenvector direction is divided by its eigenvalue $\lambda_j$. If $\lambda_j$ is very small, that component can create a large coefficient change.
Another overstatement: “A large condition number guarantees slow Newton convergence.”
Correction: It signals sensitivity and uneven curvature. An exact Newton method still solves a quadratic problem in one step; slow progress can arise when nonlinear curvature or step acceptance makes the computed direction difficult to use.

Key takeaway: Read the information matrix direction by direction. Its eigenvectors identify coefficient combinations, its eigenvalues measure local curvature, and small eigenvalues explain how modest score components or numerical perturbations can produce erratic coefficient updates.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

If a score is entirely in an eigenvector direction with eigenvalue $\lambda$, what is the Newton-step component in that direction?
In the Gaussian example, which coefficient combination is weakly informed: the sum or the contrast? What eigenvalue supports your answer?
Why does a small minimum eigenvalue make an information-system solution sensitive to curvature perturbations?
Does a large condition number alone prove that an exact Newton method will converge slowly? Explain.
In the logistic example, what feature of the design creates the small information eigenvalue?