Indefinite Hessians and Non-Ascent Newton Directions

Why the Sign of the Newton Direction Matters

Newton’s method for maximizing a log-likelihood uses its local curvature to choose a direction. When the observed curvature is appropriate, the Newton direction is locally ascending. But observed curvature can be indefinite: in some parameter directions the log-likelihood curves downward, while in others it curves upward. In that case, solving the Newton equations does not by itself ensure an improvement in the likelihood.

The relevant check is not whether a step is called a “Newton step,” but whether the score increases in the proposed direction. This distinction matters in GLM computation: a line search can shorten an ascending direction, but it cannot make a locally descending direction ascending.

Definition: Let $\ell(\beta)$ be a twice differentiable log-likelihood, with score $U(\beta)=\nabla\ell(\beta)$ and Hessian $H(\beta)=\nabla^2\ell(\beta)$. When $H(\beta)$ is nonsingular, the Newton direction for maximizing $\ell$ is $d_N=-H(\beta)^{-1}U(\beta)$. The observed information is $J_{\mathrm{obs}}(\beta)=-H(\beta)$, so equivalently $d_N=J_{\mathrm{obs}}(\beta)^{-1}U(\beta)$.

A symmetric matrix is indefinite if it has both positive and negative eigenvalues. The observed information is indefinite exactly when the Hessian is indefinite. If the Hessian is negative definite, the log-likelihood has downward curvature in every nonzero direction, and the nonsingular Newton direction is ascending unless the score is zero. Indefinite curvature has no such guarantee.

The Directional Derivative Test

For any direction $d$, the derivative of the log-likelihood along the path $\beta+\alpha d$ at $\alpha=0$ is $U(\beta)^{\mathsf T}d$. A positive value means the log-likelihood increases for sufficiently small positive $\alpha$; a negative value means it decreases for sufficiently small positive $\alpha$. This is a local statement, not a guarantee that a full step of length one improves the likelihood.

Theorem: Suppose $\ell$ is differentiable at $\beta$ and $H(\beta)$ is nonsingular. The Newton direction is a strict ascent direction if and only if $$ U(\beta)^{\mathsf T}d_N=-U(\beta)^{\mathsf T}H(\beta)^{-1}U(\beta)>0. $$ If this quantity is negative, then $\ell(\beta+\alpha d_N)<\ell(\beta)$ for all sufficiently small $\alpha>0$.

Proof. By the definition of the Newton direction, $d_N=-H^{-1}U$, so its directional derivative is $U^{\mathsf T}d_N=-U^{\mathsf T}H^{-1}U$. If this is positive, differentiability gives $\ell(\beta+\alpha d_N)-\ell(\beta)=\alpha U^{\mathsf T}d_N+o(\alpha)$, which is positive for sufficiently small positive $\alpha$. If it is negative, the same expansion is negative for sufficiently small positive $\alpha$. Thus the sign of the displayed quantity gives the stated test. $\square$

When $H$ is indefinite, $H^{-1}$ is indefinite as well. Consequently, the quadratic form $U^{\mathsf T}H^{-1}U$ can have either sign. An indefinite Hessian therefore does not mean every Newton direction is descending; it means the sign must be checked rather than assumed.

Worked Example: A Newton Direction That Decreases a GLM Log-Likelihood

Let $Y_1,Y_2$ be independent with $Y_i\sim N(\mu_i,1)$, where the GLM has a log link and design matrix equal to the $2$-by-$2$ identity matrix. Thus $\mu_i=\exp(\beta_i)$. For observed data $y_1=10,y_2=0$, the log-likelihood, omitting a constant, is $\ell(\beta)=-\tfrac12\sum_{i=1}^2(y_i-\exp(\beta_i))^2$. Evaluate its Newton direction at $\beta=(0,0)^{\mathsf T}$.

Writing $\mu_i=\exp(\beta_i)$, differentiation gives $U_i=(y_i-\mu_i)\mu_i$ and $H_{ii}=\mu_i(y_i-2\mu_i)$; the off-diagonal Hessian entries are zero because the two predictors involve separate parameters. At $\beta=(0,0)^{\mathsf T}$, both means equal $1$, so

U=\begin{pmatrix}9\\-1\end{pmatrix}, \qquad H=\begin{pmatrix}8&0\\0&-2\end{pmatrix}.

The Hessian has one positive and one negative eigenvalue, so it is indefinite and nonsingular. The maximizing Newton direction is

d_N=-H^{-1}U =-\begin{pmatrix}1/8&0\\0&-1/2\end{pmatrix} \begin{pmatrix}9\\-1\end{pmatrix} =\begin{pmatrix}-9/8\\-1/2\end{pmatrix}.

Its directional derivative is $U^{\mathsf T}d_N=9(-9/8)+(-1)(-1/2)=-81/8+4/8=-77/8$, which is negative. In particular, $\ell(\beta+\alpha d_N)-\ell(\beta)=\alpha(-77/8)+o(\alpha)$, so the likelihood decreases for all sufficiently small positive step lengths. The Newton equations have been solved correctly; the failure is that their direction is not an ascent direction.

Fisher Scoring as a Curvature Safeguard

Fisher scoring replaces the observed information $J_{\mathrm{obs}}=-H$ by the expected information $J_F=E[J_{\mathrm{obs}}]$, evaluated at the current parameter. Under regularity conditions— a common support for the model, a twice differentiable density or mass function, and conditions permitting differentiation under the integral or sum—the score has mean zero and $J_F=E[UU^{\mathsf T}]=-E[H]$. This matrix is positive semidefinite, since for every vector $v$, $v^{\mathsf T}E[UU^{\mathsf T}]v=E[(v^{\mathsf T}U)^2]\geq 0$.

When $J_F$ is positive definite, the Fisher scoring direction $d_F=J_F^{-1}U$ is a strict ascent direction whenever $U\neq 0$. Positive definiteness is important: if the expected information is singular, its inverse may not exist, and the strict-ascent conclusion does not automatically follow.

Result: If $J_F$ is positive definite, then the Fisher scoring direction satisfies $$ U^{\mathsf T}d_F=U^{\mathsf T}J_F^{-1}U>0 $$ for every nonzero score $U$. Thus it is a strict ascent direction away from a stationary point.

Justification. The inverse of a positive-definite symmetric matrix is positive definite. Applying its defining quadratic-form property to the nonzero vector $U$ gives $U^{\mathsf T}J_F^{-1}U>0$. The directional-derivative test then gives ascent for sufficiently small positive step lengths. $\square$

Worked Example: Fisher Scoring for the Same Gaussian GLM

Continue with the independent Gaussian observations and log link from the previous example, at $\beta=(0,0)^{\mathsf T}$ and data $(10,0)$. For each observation, $H_{ii}=\mu_i(y_i-2\mu_i)$. Since $E[Y_i]=\mu_i$, $E[H_{ii}]=\mu_i(\mu_i-2\mu_i)=-\mu_i^2$. Therefore $J_F=-E[H]=\operatorname{diag}(\mu_1^2,\mu_2^2)$, which at the current parameter is the identity matrix.

The score is still $U=(9,-1)^{\mathsf T}$, so the Fisher scoring direction is $d_F=J_F^{-1}U=(9,-1)^{\mathsf T}$. Its directional derivative is $U^{\mathsf T}d_F=9^2+(-1)^2=82>0$. Thus Fisher scoring points uphill locally in this example, while the observed-Hessian Newton direction points downhill. This comparison concerns direction; a full Fisher scoring step can still require shortening if it does not improve the likelihood.

Modified Curvature

Another safeguard is to replace the indefinite observed information with a positive-definite matrix $B$, then use $d=B^{-1}U$. For example, if $J_{\mathrm{obs}}=Q\operatorname{diag}(\lambda_1,\ldots,\lambda_p)Q^{\mathsf T}$ is an eigendecomposition, one can replace each eigenvalue by $\max(\lambda_i,\delta)$, where $\delta>0$. The resulting matrix $B$ is positive definite. Alternatively, a diagonal shift $B=J_{\mathrm{obs}}+\tau I$ is positive definite whenever $\tau>-\lambda_{\min}(J_{\mathrm{obs}})$.

Result: For any symmetric positive-definite modified curvature matrix $B$, the direction $d=B^{-1}U$ is a strict ascent direction whenever $U\neq0$, because $U^{\mathsf T}d=U^{\mathsf T}B^{-1}U>0$.

The guarantee is about the local direction, not about the size of the step. A likelihood-based step-halving or backtracking procedure can still be used to select a step length. Modified curvature changes the proposed direction; a line search changes how far to move along that direction.

Worked Example: Shifting the Observed Information

For the Gaussian GLM counterexample, the observed information at $\beta=(0,0)^{\mathsf T}$ is $J_{\mathrm{obs}}=-H=\operatorname{diag}(-8,2)$. Its smallest eigenvalue is $-8$. Choose the shift $\tau=9$, which exceeds $8$, and set $B=J_{\mathrm{obs}}+9I=\operatorname{diag}(1,11)$. This matrix is positive definite.

Using the same score $U=(9,-1)^{\mathsf T}$, the modified direction is $d=B^{-1}U=(9,-1/11)^{\mathsf T}$. Its directional derivative is $U^{\mathsf T}d=9(9)+(-1)(-1/11)=81+1/11=892/11>0$. The modified curvature therefore gives a locally ascending direction. The shift is a safeguard, not a claim that the modified matrix equals the observed or expected information.

Worked Example: Indefinite Curvature Does Not Always Mean Descent

Keep the same Gaussian GLM and evaluate at $\beta=(0,0)^{\mathsf T}$, but now observe $y_1=3,y_2=1.9$. The score and Hessian are $U=(2,0.9)^{\mathsf T}$ and $H=\operatorname{diag}(1,-0.1)$. The Hessian is still indefinite. Nevertheless, the Newton direction is $d_N=-H^{-1}U=(-2,9)^{\mathsf T}$.

Here $U^{\mathsf T}d_N=2(-2)+0.9(9)=-4+8.1=4.1>0$, so the direction is ascending locally despite the indefinite Hessian. This confirms that indefiniteness removes a guarantee; it does not determine the direction’s sign without also considering the score.

Common Pitfall and Practical Interpretation

A common mistake is to reason, “The Newton equations use the Hessian, so the Newton step must maximize the local likelihood.” That conclusion is valid when the Hessian is negative definite, but not when it is indefinite. The quadratic approximation itself can have both upward- and downward-curving directions, and the Newton solution can point downhill according to the score test.

The correction is to compute $U^{\mathsf T}d$ before interpreting a proposed direction. If it is nonpositive, do not expect a standard positive step length along that direction to give a local likelihood increase. Fisher scoring can provide a direction with positive score product when its expected information is positive definite; modified positive-definite curvature provides another option. If the direction is ascending, a line search may still be needed to control step length.

Check Your Understanding

Use the score-direction test and the curvature definitions to answer these questions.

State the maximizing Newton direction in terms of the Hessian $H$, and in terms of observed information $J_{\mathrm{obs}}$.
Why does an indefinite Hessian fail to guarantee ascent, and why does it also fail to imply that every Newton direction is descending?
In the first Gaussian GLM example, verify the sign of $U^{\mathsf T}d_N$ and explain what that sign implies for sufficiently small positive step lengths.
What condition on the Fisher information makes the Fisher scoring direction a strict ascent direction for a nonzero score?
Why can step-halving fail to rescue a direction whose directional derivative is negative at the current parameter?