Why Inspect the Hessian After Convergence?
A small gradient is evidence that an optimizer has reached a stationary point, but it does not by itself show that the point is a well-defined minimum. The objective may be nearly flat in some parameter directions, or the stationary point may have downward curvature in at least one direction. Either situation can make estimates sensitive to small numerical changes and can obstruct a covariance calculation.
The Hessian summarizes the local curvature of the objective. In a pharmacometric fit, examining it is a diagnostic step: it helps assess whether the fitted point has the expected local shape and whether numerical uncertainty calculations are likely to be stable. It does not, on its own, identify the cause of a problem. For example, curvature can be affected by parameter units, objective scaling, model structure, data information, boundaries, and numerical error.
The sign convention matters. Here \(Q\) is an objective to be minimized, such as the negative log-likelihood. At an unconstrained, regular local minimum, the gradient is zero and the Hessian is positive semidefinite; a positive-definite Hessian indicates strictly positive curvature in every nonzero local direction. If software reports a maximized log-likelihood instead, the expected curvature sign is reversed. Always establish what function was differentiated before interpreting the signs.
Curvature From a Local Approximation
Near a candidate estimate \(\hat{\theta}\), a second-order Taylor approximation gives a direct interpretation of the Hessian. Write a small displacement as \(v\). If the gradient at \(\hat{\theta}\) is zero, then the leading change in the objective is approximately one-half the quadratic form \(v^T H(\hat{\theta})v\).
A positive value of \(v^T H v\) means the quadratic approximation rises along that displacement; a value near zero means little local curvature in that direction. If the form is negative for some \(v\), the approximation falls away from the candidate point in that direction, which is inconsistent with an unconstrained local minimum. These statements concern local curvature, not the global shape of the objective.
For a symmetric Hessian, positive definiteness can be checked in several equivalent ways. One is to verify that every eigenvalue is positive. For a small matrix, Sylvester’s criterion is often simpler: all leading principal minors must be positive. For larger matrices, a successful Cholesky factorization is a commonly used computational check. A failed factorization is a warning to investigate, not an explanation by itself; rounding error, poor scaling, and the point of evaluation all matter.
Worked Example: Verifying Positive Curvature in Two Parameters
Suppose a hypothetical negative log-likelihood has the local quadratic form \(Q(\theta)=Q(\hat{\theta})+\frac{1}{2}v^T H v\), with \(v=(u,w)^T\) and \(H=\begin{pmatrix}4&1\\1&2\end{pmatrix}\). Check whether this quadratic approximation has positive curvature in every nonzero direction.
First, the leading principal minors are \(4\) and \(\det(H)=4(2)-1(1)=7\). Both are positive, so Sylvester’s criterion establishes that \(H\) is positive definite.
The quadratic form gives the same conclusion directly: \(v^T H v=4u^2+2uw+2w^2\). Completing the square, \(4u^2+2uw+2w^2=4(u+w/4)^2+\frac{7}{4}w^2\). This is positive for every nonzero pair \((u,w)\): if \(w\ne 0\), the second term is positive, and if \(w=0\) with \(u\ne 0\), the first term is positive. Thus the local quadratic approximation rises for every nonzero displacement.
For this matrix, the eigenvalues are \(3+\sqrt{2}\approx4.414\) and \(3-\sqrt{2}\approx1.586\), both positive. Their ratio is approximately \(2.784\), so this example has no extreme eigenvalue disparity. Positive definiteness establishes the curvature sign; it does not guarantee that a real fitted model is globally well behaved.
Conditioning and Coordinate Dependence
For a symmetric positive-definite Hessian, the spectral condition number is the ratio of its largest eigenvalue to its smallest eigenvalue. A large ratio means that curvature differs greatly across some directions. In numerical linear algebra, this often signals that solving systems involving the Hessian or inverting it may be sensitive to rounding error or small perturbations. It is a relative measure: it does not say whether the objective itself has been scaled to very large or very small values.
Conditioning depends on parameter coordinates. If one parameter is expressed in milligrams and then re-expressed in grams, the numerical entries of the Hessian change. Consequently, an extreme condition number can partly reflect units rather than poor information in a scientifically meaningful, scale-adjusted sense. This does not make the computational warning irrelevant: the optimizer and covariance calculation still operate on numerical coordinates. It means that the condition number should be interpreted alongside parameter scales and units, rather than as a stand-alone verdict.
Worked Example: A Large Condition Number From Unequal Scales
Consider the hypothetical local objective \(Q(x,y)=\frac{1}{2}(x^2+10^{-6}y^2)\), where the numerical units of \(x\) and \(y\) differ substantially. At the minimum \((0,0)\), the Hessian in these coordinates is \(H=\begin{pmatrix}1&0\\0&10^{-6}\end{pmatrix}\).
Its eigenvalues are \(1\) and \(10^{-6}\), so its condition number is \(1/(10^{-6})=10^6\). Curvature along the \(y\) coordinate is a million times smaller than along \(x\). That difference can make computations in these coordinates numerically sensitive.
Now set \(z=y/1000\), or equivalently \(y=1000z\). The same objective becomes \(Q(x,1000z)=\frac{1}{2}(x^2+z^2)\), whose Hessian in \((x,z)\) coordinates is the identity matrix and whose condition number is \(1\). The change demonstrates coordinate dependence: it improves the numerical balance of the representation, but it does not change the underlying objective or the parameter’s meaning. In the original coordinates, the inverse Hessian is \(\operatorname{diag}(1,10^6)\); the large variance scale for \(y\) remains when uncertainty is expressed in its original units.
Objective scaling matters as well. Multiplying the entire objective by a positive constant multiplies all Hessian eigenvalues by that constant but leaves their ratio unchanged. If \(Q=-\ell\), where \(\ell\) is the log-likelihood, let \(H\) denote the Hessian of \(-\ell\); the usual local likelihood covariance approximation is \(H^{-1}\). If the reported objective is \(-2\ell\), its Hessian is \(2H\), whose inverse is \(\tfrac12H^{-1}\); the likelihood covariance approximation remains \(H^{-1}=2(2H)^{-1}\). Confirming the objective convention prevents a scale factor from being mistaken for a curvature defect.
When Curvature Is Indefinite
A negative Hessian eigenvalue at a purported unconstrained minimum indicates negative curvature in some direction in the quadratic approximation. Possible explanations include a point that is not actually a local minimum, an inaccurate Hessian, or a constrained or boundary situation where unconstrained local-minimum conditions do not apply in the same way. A covariance calculation based on an inverse Hessian also requires care: an indefinite inverse is not a valid covariance matrix because it is not positive semidefinite.
Worked Example: An Indefinite Hessian and an Invalid Covariance
Suppose a hypothetical objective near a stationary point has Hessian \(H=\begin{pmatrix}2&3\\3&2\end{pmatrix}\). Its determinant is \(4-9=-5\), so it is not positive definite.
The eigenvectors in the directions \((1,1)^T\) and \((1,-1)^T\) have eigenvalues \(5\) and \(-1\), respectively. Along the unit direction \(a=(1,-1)^T/\sqrt{2}\), the quadratic contribution for displacement \(ta\) is \(\frac{1}{2}t^2a^THa=-\frac{1}{2}t^2\). It decreases for every nonzero \(t\), so the quadratic approximation cannot describe an unconstrained local minimum at this stationary point.
The inverse is \(H^{-1}=\frac{1}{-5}\begin{pmatrix}2&-3\\-3&2\end{pmatrix} =\begin{pmatrix}-0.4&0.6\\0.6&-0.4\end{pmatrix}\). Its negative diagonal entries rule it out as a covariance matrix. The appropriate conclusion is that the covariance approximation is invalid at this Hessian; the matrix alone does not establish why the estimate or Hessian has this shape.
Practical Diagnostic Sequence
Inspect the Hessian at the final estimate and relate the result to the optimization evidence. Gradient-based convergence checks are relevant because a Hessian evaluated far from a stationary point may describe curvature that is not representative of the fitted solution. At the same time, a small gradient does not guarantee a well-conditioned positive-definite Hessian. These are complementary checks.
Identify whether the software used a minimized negative log-likelihood, an objective such as \(-2\ell\), or another scaled function. Check whether parameters were transformed and whether the Hessian is reported in transformed or original coordinates.
Verify that the Hessian corresponds to the final estimate. Compare opposite off-diagonal entries; a substantial discrepancy in a matrix expected to be symmetric suggests a numerical, implementation, or reporting issue worth investigating.
Use eigenvalues, a factorization, or suitable matrix diagnostics. Record whether negative curvature is present and whether positive curvature is spread over a wide numerical range.
Where feasible, compare Hessians computed with a more accurate method or modestly changed numerical settings. Check whether rescaling parameters changes the numerical conditioning while preserving the fitted objective and its interpretation.
For a Hessian based on finite differences, the result can depend on step sizes: steps that are too large blur local curvature, while steps that are too small can amplify rounding error. Automatic differentiation can reduce some differentiation errors, but it cannot repair a poorly scaled objective, a model that is not sufficiently smooth, or a point that is not a local minimum. If small changes in numerical settings produce large changes in curvature diagnostics, treat the covariance calculation as unstable until the discrepancy is understood.
Common clues include a failed positive-definite factorization, an inverse with implausible or negative diagonal entries, marked asymmetry, or sensitivity to reasonable numerical settings. These are diagnostic signals rather than unique fingerprints. Potential causes include inadequate numerical accuracy, poor coordinate scaling, a nonstationary solution, parameter constraints, or weak local curvature. The estimate and objective trace, numerical configuration, parameter transformations, and model behavior should be considered together.
What the Hessian Can and Cannot Establish
Hessian diagnostics are useful because they separate several questions that are sometimes conflated. A positive-definite Hessian supports positive local curvature in the coordinates and objective being examined. A large condition number warns of numerical sensitivity. Neither result proves that the model is scientifically adequate, that a global optimum has been found, or that all parameters are practically well determined. Conversely, a numerical Hessian problem does not by itself prove that the model is unusable.
The main alternatives are complementary rather than interchangeable. A gradient check assesses stationarity, not curvature. An optimizer’s convergence message summarizes its stopping rules but does not replace examination of the local second-order structure. Recomputing the Hessian using an alternative differentiation method can help assess numerical reliability, while rescaling parameters can help identify coordinate-related conditioning. Neither computation alone identifies whether a model or data feature is responsible for a flat direction. Later diagnostics can investigate those specific explanations in more detail.
A practical conclusion should therefore state what was observed and what remains uncertain: for example, “the computed Hessian is positive definite but poorly conditioned in the reported parameter coordinates; the covariance approximation is sensitive and should be treated cautiously.” This is more informative than calling the fit simply “converged” or “failed.”
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A symmetric two-parameter Hessian has leading principal minors \(3\) and \(8\). What does Sylvester’s criterion imply about its curvature?
- A positive-definite Hessian has eigenvalues \(12\) and \(0.03\). Calculate its spectral condition number and state what it indicates.
- Why can multiplying a minimized objective by \(5\) change Hessian entries but leave the spectral condition number unchanged?
- A computed inverse Hessian has a negative diagonal entry. Is it a valid covariance matrix? What is the immediate diagnostic conclusion?
- Give two reasons why a Hessian diagnostic should not be interpreted as a complete explanation of an estimation failure.
Solutions. (1) Both leading principal minors are positive, so the Hessian is positive definite and has positive curvature in every nonzero direction. (2) The condition number is \(12/0.03=400\), indicating substantial curvature disparity and potential numerical sensitivity; it is not, by itself, proof of a model defect. (3) Multiplying the objective by \(5\) multiplies every Hessian eigenvalue by \(5\), so their ratio is unchanged. (4) No. A covariance matrix must be positive semidefinite, so the computed inverse is invalid as a covariance approximation and requires investigation. (5) For example, the Hessian may be inaccurate because of numerical differentiation, or the point may not be a stationary local minimum; scaling and constraints can also affect interpretation. The diagnostic identifies a problem to investigate, not a unique cause.