Why Look Beyond a Convergence Flag?
An optimizer can stop near a point where the gradient is small, yet the estimate may still be poorly determined in a particular combination of parameters. For example, a pharmacokinetic model may fit similarly well when clearance and volume shift together along a specific direction. Looking only at each parameter separately can obscure that direction. The Hessian eigenvectors identify combinations of parameters, while their eigenvalues describe the local curvature of the objective along those combinations.
This is a local diagnostic, not a verdict about the entire model. A small eigenvalue can indicate a direction in which the fitted objective changes slowly near the estimate, but it does not by itself establish structural nonidentifiability or explain why the information is weak. Numerical accuracy, parameter scaling, the objective’s scale, and whether the estimate is truly at an interior minimum all matter.
Definition and Interpretation
Let \(\boldsymbol{\theta}\) be a vector of model parameters and let \(F(\boldsymbol{\theta})\) be the objective being minimized. For a likelihood-based fit, take \(F\) to be the negative log likelihood unless a different convention is stated. The Hessian \(H\) is the matrix of second partial derivatives of \(F\), evaluated at the fitted estimate \(\hat{\boldsymbol{\theta}}\). If \(F\) is twice continuously differentiable, its exact Hessian is symmetric.
Near an interior estimate, a second-order approximation gives
When the gradient is negligible and \(\mathbf{d}=t\mathbf{v}_i\), the quadratic term is \(\frac{1}{2}\lambda_i t^2\), because \(\mathbf{v}_i^{T}\mathbf{v}_i=1\). Thus a small positive \(\lambda_i\) means that moving a given distance in that direction produces only a small second-order increase in the objective. A negative \(\lambda_i\) means the quadratic approximation decreases in that direction, which is inconsistent with a well-resolved interior local minimum.
The eigenvector is as important as the eigenvalue. Its entries describe the relative changes in the parameters that form the direction. The sign of an eigenvector is arbitrary: \(\mathbf{v}_i\) and \(-\mathbf{v}_i\) represent the same direction. Interpret the relative pattern of its entries, not the choice of sign.
Worked Example: A Weak Joint Direction
Worked Example: A Weak Joint Direction
Suppose a hypothetical two-parameter model is fitted in standardized coordinates \(z_1\) and \(z_2\), where \(z_1\) represents a scaled change in log clearance and \(z_2\) a scaled change in log volume. At an estimate with a negligible gradient, the computed Hessian is
For a matrix with equal diagonal elements \(a\) and equal off-diagonal elements \(b\), the vectors \((1,1)^T\) and \((1,-1)^T\) are eigenvectors. Substitution gives the eigenvalues \(a+b=0.0010\) and \(a-b=10.000\), respectively. Normalizing the vectors gives \(\mathbf{v}_1=(1,1)^T/\sqrt{2}\) for the small eigenvalue and \(\mathbf{v}_2=(1,-1)^T/\sqrt{2}\) for the large one.
The weak direction is therefore a joint increase or joint decrease in the two standardized parameters. Along a unit step in that direction, the quadratic approximation changes by \(\frac{1}{2}(0.0010)(1)^2=0.000500\) objective units. Along the other unit direction, the change is \(\frac{1}{2}(10.000)(1)^2=5.000\) objective units. The fitted objective is locally much less sensitive to the joint direction than to the contrasting direction. This does not mean either parameter alone is necessarily weakly informed; the weak direction is a particular combination.
Because these coordinates are standardized, the comparison is interpretable within the chosen scaling. The numerical value \(0.0010\) is not a universal cutoff for declaring a direction weak.
Why the Curvature Interpretation Works
The Hessian’s eigenvectors provide a set of orthogonal directions in which the quadratic approximation separates into independent curvature terms. For a symmetric Hessian, write \(H=Q\Lambda Q^T\), where the columns of \(Q\) are unit eigenvectors and \(\Lambda\) is diagonal with eigenvalues \(\lambda_i\). Express a small displacement as \(\mathbf{d}=Q\mathbf{a}\), where \(a_i\) is the displacement along eigenvector \(i\). Then
The first equality substitutes \(H=Q\Lambda Q^T\); the second uses \(Q^TQ=I\). This decomposition justifies reading each eigenvalue as curvature along its associated direction. If every eigenvalue is positive, the quadratic approximation is locally bowl-shaped. If any eigenvalue is negative, it bends downward along that eigenvector; if an eigenvalue is zero, the quadratic approximation has no curvature in that direction.
For a regular likelihood model at an interior optimum, a positive-definite Hessian is also required for the usual local covariance approximation. If \(F\) is the negative log likelihood, that approximation is \(H^{-1}\). If the minimized objective is instead \(F=-2\log L\), its Hessian is twice the negative-log-likelihood Hessian, so the corresponding approximation is \(2H^{-1}\). These formulas rely on regularity and a trustworthy local quadratic approximation; they are not a general guarantee that the model is well determined.
Worked Example: A Negative Eigenvalue
Worked Example: A Negative Eigenvalue
In a second hypothetical fit, the gradient is reported as small, but the Hessian in two scaled coordinates is
The normalized eigenvector \((1,-1)^T/\sqrt{2}\) has eigenvalue \(3.99-4.01=-0.0200\); the normalized eigenvector \((1,1)^T/\sqrt{2}\) has eigenvalue \(3.99+4.01=8.00\). A unit step along the negative-curvature direction gives the quadratic change
Under an accurate quadratic approximation at an interior stationary point, the objective would decrease by about 0.0100 units along this direction. That is not the curvature pattern expected at a local minimum. The result calls for investigation, not an immediate claim that the model has a scientifically meaningful downhill direction. The Hessian may be inaccurate, the gradient may not actually be negligible, or the optimizer may not have reached a local minimum.
A suitable check is to evaluate the objective at small positive and negative displacements along the reported eigenvector and compare the changes with the quadratic prediction. Repeat the Hessian calculation with tighter numerical tolerances or a different finite-difference step, if applicable. If the negative eigenvalue persists and direct evaluations confirm local decrease, treat the fitted point as not established as an interior minimum. If its sign changes with numerical settings and the predicted changes are below objective-evaluation noise, the sign is not reliable evidence of negative curvature.
Scaling, Numerical Stability, and Comparisons
Eigenvalues have units tied to the parameter coordinates and the objective scale. Changing units or rescaling parameters changes the numerical Hessian and can change its eigenvalues. Under a linear coordinate change \(\boldsymbol{\theta}=D\mathbf{z}\), the Hessian at a stationary point transforms as \(H_{\mathbf{z}}=D^TH_{\boldsymbol{\theta}}D\). Consequently, an absolute threshold applied to raw eigenvalues is not meaningful without specifying the coordinates and objective convention. Multiplying the objective by a positive constant also multiplies every eigenvalue by that constant, without changing the eigenvectors or the relative curvature pattern.
Worked Example: The Same Curvature in Different Coordinates
Suppose a hypothetical fit has a Hessian \(H_{\mathbf{z}}=\begin{pmatrix}4&0\\0&0.0400\end{pmatrix}\) in coordinates \(z_1,z_2\). Its eigenvalues are 4 and 0.0400, so the second coordinate is locally weaker on this scale. Now define coordinates by \(z_1=10u_1\) and \(z_2=u_2\). Here \(D=\operatorname{diag}(10,1)\), so
The eigenvalues are now 400 and 0.0400. The underlying fit has not gained information; the first coordinate was simply rescaled. The ratio of the larger to smaller eigenvalue changed from \(4/0.0400=100\) to \(400/0.0400=10{,}000\). This example shows why investigators should inspect the parameterization and use stated, sensible scales before comparing eigenvalue magnitudes or condition measures.
In practice, a useful sequence is to inspect the eigenvectors in scientifically interpretable or scaled coordinates, then check whether the weak direction persists under reasonable numerical settings and objective evaluations. Scaling can make the diagnostic easier to read, but it cannot repair a likelihood that genuinely changes very little in a direction.
The main alternatives answer related but different questions. The gradient-based convergence check asks whether the objective’s first derivatives are small; it does not establish positive curvature. Hessian-based standard errors summarize local uncertainty under a quadratic approximation, but an inverse Hessian can be unstable or unavailable when eigenvalues are small or negative. A covariance-step success or failure is a useful computational signal, but does not replace inspection of which parameter combinations cause the difficulty. The eigenvalue diagnostic adds direction-specific curvature information; it does not replace those checks.
When reporting a weak direction, state the parameterization, objective convention, eigenvalue, and the important signed components of the corresponding eigenvector. Also state whether the result was stable to reasonable changes in numerical calculation. Avoid interpreting an eigenvalue without its scale, or treating a local Hessian as proof about the global likelihood surface.
Common Pitfalls and Limits
- Calling every small eigenvalue evidence of a model defect. A small number may reflect units, parameter scaling, objective scaling, or numerical error. First make those conventions explicit and check stability.
- Reading a negative eigenvalue as ordinary weak information. A negative value indicates downward local curvature in the computed Hessian. At a well-resolved interior minimum, that should not occur; investigate the estimate and calculation.
- Interpreting a direction as one parameter. The eigenvector may combine several parameters. Describe the combination rather than attributing the result to whichever component has the largest entry alone.
- Assuming positive eigenvalues guarantee reliable estimates. Positive curvature is a local property. It does not establish that the objective approximation is accurate over a useful range, or that the fitted model is otherwise appropriate.
In hierarchical pharmacometric models, the objective can itself depend on approximations and numerical integration. The Hessian is then a diagnostic of the objective actually evaluated, and instability in that evaluation can affect its eigenvalues. Report an eigenvalue result as evidence about local curvature under the stated computation, not as a standalone biological conclusion.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- At a stationary point, a unit eigenvector has eigenvalue \(0.006\). What is the quadratic objective change for a step of length 2 in this direction?
- What does a negative Hessian eigenvalue imply if the objective is being minimized, and what should be checked before concluding that the fit is not at a minimum?
- A parameter is re-expressed in different units and the Hessian eigenvalues change. Does that alone show the fit’s information content changed? Explain.
- Why can a small Hessian eigenvalue correspond to a weak combination of parameters rather than a single weak parameter?
Solutions. (1) The quadratic change is \(\frac{1}{2}\lambda t^2=\frac{1}{2}(0.006)(2^2)=0.012\) objective units, assuming a negligible gradient and a useful local quadratic approximation. (2) It indicates negative curvature in the computed direction, inconsistent with a resolved interior local minimum. Check the gradient, numerical Hessian stability, objective evaluations along the direction, and whether the fit is interior. (3) No. A linear rescaling transforms the Hessian and changes its numerical eigenvalues; inspect the scale and coordinate transformation before interpreting differences. (4) The eigenvector contains a weighted set of parameter changes. Curvature is measured for that whole direction, so a coordinated change can be weak even if separate coordinate directions are not.