Why the Covariance Step Can Fail After Optimization
An optimizer can report that it has stopped, yet the software may be unable to calculate standard errors or a covariance matrix. These are related but distinct tasks. Optimization searches for a parameter vector with a suitably small gradient; the covariance step uses local curvature around that vector. A covariance-step failure therefore does not, by itself, show whether the optimizer found a minimum, a saddle point, a nearly flat region, or a numerically unreliable approximation to curvature.
This tutorial uses the objective \(Q(\theta)=-2\ell(\theta)\), where \(\ell(\theta)\) is the log-likelihood and \(\theta\) is the vector of estimated parameters. Some software uses a different objective or scaling. The factor in a covariance approximation must match the objective convention, so check what the software reports before interpreting its Hessian.
The observed-information covariance approximation, when the estimate is an interior local minimum and the likelihood is sufficiently regular, is \(\widehat{\operatorname{Cov}}(\hat{\theta})\approx 2H(\hat{\theta})^{-1}\) for this \(Q=-2\ell\) convention. It is a local, large-sample approximation, not an exact finite-sample guarantee. It requires an invertible, positive-definite Hessian for an ordinary covariance matrix. A negative or nearly zero eigenvalue makes that requirement fail or become unstable.
What Eigenvalues and the Condition Number Tell You
For a symmetric Hessian, its eigenvectors give directions in parameter space and its eigenvalues give the curvature along those directions. A positive eigenvalue corresponds to upward curvature in that direction for \(Q\); a negative eigenvalue corresponds to downward curvature. A zero eigenvalue means the quadratic approximation has no curvature in that direction. In practice, a very small eigenvalue relative to the others can be nearly as troublesome as zero because inversion magnifies it.
If all eigenvalues are positive, define the spectral condition number as \(\kappa(H)=\lambda_{\max}/\lambda_{\min}\), where \(\lambda_{\min}\) and \(\lambda_{\max}\) are the smallest and largest eigenvalues. A large value indicates that curvature differs greatly across directions and that the inverse is sensitive to small perturbations. There is no universal cutoff that diagnoses the cause. Moreover, raw Hessian condition numbers depend on parameter units: rescaling a parameter changes the numerical Hessian. Inspect curvature in sensible scaled coordinates as well as in the software's reported coordinates.
These tests assume a symmetric Hessian evaluated at the point of interest. At an exact interior stationary point, positive definiteness is the standard second-order condition for a strict local minimum. If the gradient is merely small, not exactly zero, eigenvalues still describe curvature of the local quadratic approximation, but they do not alone prove that the point is a stationary minimum. At a parameter bound, the unconstrained second-order test is not sufficient: feasible directions and the boundary conditions matter.
A Short Derivation: Why Inversion Needs Positive Curvature
A second-order Taylor expansion around an estimate \(\hat{\theta}\), for a displacement \(h\), is
The linear term vanishes only when the gradient is exactly zero. Thus, the quadratic-only expression is justified at an exactly stationary estimate, or as an approximation when the gradient term is explicitly negligible for the displacements being considered. If \(H v=\lambda v\) for a unit eigenvector \(v\), then at an exactly stationary point, setting \(h=t v\) gives the second-order change \(\tfrac12\lambda t^2\). A negative \(\lambda\) therefore gives a direction of local decrease in \(Q\), inconsistent with an interior local minimum.
The covariance link follows from the same local expansion. Since \(Q=-2\ell\), the log-likelihood near a stationary maximum is approximately \(\ell(\hat{\theta})-\tfrac14 h^\mathsf{T}Hh\). Matching this quadratic form to a multivariate normal log-density, whose quadratic term is \(-\tfrac12 h^\mathsf{T}\Sigma^{-1}h\), gives \(\Sigma^{-1}\approx H/2\), or \(\Sigma\approx2H^{-1}\). This derivation also shows the limitation: if curvature is nonpositive or too poorly determined to invert stably, the usual covariance approximation is not trustworthy.
Worked Example: A Negative Eigenvalue Indicates a Saddle
Suppose a hypothetical two-parameter fit has an exactly stationary estimate and the Hessian of \(Q=-2\ell\) there is \(H=\begin{pmatrix}8&0\\0&-0.5\end{pmatrix}\). The eigenvalues are \(8\) and \(-0.5\), with the second coordinate direction corresponding to the negative eigenvalue.
For a displacement \(h=(0,t)^\mathsf{T}\), the gradient term is zero because the estimate is exactly stationary, and the quadratic change is \[ \Delta Q \approx \frac{1}{2}h^\mathsf{T}Hh =\frac{1}{2}(-0.5)t^2=-0.25t^2. \] For \(t=0.1\), this gives \(\Delta Q\approx-0.0025\). The local quadratic approximation predicts a decrease in \(Q\) along that direction, so the point is not an interior local minimum; it is a saddle under this Hessian pattern. Direct inversion would also produce a negative diagonal entry in \(H^{-1}\), which cannot be a valid covariance variance. The appropriate response is to investigate the fitted point and curvature calculation, not to report that inverse as a covariance matrix.
Flat Directions and Unstable Covariance Estimates
When every eigenvalue is positive but the smallest is tiny, the point may be a local minimum while the covariance calculation is still practically unstable. The corresponding eigenvector identifies a combination of parameters that the objective constrains weakly near the estimate. In an NLME model, that direction may combine fixed effects, variance components, or other parameters; the eigenvector is more informative than labeling one parameter “the problem” without checking its components.
The condition number measures relative ill-conditioning, not absolute information. It can be large because one direction is very flat, because another is very steep, or both. Changing units can change the raw condition number, so compare parameters after meaningful scaling. A flat direction can be consistent with weak identification, but the Hessian alone does not establish the scientific cause. Use the model structure and the other diagnostics, including gradient and multiple-start information, to interpret it.
Worked Example: A Positive but Ill-Conditioned Hessian
Consider a hypothetical scaled-coordinate Hessian \(H=\begin{pmatrix}20&0\\0&0.5\end{pmatrix}\) at an interior stationary estimate. Its eigenvalues are \(20\) and \(0.5\), so it is positive definite, while \(\kappa(H)=20/0.5=40\). The Hessian is invertible, although curvature is stronger in one direction.
Under the stated \(Q=-2\ell\) convention, the local covariance approximation is \[ 2H^{-1}=2\begin{pmatrix}1/20&0\\0&1/0.5\end{pmatrix} =\begin{pmatrix}0.1&0\\0&4\end{pmatrix}. \] The corresponding approximate standard errors in these scaled coordinates are \(\sqrt{0.1}=0.3162\) and \(\sqrt{4}=2.000\). These are not standard errors in original units unless the coordinates were already in those units. Here the covariance step is mathematically possible, but the second direction is much less precisely estimated locally. This example illustrates why positive definiteness and good conditioning are different questions.
Covariance-Step Messages and Numerical Precision
Common messages include “Hessian not positive definite,” “singular Hessian,” “covariance step failed,” or a warning that standard errors could not be computed. These messages describe a failed operation or a property of the numerical approximation; they are not interchangeable diagnoses. A non-positive-definite Hessian may indicate a saddle, an estimate near a constrained boundary, or an inaccurate Hessian. A singular or nearly singular Hessian points to noninvertibility or sensitivity. A software message alone rarely distinguishes these causes.
Numerical differentiation can add another failure mode. If the Hessian is obtained by finite differences, the calculation balances truncation error against rounding and cancellation error. A step that is too large may poorly approximate local curvature; a step that is too small may make differences of nearly equal objective values dominated by floating-point precision. Parameter scaling affects the appropriate step size. Analytic or automatic derivatives can help, but they do not fix a genuinely flat likelihood direction.
Worked Example: A Precision Problem Creates Apparent Zero Curvature
Suppose a hypothetical one-dimensional objective near its minimum is \(Q(\theta)=100{,}000{,}000+\theta^2\), so the exact second derivative is \(Q''(\theta)=2\). A central finite-difference estimate at \(\theta=0\) with step \(h\) is \[ \widehat{Q''}(0)=\frac{Q(h)-2Q(0)+Q(-h)}{h^2}. \]
If \(h=10^{-5}\), the added term \(h^2=10^{-10}\) is smaller than the spacing of representable double-precision numbers near \(10^8\) (about \(1.5\times10^{-8}\)). In this hypothetical computation the three objective values can therefore all round to \(100{,}000{,}000\), giving \(\widehat{Q''}(0)=(0)/10^{-10}=0\), although the exact curvature is 2. The resulting apparent zero eigenvalue makes inversion fail, but it does not demonstrate a truly flat objective.
With \(h=0.01\), the exact objective values are \(100{,}000{,}000.0001\), \(100{,}000{,}000\), and \(100{,}000{,}000.0001\). Ignoring rounding, the numerator is \(0.0002\), and \(0.0002/(0.01)^2=2\). The computation is much better resolved at this step size in this simple example. In an actual model, however, a larger step can introduce approximation error; compare sensible step sizes and scales rather than assuming that one step fixes the problem.
A Practical Diagnostic Sequence
Interpret Hessian and covariance evidence alongside the optimizer's stopping information. The earlier material on gradient diagnostics is useful here: a small gradient supports, but does not by itself establish, that the estimate is stationary. Use the following sequence to keep the diagnoses distinct.
Confirm the objective scaling, parameter units, transformations, and whether any estimate is at or near a bound. A Hessian in transformed coordinates must be interpreted in those coordinates.
A materially negative eigenvalue at a purported interior stationary point rules out a local minimum under the quadratic approximation. Small negative values should be checked for numerical sensitivity before being treated as decisive.
Find the eigenvector for the weakest direction, compute the condition number in appropriately scaled coordinates, and determine whether the direction is stable to reasonable numerical settings.
Compare Hessians across suitable finite-difference steps or derivative methods, and verify that parameter scaling is sensible. A covariance matrix that changes dramatically under small computational changes is not a reliable inferential result.
If a covariance step remains unsuccessful, report that standard errors were unavailable or unreliable. Do not replace the failed matrix with an ordinary inverse that has negative variances or is numerically unstable.
Distinguishing Causes and Recognizing Limitations
| Evidence | What it suggests | What it does not prove |
|---|---|---|
| Clear negative eigenvalue at an interior stationary point | Saddle-like local curvature; not a local minimum | Which model change will resolve it, or that the global optimum is known |
| Positive eigenvalues with a very small minimum eigenvalue | A weakly curved direction and potentially unstable covariance | A specific cause such as model misspecification |
| Curvature changes greatly with finite-difference step or scaling | Numerical approximation or parameterization may be problematic | That the underlying likelihood is well identified |
| Covariance step fails at a boundary estimate | The ordinary interior Hessian approximation may not apply | That the estimate is automatically erroneous |
The main alternative to a Hessian-based covariance approximation is to use another inferential method appropriate to the model and analysis, rather than force an unstable inverse. Profile likelihood, bootstrap, or Bayesian uncertainty summaries have different assumptions and computational demands; they are not automatic repairs for a failed Hessian. This tutorial's diagnostics concern local curvature after optimization, not model adequacy, a full global search, or a replacement uncertainty procedure.
A common mistake is to reason, “The optimizer stopped, so the Hessian must be positive definite; if covariance failed, the model is simply unidentifiable.” Both steps overstate the evidence. A stopping rule can be met at a saddle or with an inaccurate derivative, and a failed inverse can reflect finite precision, scale, constraints, or flat curvature. The correction is to inspect gradient, eigenvalues, scaling, boundary status, and numerical stability together, and to state uncertainty honestly when those checks do not support standard errors.
Check Your Understanding
Answer each question, then compare with the solutions below.
- At an exactly stationary interior estimate, the Hessian has eigenvalues \(6\) and \(-0.2\). Is it positive definite, and what does the negative eigenvalue imply?
- A Hessian has positive eigenvalues \(12\) and \(0.03\). Calculate its spectral condition number and explain what the value indicates.
- Why is the quadratic-only Taylor expansion not exact when the gradient at the estimate is nonzero?
- A covariance step reports a singular Hessian after finite differences with extremely small steps. Name one plausible numerical explanation and one check.
Solutions. (1) It is not positive definite because one eigenvalue is negative; at an exactly stationary interior point, the quadratic approximation decreases along that eigenvector, indicating saddle-like curvature rather than a local minimum. (2) \(\kappa=12/0.03=400\); curvature is relatively much weaker in one direction, so inversion may be sensitive, though the number alone does not identify the cause. (3) The expansion includes \(\nabla Q(\hat{\theta})^\mathsf{T}h\), which is generally nonzero unless the gradient is zero or the term is explicitly negligible. (4) Objective differences may be lost to floating-point rounding or cancellation; compare Hessians using sensible step sizes and scaled parameters, or another derivative calculation, and assess whether the result is stable.