Why Separate the Two Failure Questions?
A model run can return parameter estimates without returning standard errors. It can also return standard errors after the optimizer stopped without meeting its convergence criteria. These are different outcomes: optimization searches for parameter values that minimize the objective function, while the covariance step uses local curvature near the reported values to approximate parameter uncertainty. A failure in one step does not, by itself, establish failure in the other.
The practical task is therefore to diagnose two questions separately. Did the optimizer provide adequate evidence that it found a suitable minimum? If so, did the covariance calculation successfully estimate local uncertainty at that point? Termination messages, objective-function and gradient diagnostics, standard errors, and curvature information answer different parts of this assessment. A single status label or a set of plausible-looking estimates is not enough.
What the Covariance Step Is Trying to Calculate
Let \(\theta\) denote the vector of estimated model parameters, and let \(F(\theta)\) be an objective function defined as minus twice the log likelihood. At an interior optimum, the gradient of \(F\) should be approximately zero. The Hessian \(H_F(\hat{\theta})\) is the matrix of second derivatives of \(F\), evaluated at the estimate \(\hat{\theta}\). It describes the local curvature of the objective function.
For a smooth likelihood, a well-behaved interior optimum, and a sufficiently accurate local quadratic approximation, the estimated covariance matrix is approximately twice the inverse Hessian when the objective is minus twice the log likelihood. This convention follows because the negative log likelihood has Hessian \(H_F/2\). Software conventions can differ, so the objective-function definition and reported covariance convention must be checked before comparing numerical values.
For this calculation to give the usual local standard errors, the objective should be sufficiently smooth near the estimate; the estimate should be an adequately converged, interior optimum; and the Hessian should be finite, symmetric to numerical accuracy, and positive definite in the parameter coordinates used. The resulting approximation also relies on the local quadratic description being informative. A matrix inverse can exist algebraically yet still be a poor uncertainty approximation if the optimization point or local curvature is unreliable.
The link between curvature and uncertainty follows from a second-order expansion. Near an interior minimum, the objective is approximately its value at the estimate plus one half of the parameter displacement multiplied by the Hessian on both sides. Equating this quadratic form to the log-likelihood scale gives a local normal approximation; its precision matrix is approximately \(H_F/2\), so its covariance matrix is approximately \(2H_F^{-1}\). This argument explains both why curvature is needed and why a valid covariance calculation cannot substitute for successful optimization.
A covariance calculation can fail if the estimated curvature is not positive definite, if numerical derivatives are unstable, or if the inversion is numerically unreliable. In those cases the software may omit standard errors, flag a covariance error, or report values that are extremely large or sensitive to numerical settings. Conversely, a covariance matrix may be returned even when the optimizer did not meet its stopping criteria. Its availability is not a certificate that the estimate is valid.
Worked Example: Converged Search, Invalid Local Curvature
Suppose a hypothetical model is fitted in two dimensionless parameter coordinates, and the optimizer reports that its convergence criteria were met. The gradient is small, but the covariance step reports that the Hessian is not positive definite. The estimated Hessian of the minus-twice-log-likelihood objective is
For a symmetric two-parameter matrix to be positive definite, its first diagonal element and determinant must both be positive. Here the first diagonal element is \(4>0\), but
Thus the matrix is not positive definite, and it cannot provide the ordinary local covariance approximation. The optimizer’s termination report and the covariance failure are not contradictory: the first concerns the stopping criteria used by the search, while the second concerns whether the estimated local curvature supports inversion. The small gradient is relevant evidence about the search, but it does not make this Hessian suitable for standard errors. The appropriate conclusion is that the point may satisfy the optimizer’s stated stopping rules, while the covariance estimate is unavailable and the local behavior needs investigation.
Read the Failure Pattern, Not Just the Final Message
A useful diagnosis records the optimizer status and covariance status separately, then checks whether the other available evidence agrees with each. The objective-function trace and gradient checks help assess the search; the Hessian and standard-error output help assess the covariance calculation. The earlier material on gradient-based convergence checks and Hessian and curvature diagnostics provides the detailed interpretation of those quantities. Here the emphasis is on how their patterns distinguish the two types of failure.
| Observed pattern | What it supports | What it does not establish |
|---|---|---|
| Optimizer reports failure; covariance is unavailable | The estimation result is not established, and uncertainty is also unavailable. | It does not show whether covariance would work at a properly converged estimate. |
| Optimizer reports convergence; covariance fails | The search met its reported stopping criteria, but local uncertainty could not be calculated reliably. | It does not prove the estimate is a scientifically adequate optimum. |
| Optimizer reports failure; covariance is returned | A local matrix was calculated at the stopping point. | It does not validate the stopping point or make its standard errors trustworthy. |
| Optimizer reports convergence; covariance and standard errors are returned | Both computational steps completed. | It does not alone establish that the model, assumptions, or approximation are adequate. |
Standard errors provide a useful signature, but not a stand-alone verdict. Missing or explicitly flagged standard errors point directly to an unsuccessful covariance calculation. Very large or unexpected standard errors can signal problematic local curvature or numerical instability, but they can also reflect genuine uncertainty. Standard errors that look plausible do not prove that optimization converged. Always check which parameters have reported standard errors, whether the covariance matrix is valid, and whether the optimizer’s termination and gradient evidence support the estimate.
Worked Example: Covariance Returned After an Optimization Limit
In a hypothetical fit, the optimizer stops at its evaluation limit before satisfying its convergence criteria. It reports estimates \(CL=5.2\) L/h and \(V=19.4\) L. For illustration, define dimensionless coordinates \(z_1=CL/(5\ \mathrm{L/h})\) and \(z_2=V/(20\ \mathrm{L})\). A covariance calculation is nevertheless completed at the stopping point, with
The determinant is \(8(5)-2(2)=36>0\), and the first diagonal element is positive, so this two-by-two Hessian is positive definite. Its inverse and the approximate covariance in \(z\)-coordinates are
The standard errors in these coordinates are \(SE(z_1)=\sqrt{0.2778}=0.5271\) and \(SE(z_2)=\sqrt{0.4444}=0.6667\). Converting back to the stated parameter units gives \(SE(CL)=5(0.5271)=2.636\) L/h and \(SE(V)=20(0.6667)=13.33\) L. The covariance calculation has produced a matrix and standard errors, but the optimizer did not establish convergence. These values describe curvature at the reported stopping point; they should not be presented as validated uncertainty estimates for a confirmed optimum. The next diagnostic action should address the optimization failure, not treat covariance availability as proof that the fit succeeded.
When Numerical Curvature Is Unstable
Some covariance failures arise because the curvature calculation is sensitive to numerical details. A Hessian is often estimated from derivatives or finite differences, so small changes in numerical step size, precision, or objective-function evaluation can change the estimated second derivatives. Warning signs include materially different Hessian entries under reasonable derivative settings, a matrix that is not symmetric to numerical accuracy, or a change from positive definite to non-positive definite curvature under small numerical perturbations.
These signs call for investigation rather than an automatic conclusion about the parameter estimates. A covariance failure can occur after the search has found a stable-looking point, especially when curvature estimation is numerically fragile. But a convergence label alone is not enough to declare the point reliable. Check whether objective-function evaluations are reproducible near the estimate, whether numerical settings are adequate, and whether the optimization diagnostics remain satisfactory when the fit is repeated. Change one relevant setting at a time so that a change in output can be interpreted.
Worked Example: Step-Sensitive Covariance Calculation
A hypothetical fit reports satisfactory optimizer termination and gradient checks. The covariance routine estimates the Hessian using two reasonable finite-difference step settings. At the first setting it gives
This matrix is positive definite because its first diagonal element and determinant are positive. At the second setting, the result is
The second matrix is not positive definite, and the two estimates imply materially different curvature assessments. Therefore, the covariance result is not robust to the numerical differentiation setting in this example. The optimizer status does not resolve this discrepancy; nor does one positive-definite matrix justify ignoring the other. The diagnosis is a numerically unstable covariance calculation. Check local objective evaluations and derivative settings, then reassess whether the curvature and resulting standard errors are reproducible. Until they are, report the covariance step as unresolved rather than selecting whichever output looks more favorable.
A Practical Diagnostic Sequence
A disciplined sequence keeps the two questions separate and avoids changing many settings without learning which part of the workflow is affected.
Write down the optimizer termination message and the covariance-step message separately. Note whether parameter estimates, a covariance matrix, and standard errors were actually produced.
Check the stated stopping criteria, gradient evidence, and objective-function behavior. If the search did not meet its criteria, treat the estimate as unconfirmed even if a covariance matrix is available.
Check whether the Hessian is finite and adequately symmetric, whether its curvature supports inversion, and whether standard errors are complete and reproducible.
For an optimization failure, investigate the search and its termination. For a covariance-only failure, investigate curvature estimation and numerical evaluation at the reported point. If both failed, resolve estimation before interpreting local standard errors.
The main alternative to a Hessian-based covariance calculation is an uncertainty method based on repeated refitting, such as a bootstrap, where appropriate and computationally feasible. Such methods have their own assumptions and failure modes; they do not repair an optimizer that has not reliably fitted the model. Re-running the covariance calculation with different numerical settings is a diagnostic comparison, not independent confirmation that the model is valid. The method used should be reported clearly, along with any unresolved estimation or covariance warnings.
Common Mistakes and Limitations
- “The estimates are present, so estimation succeeded.” Estimates can be printed at a stopping point that did not meet convergence criteria. Check the optimizer status and diagnostics.
- “The covariance step failed, so the optimizer failed.” The search and covariance calculation are distinct. A covariance-only failure does not automatically invalidate every feature of the optimization, though it leaves standard errors unresolved.
- “Standard errors were returned, so the fit converged.” A covariance matrix may be calculated at a nonconverged point. Its availability does not establish a valid optimum.
- “A large standard error proves a covariance bug.” Large uncertainty can reflect data and model information as well as computational problems. Check the matrix, termination evidence, and numerical stability before deciding.
- “A successful status proves the model is adequate.” Convergence and covariance calculations are computational diagnostics, not guarantees of correct model specification or scientific interpretation.
Local Hessian-based standard errors are approximations. They summarize curvature near one fitted point and can be misleading when the local quadratic approximation is poor, the estimate is not interior, or numerical calculations are unstable. A covariance-step failure should be reported as such; it should not be converted into an unqualified uncertainty statement. Equally, a covariance-step success should not conceal a failed optimization.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- An optimizer meets its stated stopping criteria, but the covariance routine reports a non-positive-definite Hessian and omits standard errors. Which step failed, and what remains unresolved?
- An optimizer stops at its evaluation limit, but a positive-definite covariance matrix and standard errors are printed. Does this establish a valid estimate? Why or why not?
- For an objective defined as minus twice the log likelihood, a two-parameter Hessian is \(\begin{pmatrix}5&1\\1&4\end{pmatrix}\). Calculate the approximate covariance matrix and standard errors in the parameter coordinates.
- The covariance routine gives different curvature classifications under two reasonable numerical derivative settings. What is the most appropriate immediate interpretation?
Solutions. (1) The covariance step failed. The local Hessian does not support the usual covariance calculation, so standard errors are unresolved; the optimizer status is separate evidence that its stopping criteria were met, not proof that every aspect of the fit is adequate. (2) No. The covariance calculation was performed at a stopping point where convergence was not established, so its availability does not validate the estimate or its standard errors. (3) The determinant is \(5(4)-1(1)=19\), so the matrix is positive definite. Its inverse is \(\frac{1}{19}\begin{pmatrix}4&-1\\-1&5\end{pmatrix}\). Thus the approximate covariance is \(\frac{2}{19}\begin{pmatrix}4&-1\\-1&5\end{pmatrix}=\begin{pmatrix}0.4211&-0.1053\\-0.1053&0.5263\end{pmatrix}\); the standard errors are \(\sqrt{0.4211}=0.6489\) and \(\sqrt{0.5263}=0.7255\), in the respective parameter units. (4) Treat the covariance calculation as numerically unstable and investigate reproducibility of objective evaluations and derivative settings. Do not select the favorable result without resolving the discrepancy.