Hessian Diagnostics And Uncertainty Estimates

Why Inspect Curvature After a Fit?

A fitted joint model can return parameter estimates even when the uncertainty attached to them is unreliable. The fitted values describe a location in parameter space; the local curvature of the objective describes how sharply the likelihood distinguishes nearby values. If the likelihood is nearly flat in some direction, small numerical changes can produce large changes in estimated uncertainty. If the curvature has the wrong sign at a claimed interior maximum, the fit may not be at a local maximum at all.

Hessian diagnostics are therefore a check on the local geometry behind standard errors, not a proof that the model is correctly specified or that its parameters are practically identifiable. They complement checks of the reported gradient and the earlier material on structural and practical identifiability.

Hessian and Observed Information

Let \(\theta=(\theta_1,\ldots,\theta_p)^\mathsf{T}\) be the vector of free model parameters, and let \(\ell(\theta)\) be the log-likelihood being optimized. In a joint model, this is typically the likelihood for the observed longitudinal and event data, with latent random effects integrated out or otherwise handled by the specified estimation method. At a fitted estimate \(\hat{\theta}\), the Hessian is the matrix of second derivatives:

Here \(J(\hat{\theta})\) is the observed information. For an unconstrained, regular, interior maximum, the likelihood should curve downward locally: \(H\) is negative definite, equivalently \(J\) is positive definite. Positive definiteness means \(v^\mathsf{T}Jv>0\) for every nonzero direction \(v\). The fitted maximum should also have a gradient near zero; curvature cannot compensate for a fit that has not reached a stationary point.

This approximation relies on a smooth likelihood, a well-defined interior maximum, local identifiability, a nonsingular information matrix, and the usual large-sample regularity conditions. It is a local, approximately normal description of sampling uncertainty. It is not appropriate to interpret an inverse as a trustworthy covariance estimate merely because software was able to calculate one.

Why the Inverse Information Gives a Local Uncertainty Estimate

Write \(\delta=\theta-\hat{\theta}\). A second-order Taylor expansion around a stationary estimate gives a local approximation:

Thus, the local log-likelihood drops according to the quadratic form involving \(J\). Under the usual likelihood approximation, the corresponding likelihood contours are approximately ellipses, and the covariance matrix describing their local scale is \(J^{-1}\). A direction with small information has a large variance in the inverse. This also explains why strong correlation between parameters can make individual standard errors large: uncertainty may be concentrated along a particular combination of parameters.

The approximation is local. It can be misleading if the likelihood is skewed, has multiple separated modes, approaches a parameter boundary, or is so flat that higher-order terms matter over the range of plausible values.

Definiteness and Conditioning Are Different Checks

Definiteness asks whether the curvature has the correct sign in every direction. Conditioning asks how sensitive matrix inversion is to small perturbations. A matrix can be positive definite yet so poorly conditioned that its inverse is unstable. For a symmetric positive-definite matrix, its eigenvalues are positive. A very small eigenvalue indicates a direction with little curvature; the condition number, commonly the largest eigenvalue divided by the smallest, summarizes the relative disparity in curvature.

Eigenvectors help locate the combinations of parameters that are weakly informed. Inspecting only the diagonal of \(J\) can miss this: two parameters may each appear to have substantial curvature while a contrast between them is nearly flat. The eigenvalue scale also depends on parameter units. Rescaling parameters can improve numerical conditioning, but it cannot create information that the data do not contain.

Practical Diagnostic Workflow

A materially negative eigenvalue of observed information means there is a direction of upward log-likelihood curvature, inconsistent with a local maximum in an unconstrained interior parameterization. Possible explanations include a saddle point, an unfinished optimization, numerical error in the Hessian, or a boundary or constraint that changes the relevant geometry. A small negative eigenvalue should be investigated rather than automatically clipped to zero or replaced by its absolute value.

When estimates lie on a boundary, ordinary inverse-Hessian standard errors can fail even if the constrained optimum is legitimate. For example, a variance component constrained to be nonnegative may be estimated as zero. The boundary changes the sampling distribution, and the unconstrained interior approximation need not apply. A transformation such as a logarithm can enforce positivity for parameters strictly above zero, but it cannot make an estimate at the boundary an interior point of the original parameter space.

Alternatives, Limitations, and Common Mistakes

The inverse observed information is a convenient local, likelihood-based uncertainty estimate. A profile-likelihood interval instead examines the likelihood over values of one parameter while optimizing the others; it can reveal asymmetry or a flat region that a quadratic approximation conceals. A sandwich covariance estimate targets a different issue, such as variance estimation under certain forms of model misspecification, and does not repair a singular or poorly identified likelihood. Resampling approaches can assess variability under their own assumptions, but they are not a reason to ignore a pathological Hessian.

The Hessian is also sensitive to how the objective is computed. Finite-difference derivatives, rounding, and numerical approximation can perturb second derivatives, especially when curvature is small. Rescaling parameters may make calculations more stable and aid interpretation, but the corresponding uncertainty must be transformed back to the desired parameter scale. For a smooth transformation \(\phi=g(\theta)\), the local covariance is approximately \(GJ^{-1}G^\mathsf{T}\), where \(G\) is the Jacobian of \(g\) evaluated at \(\hat{\theta}\).

• Mistake: “The optimizer returned standard errors, so the Hessian must be reliable.” A routine may invert an unstable matrix or apply a numerical correction. Check eigenvalues, conditioning, and sensitivity rather than relying on successful output alone.
• Mistake: “Positive diagonal entries prove positive definiteness.” They do not. Off-diagonal entries can create a negative or nearly zero eigenvalue, so assess the full matrix.
• Mistake: “A non-positive eigenvalue always means the scientific model is invalid.” It may instead indicate an unfinished fit, a numerical derivative problem, or boundary geometry. Diagnose the cause before drawing a scientific conclusion.
• Mistake: “A small condition number can be judged without considering units.” Conditioning depends on parameter scaling. Inspect scaled curvature and parameter combinations, while remembering that rescaling does not add data information.