Hessian Diagnostics for Likelihood Fits

Why Inspect Curvature After Fitting?

A selection-model likelihood can be maximized successfully in the computational sense while still providing little information about some combinations of its parameters. For example, a change in an outcome-model parameter may be offset by a change in a response-model parameter, leaving the likelihood almost unchanged. The resulting surface is shallow in that joint direction, even if the optimizer reports convergence and each parameter estimate appears finite.

The observed Hessian summarizes local curvature at the fitted parameter vector. Its eigenvalues describe curvature along particular combinations of parameters; its condition number indicates how uneven that curvature is; and the inverse Hessian gives the usual local covariance approximation. These are diagnostics, not guarantees of identification or reliable inference. They must be interpreted alongside the likelihood, the model parameterization, and the numerical accuracy of the fit.

Observed Hessian and Information

Let \(\theta=(\theta_1,\ldots,\theta_p)^\mathsf{T}\) denote the parameters of a specified selection model, and let \(\ell(\theta)\) be its observed-data log-likelihood. At a fitted value \(\hat{\theta}\), the observed Hessian is the matrix of second derivatives of the log-likelihood. The observed information is its negative:

The Hessian and information have opposite signs. At an interior, unconstrained local maximum of a twice-differentiable log-likelihood, \(H(\hat{\theta})\) should be negative semidefinite, or equivalently \(J(\hat{\theta})\) should be positive semidefinite. A strictly positive-definite \(J\) indicates positive curvature in every local parameter direction. This is a local condition: it does not establish that the maximum is global, that the model is identified by the data, or that the likelihood has no other modes.

Under the usual local likelihood approximation—an interior, regular maximum with an adequately specified and accurately evaluated likelihood—the estimated covariance matrix is approximated by \(J(\hat{\theta})^{-1}\). The approximate standard error for \(\hat{\theta}_j\) is the square root of the \(j\)th diagonal element of that inverse. The approximate correlation between two estimates is the corresponding covariance divided by the product of their standard errors. These calculations concern the joint parameter estimates, not the entries of \(J\) themselves.

How Curvature Produces Uncertainty

The connection between the Hessian and uncertainty follows from a local Taylor expansion. Write \(u=\theta-\hat{\theta}\). At a stationary fit, where the score is approximately zero, the log-likelihood near \(\hat{\theta}\) is

If \(J\) is positive definite, this quadratic approximation falls away from the fitted point in every direction. A small eigenvalue means that the quadratic penalty is weak along its eigenvector, so appreciable movement in that direction changes the log-likelihood very little. Inverting \(J\) turns weak curvature into large variance along that direction. This is why a flat direction can make standard errors large and can produce strong correlations between individual parameter estimates.

A useful detail in selection models is that the weak direction may combine parameters rather than coincide with one coordinate axis. Looking only at the diagonal elements of \(J\) can miss this: a matrix may have apparently substantial diagonal entries but still have a small eigenvalue because its off-diagonal entries imply near-cancellation along a particular combination.

Using the Diagnostics Carefully

A practical inspection starts by checking whether the information matrix is symmetric to numerical tolerance and whether its eigenvalues have the expected signs. Small asymmetries can arise from finite-difference calculations, even though the exact Hessian of a sufficiently smooth scalar likelihood is symmetric. A negative eigenvalue at a claimed unconstrained maximum may indicate that the optimizer has not reached a local maximum, that the likelihood or derivatives were evaluated inaccurately, or that the point is constrained or otherwise nonregular. It should not be silently replaced by its absolute value.

If all eigenvalues are positive, compare their magnitudes and inspect the condition number. A large condition number signals that inversion may be sensitive to small perturbations. There is no universal cutoff: its practical meaning depends on the numerical precision, sample size, parameter scales, and accuracy of the likelihood and derivatives. Report eigenvalues with the parameterization and computational method understood, rather than treating a threshold as a formal test of identification.

The condition number is also scale-dependent. If one parameter is measured in years and another in milliseconds, the numerical curvature can differ greatly merely because the units differ. Rescaling parameters can therefore change \(\kappa_2(J)\) without changing the underlying statistical model. Correlations and eigenvectors can also be affected by parameter scaling, though correlations are unchanged by positive rescaling of individual coordinates. In practice, inspect the parameter units, consider a dimensionless or otherwise sensible scaling as a diagnostic, and verify whether conclusions about weak directions persist. This is a diagnostic check, not a substitute for evaluating the model on its scientifically meaningful scale.

In selection models, uncertainty in a parameter of interest must account for the fact that nuisance parameters are estimated too. The local relationship between joint and profile curvature makes this explicit. Partition the information matrix into a scalar parameter of interest \(\psi\) and nuisance vector \(\lambda\):

Assume \(J_{\lambda\lambda}\) is invertible locally. At a profile optimum for each nearby \(\psi\), the nuisance score is zero. Differentiating that score with respect to \(\psi\) gives \(d\lambda/d\psi=-J_{\lambda\lambda}^{-1}J_{\lambda\psi}\). Substituting this response into the second derivative of the profile log-likelihood gives profile information

The subtracted term is nonnegative when the joint information is positive definite. Consequently, allowing nuisance parameters to adjust cannot increase the local profile curvature relative to fixing them. A parameter may appear well determined if nuisance parameters are held fixed, yet be less precisely determined once their joint uncertainty is included.

When a Hessian Warning Appears

A Hessian diagnostic is most useful when treated as a reason to investigate, not as an automatic verdict. A negative or near-zero eigenvalue should prompt checks of the score at the fitted point, the optimizer's stopping conditions, and the likelihood and derivative calculations. Where derivatives are numerical, compare reasonable finite-difference settings or derivative implementations. A matrix that changes substantially under small changes in numerical settings is not a stable basis for standard errors.

A very large condition number or near-perfect parameter correlation calls for inspection of the corresponding eigenvector or covariance structure. Ask which combinations of outcome and response parameters can change together while leaving the observed likelihood nearly unchanged. Compare the local diagnostic with the profile likelihood for the parameter of interest: the profile reveals how the likelihood behaves after nuisance parameters adjust, while the Hessian summarizes only local quadratic curvature. Agreement between them is reassuring but does not prove that the local approximation is adequate over a wider range.

The principal alternatives answer different questions. The observed Hessian is a local, fit-specific curvature diagnostic. Profile likelihood examines likelihood support for a parameter while optimizing nuisance parameters, and can reveal nonquadratic shape that a Hessian compresses into one matrix. Likelihood-ratio comparisons assess relative fit under specified constraints, but their interpretation requires suitable calibration. None of these approaches repairs a poorly specified model, a numerically inaccurate likelihood, or weak information in the data.

Hessian-based standard errors rely on local quadratic behavior and regularity conditions. They can be misleading near parameter boundaries, in highly nonquadratic likelihoods, or when identification is weak. A positive-definite Hessian does not establish that the selection-model assumptions are credible, and a numerically stable inverse does not establish that the likelihood has captured all relevant uncertainty. The diagnostic’s role is narrower: to reveal whether the fitted likelihood supports a stable local curvature summary and which parameter combinations deserve closer examination.