Why Curvature Can Hide a Difficult Fit
A GLMM optimizer can report convergence even when the likelihood is nearly flat in one particular direction. In that case, several combinations of parameter values may fit almost equally well. The individual parameters can be strongly coupled: changing one while adjusting another may leave the objective almost unchanged. Looking only at separate standard errors or at the optimizer’s convergence message can miss this structure.
The Hessian describes local curvature, but its eigenvalues and eigenvectors reveal more than a list of diagonal entries. An eigenvalue measures curvature along a particular direction through the joint parameter space; its eigenvector gives the corresponding parameter combination. A very small eigenvalue therefore points to a direction in which the fitted objective changes little. This is evidence of weak local identification or numerical instability, not by itself proof of a model defect or a particular cause.
Definition and Local Interpretation
These definitions assume that the objective is twice differentiable near the fitted point and that the Hessian is computed for the same parameterization and objective used in the fit. For a smooth interior local minimum, the Hessian should be positive semidefinite; a strictly positive-definite Hessian indicates positive curvature in every local direction. GLMM objectives may be evaluated using numerical integration or approximations, so the computed Hessian can also contain numerical error. At constrained or boundary estimates, ordinary unconstrained curvature interpretations require extra care.
For a small displacement \(\boldsymbol\delta\), a second-order Taylor expansion gives
At a fitted stationary point, the gradient term is approximately zero. If \(\boldsymbol\delta=a\mathbf v_j\), with \(\mathbf v_j\) a unit eigenvector, then the quadratic change is approximately \(\tfrac12\lambda_j a^2\). Thus, a small positive \(\lambda_j\) means that a displacement of a given size along that direction produces little change in the objective. The eigenvector, not the eigenvalue alone, tells which parameters move together to create that weak direction.
Worked Example: A Weak Contrast Between Two Coefficients
Worked Example: A Weak Contrast Between Two Coefficients
Suppose a hypothetical logistic random-intercept GLMM has two fixed-effect coefficients, an intercept \(\beta_0\) and a treatment coefficient \(\beta_1\). At an interior fitted point, suppose the negative-log-likelihood Hessian in these coordinates is
The characteristic equation is
so the eigenvalues are \(1\) and \(199\). The unit eigenvector for \(\lambda=1\) is \(\mathbf v_{\min}=(1,-1)^\mathsf{T}/\sqrt{2}\), while the one for \(\lambda=199\) is \(\mathbf v_{\max}=(1,1)^\mathsf{T}/\sqrt{2}\). The condition number is \(199/1=199\).
The locally weak direction changes the intercept and treatment coefficient in opposite directions. For a displacement of size \(a=0.2\) along \(\mathbf v_{\min}\), the quadratic approximation to the objective change is \(\tfrac12(1)(0.2)^2=0.0200\). The same-size displacement along \(\mathbf v_{\max}\) gives \(\tfrac12(199)(0.2)^2=3.98\). Locally, the objective is much flatter along the first combination. This indicates that the data and fitted model constrain that combination less strongly than the other one; it does not establish that either coefficient is individually unidentified.
The interpretation is specific to these coefficient coordinates and their units. If the variables or parameter coordinates were represented differently, the numerical eigenvectors and condition number could change. The calculation diagnoses this fitted representation; scientific interpretation still requires naming the actual parameters and their scales.
Condition Numbers and Poor Identification
For a positive-definite Hessian, the condition number compares its largest and smallest eigenvalues. A large value means curvature differs greatly across directions: the objective is much steeper in some combinations than in others. Near a minimum, this can make numerical optimization sensitive to rounding, finite-difference error, or small changes in the data. There is no universal cutoff that makes a particular condition number a failure; it is a warning whose importance depends on the computation and parameter scales.
When the likelihood approximation is locally quadratic and regularity conditions are adequate, the inverse Hessian approximates the covariance matrix of the fitted parameter vector. The eigenvalues of \(H^{-1}\) are \(1/\lambda_j\), with the same eigenvectors. Therefore, a small Hessian eigenvalue corresponds to a large approximate variance in that direction. This connection is useful, but it is not a guarantee that standard errors or intervals are reliable: a local quadratic approximation can be poor, especially near a boundary or when the likelihood is markedly non-quadratic.
Worked Example: Finding the Weakest Combination in a Three-Parameter Fit
Consider a hypothetical GLMM fit with parameters \(\theta_1,\theta_2,\theta_3\). Suppose the Hessian has eigenvalues \(0.0004\), \(4\), and \(25\), and that a numerical eigendecomposition gives the unit eigenvector \(\mathbf v_{\min}=(0.707,-0.707,0)^\mathsf{T}\) for the smallest eigenvalue. The condition number is
The weak direction is approximately an increase in \(\theta_1\) paired with an equal decrease in \(\theta_2\), with little change in \(\theta_3\). For a displacement of size \(a=1\) along this unit direction, the quadratic objective change is \(\tfrac12(0.0004)(1)^2=0.0002\). Along the largest-curvature direction, the corresponding change for a unit displacement is \(\tfrac12(25)(1)^2=12.5\). The very large ratio and near-flat direction warrant investigation of the parameter combination.
Under a reliable local quadratic approximation, the approximate variance along the weak direction is \(1/0.0004=2{,}500\), giving an approximate standard deviation of \(\sqrt{2{,}500}=50\) in the coordinate distance along that unit vector. This is not the standard error of each individual parameter: the direction combines parameters. The units and meaning of the combination depend on how \(\theta_1\) and \(\theta_2\) are defined.
Reading Warning Signs Carefully
A very small positive eigenvalue and a negative eigenvalue are different warnings. A small positive eigenvalue is consistent with a local minimum that is nearly flat in one direction. A materially negative eigenvalue means the objective curves downward along some direction, which is inconsistent with a strict local minimum for an unconstrained smooth objective. It may indicate that the optimizer has not reached a minimum, that the Hessian is inaccurate, or that the point lies in a constrained or boundary setting where the unconstrained interpretation does not apply.
Worked Example: A Negative Eigenvalue Is Not a Condition Number
Suppose the eigenvalues of a numerically computed Hessian at a proposed fit are \(-0.03\), \(2\), and \(15\). Since one eigenvalue is negative, this Hessian is not positive definite. The ratio \(15/(-0.03)=-500\) is not a meaningful positive-definite condition number. Reporting it as evidence of a “condition number of 500” would hide the fact that the local curvature has the wrong sign in one direction.
The appropriate conclusion is that the Hessian does not confirm a strict unconstrained local minimum. The next checks are whether the optimizer has actually settled, whether the gradient is small, and whether the Hessian calculation is stable to reasonable numerical changes. If the estimate is constrained at a boundary, the unconstrained Hessian alone is not decisive. The value \(-0.03\) is small relative to the largest curvature \(15\), so numerical error is a possibility, but its size alone does not prove that explanation. Recomputing or checking the objective locally is needed before drawing a stronger conclusion.
Practical Checks and Limits
Eigenanalysis is most useful when reported with the coordinate system, the spectrum, the condition number (when the Hessian is positive definite), and the weak eigenvector expressed in parameter names. A condition number without the eigenvector says that curvature is uneven, but not which parameter combination is implicated. A minimum eigenvalue without the scale of the parameters can also be hard to interpret.
The condition number depends on the parameterization and units. Expressing a predictor in different units changes the numerical scale of its coefficient, and a smooth change of parameter coordinates can alter the Hessian spectrum. Thus, a large condition number is not an invariant measure of intrinsic statistical identifiability. It can reflect a genuinely weakly constrained combination, poor numerical scaling, or both. Identifying the eigenvector and checking how the model is parameterized are essential before assigning a cause. Adjusting predictor scales is a separate practical intervention; the diagnostic itself does not prescribe a particular rescaling.
Use eigenvalue diagnostics alongside, rather than instead of, the checks on the fitted solution. In particular, a small gradient addresses whether the objective is locally stationary, while the Hessian addresses local curvature. Neither one alone ensures that the model is scientifically appropriate or that the likelihood approximation is adequate. Near-zero variance estimates and singular random-effects covariance structures require boundary-aware interpretation; a small eigenvalue by itself does not classify either case.
A useful review sequence is:
Identify the objective, parameterization, and whether the fitted point is interior or constrained. The usual positive-definite minimum interpretation applies directly to a smooth interior problem.
For a positive-definite Hessian, calculate the largest-to-smallest eigenvalue ratio. Treat a very small eigenvalue as a warning; do not use that ratio as a positive-definite condition number if an eigenvalue is negative.
Describe which named parameters increase or decrease together. This identifies the locally weak combination more directly than naming a single parameter based on a large standard error.
Compare with convergence and gradient checks, assess whether the Hessian calculation is stable, and interpret results in light of parameter units and possible boundaries.
The main alternative is to rely only on marginal standard errors or optimizer messages. Those summaries are convenient, but marginal standard errors can conceal dependence among parameters, and a convergence message does not describe curvature. Eigenvectors expose joint directions; they complement rather than replace those summaries. A local Hessian is also limited: if the objective is strongly non-quadratic, local curvature may not summarize behavior farther from the fitted point.
Common Mistake
Check Your Understanding
Answer each question using the local-curvature interpretation. Brief solutions follow.
- A positive-definite Hessian has eigenvalues \(2\) and \(50\). What is its condition number, and which direction is locally flatter?
- For a unit eigenvector with eigenvalue \(0.02\), what is the quadratic approximation to the objective change for a displacement of size \(a=0.5\) along that direction?
- Why does a large condition number not, by itself, prove intrinsic statistical non-identifiability?
- A Hessian has one negative eigenvalue at a proposed interior fit. Is it appropriate to report the ratio of the largest eigenvalue to the negative eigenvalue as the positive-definite condition number?
Solutions. (1) The condition number is \(50/2=25\); the eigenvector for eigenvalue \(2\) is the flatter direction. (2) The change is approximately \(\tfrac12(0.02)(0.5)^2=0.0025\) objective units. (3) The ratio depends on parameter units and parameterization, and may reflect numerical scaling as well as weak statistical information. (4) No. A negative eigenvalue means the Hessian is not positive definite, so that ratio is not the positive-definite condition number; investigate stationarity, numerical accuracy, and constraints.