Information-Matrix Eigenvalues And Identifiability

Why Look Beyond Individual Standard Errors?

In a joint model, estimates may appear stable one at a time while particular combinations of parameters remain poorly determined. For example, the longitudinal trajectory and the survival association may compensate for one another over a range of values. The likelihood can then be sharply curved in some directions through parameter space and nearly flat in another. A list of standard errors does not always make that geometry clear.

The eigenvalues and eigenvectors of the observed information matrix provide a local description of this geometry at the fitted estimate. An eigenvector identifies a direction—that is, a combination of parameter changes—and its eigenvalue measures the likelihood curvature in that direction. This gives a focused diagnostic for the local instability associated with likelihood ridges.

From Curvature to Parameter Combinations

This interpretation follows from a local quadratic approximation. Near an interior maximum, the score is zero, so a second-order Taylor expansion gives

Because the eigenvectors of a symmetric information matrix form orthonormal directions, write a small change as \(\delta=\sum_k a_kv_k\). Substitution into the quadratic term yields

Thus moving a distance \(a_k\) along \(v_k\) produces an approximate log-likelihood decrease of \(\lambda_k a_k^2/2\). For the same sized move, a direction with a small eigenvalue changes the likelihood little; a direction with a large eigenvalue changes it more. If an eigenvalue is exactly zero, the quadratic approximation is flat in that direction. That is evidence of local degeneracy, but by itself it does not prove that the likelihood is flat at all distances or that the model is globally unidentified.

This calculation relies on a twice-differentiable log likelihood, an interior estimate with a score close to zero, and a meaningful local quadratic approximation. In a joint model, \(\theta\) may include longitudinal fixed effects, survival parameters, association parameters, and covariance parameters. The eigenvector can therefore reveal a weak combination across model components, not just a weak individual coefficient.

Eigenvalues, Uncertainty, and Scale

Under the usual local likelihood approximation, if \(J(\hat{\theta})\) is positive definite, the estimated covariance matrix of the parameter estimates is approximately \(J(\hat{\theta})^{-1}\). In the eigenvector direction \(v_k\), the corresponding approximate variance is \(1/\lambda_k\), and the standard error of the projection onto that direction is \(1/\sqrt{\lambda_k}\). A small eigenvalue can therefore imply a large uncertainty for a parameter combination even when no single coefficient’s standard error seems exceptional.

This is a local approximation, not a guarantee that a Wald interval will be reliable. It is most informative when the likelihood is reasonably quadratic near the estimate and the estimate is away from parameter boundaries. For constrained or strongly nonlinear parameterizations, the relevant directions and uncertainty may be better assessed on an unconstrained or scientifically interpretable scale.

Eigenvalue magnitudes depend on units. If one parameter is measured in days and another in years, a change of one numerical unit does not represent comparable movement. A common diagnostic is to express parameters in chosen scaled coordinates. For example, set \(\theta=\hat{\theta}+Dq\), where the diagonal entries of \(D\) are scientifically meaningful parameter scales. The information in \(q\)-coordinates is \(D^\mathsf{T}JD\). Its eigenvectors describe combinations of scaled changes, making comparisons more interpretable.

An invertible change of coordinates preserves whether the information matrix has full rank, but it can change its eigenvalues and eigenvectors. Consequently, “small” should not be judged using a universal numerical cutoff. Inspect the scale of the parameters, the relative eigenvalues, and the parameter combination represented by the eigenvector.

When the Information Matrix Is Not Positive Definite

At a well-behaved interior local maximum, the observed information should be positive semidefinite, and it is positive definite when there is positive local curvature in every parameter direction. A negative eigenvalue means the quadratic approximation curves upward in at least one direction for the log likelihood. That can indicate that the reported point is not a local maximum, that numerical differentiation or optimization is inaccurate, or that the likelihood approximation is problematic. It should not be described simply as a weakly identified parameter combination.

Practical Interpretation and Alternatives

For a joint-model fit, an eigenvector should be translated back into the model’s parameter names and units. If its largest components involve a longitudinal slope and a survival association coefficient with opposite signs, the weak direction says that certain changes in those parameters can offset one another locally. The direction is a diagnostic of the fitted likelihood, not evidence that one parameter causes changes in another.

Eigenvalue analysis complements, rather than replaces, other diagnostics. A profile likelihood examines how the likelihood behaves as a particular parameter is fixed over a range while other parameters are re-optimized; it can reveal asymmetry or non-quadratic behavior that the local eigen-analysis misses. Parameter-wise standard errors summarize uncertainty for individual coordinates, while an eigenvector summarizes uncertainty for a combination. A condition number gives a compact measure of relative curvature, but it does not identify which scientific parameters form the weak direction. The earlier material on likelihood profiles and likelihood ridges provides related views of these issues.

A useful interpretation therefore combines the eigenvalue size, its eigenvector, the coordinate scaling, and checks that the fitted point is a genuine maximum. If a direction appears weak, inspect the likelihood or profile along that combination and assess whether the conclusion is stable under reasonable changes in scaling and numerical accuracy. A local eigen-analysis alone cannot distinguish all sources of weak information, establish global identifiability, or guarantee accurate confidence intervals.

Common Pitfall

A common mistake is to compare raw eigenvalues from two analyses with different parameter units and declare the fit with smaller eigenvalues less identifiable. Rescaling a coefficient changes the numerical curvature, even when the underlying statistical information has not changed. The correction is to interpret eigenvalues only after specifying parameter scales, and to emphasize the relative weak direction and its scientific meaning. Another mistake is to equate a small eigenvalue with proof of exact non-identifiability: it is evidence of weak local curvature, which should be checked against the likelihood away from the estimate.