The Condition Number Of An Information Matrix

Why Summarize an Information Matrix With One Number?

In a population pharmacokinetic model, the information matrix describes local curvature of the likelihood near a parameter estimate. Its entries and inverse can be difficult to assess together, especially when several parameters are estimated jointly. The condition number compresses one aspect of that matrix into a single measure: how close the matrix is to being singular, relative to its largest scale. A large value warns that some directions in the parameter space are much less well determined than others, or that numerical calculations involving the matrix may be sensitive.

This summary is useful, but it is not a precision estimate for any one parameter. Its value depends on the matrix and parameterization being examined, and it does not replace inspection of the estimated covariance matrix, standard errors, or model diagnostics. The calculations below use hypothetical matrices for illustration; they do not represent results from a real study.

Information Matrices and the Condition Number

Let \(\theta\) be a vector of model parameters and let \(\ell(\theta)\) denote the log-likelihood. The observed information at an estimate is commonly defined as the negative Hessian of the log-likelihood, \(J(\hat{\theta})=-\nabla^2\ell(\hat{\theta})\). The expected Fisher information is the expected outer product of the score, \(I(\theta)=E[s(\theta)s(\theta)^\mathsf{T}]\), where \(s(\theta)=\nabla\ell(\theta)\). Under appropriate regularity conditions, expected Fisher information can also be expressed as the expected negative Hessian. In either case, the matrix is symmetric when the relevant derivatives exist and are continuous.

Expected Fisher information is positive semidefinite: for any vector \(a\), \(a^\mathsf{T}I(\theta)a=E[(a^\mathsf{T}s(\theta))^2]\geq 0\). An observed information matrix need not be positive semidefinite at every parameter value. When an information matrix is positive definite, its eigenvalues are all positive, and it is invertible. Under the usual local likelihood approximation and regularity conditions, the inverse information matrix approximates the covariance matrix of the parameter estimate. This approximation is local and does not ensure that a model is correctly specified or that its parameters are well identified by the data.

For a positive-definite information matrix, \(\kappa_2(I)\geq 1\). A value near 1 means that the largest and smallest eigenvalues are similar; a large value means that they differ substantially. In the latter case, the information matrix is near-singular relative to its largest eigenvalue. “Near” here is relative: a matrix can have a large condition number even if all its entries are numerically large, and multiplying every entry by the same positive constant leaves the condition number unchanged.

Why the Eigenvalue Ratio Measures Near-Singularity

The eigenvalue formula follows directly from the spectral theorem. A real symmetric positive-definite matrix has an orthogonal diagonalization \(I=Q\Lambda Q^\mathsf{T}\), where the diagonal entries of \(\Lambda\) are the positive eigenvalues. Orthogonal transformations preserve the 2-norm, so the largest stretching factor of \(I\) is \(\lambda_{\max}(I)\). The inverse has diagonalized form \(I^{-1}=Q\Lambda^{-1}Q^\mathsf{T}\), whose largest stretching factor is \(1/\lambda_{\min}(I)\). Therefore:

This also explains why a small minimum eigenvalue is consequential: inversion turns it into a large eigenvalue of \(I^{-1}\). A matrix with a zero eigenvalue cannot be inverted. For a singular positive-semidefinite matrix, \(\lambda_{\min}(I)=0\); its eigenvalue-ratio condition number is treated as infinite. For a symmetric indefinite matrix, however, the positive-definite eigenvalue-ratio formula does not apply. In that general case the norm-based condition number is defined through singular values, not by dividing the largest algebraic eigenvalue by the smallest algebraic eigenvalue.

Worked Calculations

Interpreting the Number in Practice

For a positive-definite information matrix, a larger condition number means that its least informative direction is much weaker relative to its most informative direction. It can also mean that linear computations involving the matrix, such as solving a system or inverting it, are more sensitive to small numerical perturbations. One standard sensitivity bound makes this connection explicit. If a nonsingular matrix \(A\) and right-hand side \(b\) are perturbed to \(A+\Delta A\) and \(b+\Delta b\), and \(\kappa_2(A)\|\Delta A\|_2/\|A\|_2<1\), then the relative solution error is bounded by

where \(Ax=b\) and \((A+\Delta A)(x+\Delta x)=b+\Delta b\). Thus, a large condition number can amplify relative perturbations. This is a worst-case bound, not a prediction that a particular calculation will have that much error.

A condition number is not an absolute measure of information. For any positive constant \(c\), \(\kappa_2(cI)=\kappa_2(I)\), even though the inverse-information covariance approximation changes from \(I^{-1}\) to \(I^{-1}/c\). Therefore, matrices can have the same condition number but imply very different absolute uncertainty. The raw condition number also depends on parameter coordinates and units; comparisons are meaningful only when the parameterization is stated and appropriately considered. Parameter scaling is a separate issue and should not be silently ignored.

Diagnostics, Limitations, and Common Mistakes

Before interpreting a reported condition number, check that the matrix is the intended observed or expected information matrix, is numerically symmetric to the expected tolerance, and has the definiteness required for the eigenvalue-ratio formula. For a positive-definite matrix, inspect its computed smallest and largest eigenvalues as well as their ratio. If the smallest eigenvalue is close to numerical precision, the reported ratio may vary with software, tolerances, or rounding. Cholesky factorization can be a useful positive-definiteness check, but successful factorization does not prove that the model is scientifically adequate.

There is no universal cutoff at which a condition number becomes unacceptable. Its practical meaning depends on numerical precision, the calculation being performed, the matrix construction, and the parameterization. A very large value is a signal to investigate, not an automatic reason to discard a model. Conversely, a moderate condition number cannot guarantee good estimation: all directions may carry little absolute information, or model assumptions may be wrong.

• Mistake: treating a large condition number as an RSE. A condition number is a matrix-level relative ratio, not the relative standard error of a particular estimate. Examine parameter-specific uncertainty separately.
• Mistake: using the smallest algebraic eigenvalue for an indefinite matrix. The positive-definite eigenvalue-ratio formula requires positive eigenvalues. For general matrices use singular values for the 2-norm condition number; singularity gives an infinite value.
• Mistake: treating determinant as a substitute. A determinant is a product, so different eigenvalue patterns can produce similar determinants. It does not summarize the largest-to-smallest singular-value ratio.
• Mistake: comparing raw values without checking parameterization. Changes in units or parameter coordinates can alter the reported condition number. State what matrix was assessed and interpret the result in that context.

In population pharmacokinetics, the condition number is most useful as a compact diagnostic alongside the matrix itself and the model’s parameter-specific precision summaries. It can flag a disparity in information and possible numerical sensitivity; it cannot identify the biological explanation, diagnose the model on its own, or replace scientific and statistical assessment.