Why Diagnose the Information Matrix?
A fitted model can produce numerical estimates for several parameters and still provide little evidence for estimating them separately. Fisher information describes how the likelihood changes as its parameters change. When that information is singular, at least one parameter direction has no local likelihood curvature under the usual regularity conditions. When it is nearly singular, the likelihood may be very flat in one direction, making estimates and standard errors unstable.
The diagnostic is not simply “Can software return an inverse?” A singular matrix has no inverse, while a nearly singular matrix can have an inverse whose entries are very large or sensitive to small changes in the data. Either situation calls for understanding which parameter combinations the data actually identify.
For a regular model, the information matrix is positive semidefinite. Its eigenvectors describe parameter directions, and the corresponding eigenvalues measure the information in those directions. A zero eigenvalue indicates a direction with zero information. A small positive eigenvalue indicates a direction with weak information. The condition number, for a positive definite matrix, is the ratio of its largest to smallest eigenvalue; a large value signals sensitivity to perturbations and numerical error.
Singularity, Identifiability, and Uncertainty
A parameter is identifiable if different parameter values imply different probability distributions for the observations. If two parameters affect the distribution only through a single combination, the data cannot distinguish their separate values. This often produces singular information. However, singular information is a local diagnostic, not a complete definition of global identifiability: nonregular models, boundary points, and likelihoods with higher-order rather than quadratic curvature can require additional analysis.
Under standard regularity conditions, the covariance matrix of a maximum likelihood estimate is approximated by the inverse information matrix. Thus a zero eigenvalue prevents the usual inverse-based covariance estimate, and a small eigenvalue can produce large uncertainty in the associated parameter direction. This approximation does not repair nonidentifiability; it is meaningful only when the model and parameterization support the regular asymptotic theory.
Worked Example: Two Exponential Rates That Cannot Be Separated
A hypothetical portfolio model assumes independent losses \(X_1,\ldots,X_4\), each exponentially distributed with rate \(\lambda_1+\lambda_2\), where \(\lambda_1>0\) and \(\lambda_2>0\). The observed losses are \(1,2,1,4\), in a common unit of measurement. There is no interest or mortality component in this likelihood. Identify the maximum likelihood estimates and diagnose the information matrix.
Likelihood and fit. Write \(r=\lambda_1+\lambda_2\). The exponential density is \(f(x)=r e^{-rx}\) for \(x>0\). The total loss is \(T=1+2+1+4=8\), so the log-likelihood is \(\ell=4\log r-8r\). Differentiating gives \(d\ell/dr=4/r-8\), and setting this equal to zero gives \(\widehat r=4/8=0.5\). The second derivative is \(-4/r^2<0\), so this is the maximum for \(r\).
Separate parameter estimates. The likelihood depends on \(\lambda_1,\lambda_2\) only through their sum. Consequently, every positive pair satisfying \(\lambda_1+\lambda_2=0.5\) has the same maximized likelihood. There is no unique maximum likelihood estimate for the two rates separately.
Information calculation. For \(r\), the Fisher information from four observations is \(4/r^2\). Since the gradient of \(r\) with respect to \((\lambda_1,\lambda_2)\) is \((1,1)^\mathsf{T}\), the information matrix at the fit is \(I= (4/0.5^2)\begin{pmatrix}1&1\\1&1\end{pmatrix} =16\begin{pmatrix}1&1\\1&1\end{pmatrix}\). Its eigenvalues are \(32\) and \(0\). The direction \((1,-1)^\mathsf{T}\) has zero information: increasing one rate while decreasing the other by the same amount leaves their sum, and hence the entire distribution, unchanged. The sum is estimable in this model; the individual rates are not.
Check. For example, \((0.2,0.3)\) and \((0.4,0.1)\) both have sum \(0.5\), so they give the same likelihood for every possible sample. The zero eigenvalue therefore agrees with the direct identifiability argument. The common mistake to avoid is reporting \((0.25,0.25)\) as the unique pair of maximum likelihood estimates: it is one possible pair, not a uniquely determined fit.
Derivation: Why Regular Reparameterizations Preserve Rank
Suppose a model is reparameterized by a smooth map \(\boldsymbol{\theta}=g(\boldsymbol{\eta})\), and let \(J=\partial\boldsymbol{\theta}/\partial\boldsymbol{\eta}\) be its Jacobian at the parameter value. If the map is a regular local change of coordinates, its Jacobian is nonsingular there. The score in the new coordinates follows from the chain rule:
Taking the expected outer product gives the information transformation. Under the usual regularity conditions, the same transformation follows from the expected Hessian:
When \(J\) is nonsingular, multiplication on the left and right by these invertible matrices preserves rank. More explicitly, the map \(z\mapsto Jz\) ranges over all parameter directions, so \(J^\mathsf{T}I_{\boldsymbol{\theta}}J\) has a zero-information direction exactly when \(I_{\boldsymbol{\theta}}\) does. Therefore rank, including whether the information is singular, is invariant under a smooth reparameterization with nonsingular Jacobian at the point. Eigenvalues and condition numbers, in contrast, can change with the scale and geometry of the coordinates.
The nonsingular-Jacobian condition matters. A smooth, locally one-to-one transformation alone is not enough: if its derivative vanishes at the point, it is not a regular local change of coordinates there. The next example shows why.
Worked Example: A Smooth Transformation With a Singular Jacobian
Let \(X_1,\ldots,X_5\) be independent observations from \(N(\theta,4)\), with known variance \(4\) and unrestricted mean \(\theta\). There is no interest or mortality component. The original parameter is \(\theta\); consider the alternative parameter \(\eta\) defined by \(\theta=\eta^3\).
Information in the original parameter. The score for one observation is \((X_i-\theta)/4\), and its variance is \(1/4\). By additivity, the information from five observations is \(I_\theta=5/4=1.25\), which is positive and nonsingular.
Information after transformation. The Jacobian is \(d\theta/d\eta=3\eta^2\). Applying the score transformation, \(I_\eta=(3\eta^2)^2(1.25)=11.25\eta^4\). At \(\eta=0\), this equals zero even though the normal model still identifies \(\theta=0\).
The map \(\eta\mapsto\eta^3\) is smooth and one-to-one, but its Jacobian is zero at \(\eta=0\); its inverse is not differentiable there. Thus it is not a regular local coordinate change at that point. The zero transformed information does not contradict rank invariance, whose nonsingular-Jacobian condition is not met. The exam trap is to claim that smoothness and local one-to-one behavior alone guarantee rank invariance. They do not.
Evaluating a Nearly Singular Fitted Model
A small positive eigenvalue signals weak identification rather than exact nonidentifiability. It is useful to inspect its eigenvector: that vector indicates the combination of parameters for which the likelihood changes least. The following fitted normal model makes the distinction concrete.
Worked Example: Nearly Collinear Predictors and Unstable Standard Errors
Consider a hypothetical regression model with three independent responses \(Y_i=\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i\), where \(\varepsilon_i\sim N(0,1)\), the variance is known, and there is no intercept. There is no interest or mortality component. The predictor vectors are \(x_1=(10,0,0)^\mathsf{T}\) and \(x_2=(9.999,\sqrt{0.019999},0)^\mathsf{T}\). Suppose the observed response vector is \(y=x_1+2x_2\).
Fit and information. The squared lengths are \(x_1^\mathsf{T}x_1=100\) and \(x_2^\mathsf{T}x_2=9.999^2+0.019999=99.980001+0.019999=100\). Their inner product is \(x_1^\mathsf{T}x_2=10(9.999)=99.99\). Thus the information, equal to \(X^\mathsf{T}X\) for known unit variance, is \(I=\begin{pmatrix}100&99.99\\99.99&100\end{pmatrix}\). The response was constructed as \(y=X(1,2)^\mathsf{T}\), so the full-rank least-squares and maximum likelihood estimates are \((\widehat\beta_1,\widehat\beta_2)=(1,2)\).
Eigenvalues and conditioning. A symmetric matrix with diagonal entries \(100\) and off-diagonal entries \(99.99\) has eigenvectors proportional to \((1,1)^\mathsf{T}\) and \((1,-1)^\mathsf{T}\), with eigenvalues \(199.99\) and \(0.01\), respectively. Its condition number is \(199.99/0.01=19{,}999\). The model is not exactly singular, but the predictor columns are almost parallel, so the data give very little information about the difference between the coefficients.
Covariance and standard errors. The determinant is \(100(100)-99.99^2=10{,}000-9{,}998.0001=1.9999\). Hence \(I^{-1}=\frac{1}{1.9999}\begin{pmatrix}100&-99.99\\-99.99&100\end{pmatrix}\). Each estimated coefficient has approximate variance \(100/1.9999\approx50.0025\) and standard error \(\sqrt{50.0025}\approx7.0712\). By contrast, the estimated sum has variance \((100-99.99-99.99+100)/1.9999=0.02/1.9999\approx0.0100\), giving standard error about \(0.1000\). The estimated difference has variance \((100+99.99+99.99+100)/1.9999=399.98/1.9999=200\), giving standard error about \(14.1421\). Standard errors are rounded to four decimal places.
Check and interpretation. The sum is estimated much more precisely than either coefficient separately, while the difference is highly uncertain. The large individual standard errors are compatible with the small residuals in this constructed fit: the predictor columns make many coefficient pairs produce nearly the same fitted values. A common mistake is to treat a finite inverse as evidence of stable estimation. Here the matrix is invertible, but its condition number and the uncertainty in the difference show that the separate coefficient estimates are fragile.
Practical Diagnostic Steps and Common Traps
State the likelihood, parameters, and regularity assumptions relevant to the information calculation. A singularity may arise from the model itself, an overparameterized description, or a nonregular boundary or coordinate choice.
Determine whether an eigenvalue is zero or merely small relative to the others. Use its eigenvector to identify the parameter direction with no or weak information.
If information is singular, do not report an ordinary inverse-based covariance matrix. If it is nearly singular, explain which combinations are unstable and qualify standard errors that depend on the inverse.
Another trap is to compare eigenvalues without considering parameter scale. Rescaling a parameter can change eigenvalue magnitudes and the condition number, even though a regular nonsingular coordinate change does not change rank. Conditioning is therefore a useful numerical warning, not a parameterization-free measure of identifiability. Examine the likelihood, model structure, and practically meaningful parameter combinations as well.
Check Your Understanding
For each problem, identify what the information matrix says about the parameter estimates. Answers are included for checking.
- A two-parameter model has \(I=\begin{pmatrix}8&8\\8&8\end{pmatrix}\). Find its eigenvalues and a zero-information direction. (Answer: eigenvalues \(16\) and \(0\); the direction \((1,-1)^\mathsf{T}\) has zero information.)
- A fitted model has \(I=\begin{pmatrix}25&24\\24&25\end{pmatrix}\). Find the eigenvalues and condition number. Is the matrix singular? (Answer: eigenvalues \(49\) and \(1\), condition number \(49\); it is nonsingular, though the smaller eigenvalue indicates weaker information in the difference direction.)
- Ten independent observations have distribution \(N(\alpha+2\beta,4)\), with known variance and unrestricted \(\alpha,\beta\). Find the information matrix and a nonidentifiable parameter direction. (Answer: \(I=2.5\begin{pmatrix}1&2\\2&4\end{pmatrix}\), which is singular; \((-2,1)^\mathsf{T}\) leaves \(\alpha+2\beta\) unchanged.)
- A scalar model has information \(I_\theta=3\) at \(\theta=1\). Reparameterize by \(\theta=e^\eta\). Find the information at \(\eta=0\), where \(\theta=1\). (Answer: \(d\theta/d\eta=e^\eta=1\), so \(I_\eta=1^2(3)=3\); the Jacobian is nonsingular and rank is preserved.)
- For \(I=\begin{pmatrix}9&8.9\\8.9&9\end{pmatrix}\), calculate the determinant and the standard error corresponding to either diagonal entry of \(I^{-1}\). What does the result warn you about? (Answer: the determinant is \(81-8.9^2=1.79\); the inverse diagonal is \(9/1.79\approx5.0279\), giving standard error \(\sqrt{5.0279}\approx2.2423\). The matrix is invertible, but the large uncertainty relative to the information scale warns of weak separation of the two parameter directions.)