Singular Information And Local Identifiability

When Information Cannot Separate Parameters

A model may contain several parameters, yet the observations may respond to them in the same way. In that situation, fitting the model cannot use the data to distinguish some parameter combinations, even if the recorded outcomes are precise. The rank and eigenvalues of the local Fisher information matrix provide a systematic way to detect this problem at a specified nominal parameter value and observation design.

The key is to interpret singularity as a statement about first-order information. A zero-information direction means that small changes in that parameter direction produce no first-order change in the modeled observations, under the conditions used to calculate the expected information. It does not, by itself, establish that the entire likelihood is flat in that direction or that parameters are globally indistinguishable.

Definition: At a nominal parameter vector \(\theta_0\), the local Fisher information matrix \(J(\theta_0)\) is singular if its rank is less than the number \(p\) of parameters. A nonzero vector \(v\) in its null space, so that \(J(\theta_0)v=0\), is a locally uninformative parameter direction to first order. If \(J\) is positive definite, every nonzero direction has positive information at that point.

This diagnosis assumes the information matrix is defined for the stated model and design. For example, in a regular likelihood model, expected Fisher information is the expected outer product of the score, or equivalently the expected negative Hessian under the usual regularity conditions. In the common case of independent normally distributed observations with known, parameter-independent variances, \(\operatorname{Var}(Y_i)=\sigma_i^2\), and means \(\mu_i(\theta)\), it is

J(\theta_0)=\sum_{i=1}^{n}\frac{1}{\sigma_i^2} \left.\frac{\partial \mu_i}{\partial\theta}\right|_{\theta_0} \left.\frac{\partial \mu_i}{\partial\theta}\right|_{\theta_0}^{\mathsf T}.

Here each sensitivity \(\partial\mu_i/\partial\theta\) is a column vector with \(p\) entries. The formula follows from the independent normal likelihood with fixed variances; it should not be applied unchanged when the variance depends on parameters or the observation model differs. Those cases require the appropriate information expression for that likelihood.

Why Rank Reveals Uninformative Directions

For the normal-mean model above, let \(g_i=\partial\mu_i/\partial\theta\) at \(\theta_0\). For any direction \(v\),

v^{\mathsf T}Jv =\sum_{i=1}^{n}\frac{(g_i^{\mathsf T}v)^2}{\sigma_i^2}.

Every term is nonnegative. Thus \(v^{\mathsf T}Jv=0\) exactly when \(g_i^{\mathsf T}v=0\) for every observation. A small move in direction \(v\) then has zero first-order effect on every modeled mean in this design. The information matrix is positive semidefinite, and its null space contains precisely these directions. Its rank counts the number of independent directions in parameter space to which the observations are locally sensitive.

More generally, for regular models the information matrix is positive semidefinite because it is a score covariance matrix. A zero eigenvalue identifies a direction with zero variance of the score projection at the nominal point. This is a local, expected-information statement, not a guarantee about the curvature of the realized log likelihood.

Worked Example: Redundant Parameters in a Mean Model

Suppose two independent observations have known variance \(1\) and both have mean \(\mu=a+b\), where \(a\) and \(b\) are the parameters. Evaluate information at any nominal \((a,b)\), and identify what the data can and cannot distinguish.

Each observation has sensitivity \(g_i=(1,1)^{\mathsf T}\). Therefore,

J=2 \begin{pmatrix}1&1\\1&1\end{pmatrix} = \begin{pmatrix}2&2\\2&2\end{pmatrix}.

The determinant is \(2(2)-2(2)=0\), so the rank is one, not two. The vector \(v=(1,-1)^{\mathsf T}\) is a null direction: \(Jv=(0,0)^{\mathsf T}\). Increasing \(a\) by a small amount and decreasing \(b\) by the same amount leaves \(a+b\) unchanged. The observations inform the sum, but do not separately determine the two parameters. Replicating these same observations multiplies \(J\) by the replication count; it does not remove the null direction.

Design Can Be Insufficient Even When the Model Is Not Redundant

A different cause of rank deficiency is an observation schedule that fails to excite enough distinct model behavior. The underlying model may have distinct parameter effects over a broader range of observation conditions, while the chosen design samples only where those effects coincide. This is a design-specific failure of first-order information, rather than necessarily a structural redundancy throughout the model.

Worked Example: One Observation for Two Parameters

Consider the linear mean model \(\mu(t)=A+Bt\), with independent measurement error of known standard deviation \(2\). Suppose one observation is collected at \(t=3\). Its sensitivity with respect to \((A,B)\) is \(g=(1,3)^{\mathsf T}\), so

J=\frac{1}{2^2} \begin{pmatrix}1\\3\end{pmatrix} \begin{pmatrix}1&3\end{pmatrix} = \begin{pmatrix}0.25&0.75\\0.75&2.25\end{pmatrix}.

The determinant is \(0.25(2.25)-0.75^2=0\), and the rank is one. For example, \(v=(-3,1)^{\mathsf T}\) is a null direction because \(1(-3)+3(1)=0\). Changes \((A,B)\mapsto(A,B)+\varepsilon(-3,1)\) leave the mean at \(t=3\) unchanged: the change is \(-3\varepsilon+3\varepsilon=0\). One observation can inform the single mean \(A+3B\), but it cannot separately inform both coefficients. Observations at additional, distinct times can supply new sensitivity directions; whether they suffice must be checked from the resulting information matrix.

Eigenvalues: Exact Singularity and Weak Information

Because \(J\) is symmetric and positive semidefinite, its eigenvalues are real and nonnegative. A zero eigenvalue corresponds to an exact null direction. A very small positive eigenvalue instead indicates a direction with little information relative to directions with larger eigenvalues. In regular large-sample approximations, the inverse information is used to approximate parameter covariance; along an eigenvector with eigenvalue \(\lambda\), the corresponding variance scale is proportional to \(1/\lambda\). Thus small positive eigenvalues can signal imprecise estimation even when the matrix is technically full rank.

Eigenvalue magnitudes depend on parameter units. Measuring a parameter in milligrams rather than grams rescales its coordinate and changes the numerical eigenvalues. Rank is unchanged by a nonsingular change of coordinates, but comparing eigenvalue sizes is meaningful only after choosing sensible parameter scales. Inspect the eigenvectors as well: they show which parameter combinations align with the weak directions. A numerical rank decision also depends on rounding and the tolerance used by software, so report the scale and tolerance when the distinction matters.

Worked Example: Nearly Indistinguishable Sensitivities

Suppose two independent observations have known variance \(1\), and their sensitivities with respect to parameters \((\theta_1,\theta_2)\) are \((1,0.99)^{\mathsf T}\) and \((1,1.01)^{\mathsf T}\). Their information matrix is

J= \begin{pmatrix}1&0.99\\0.99&0.9801\end{pmatrix} + \begin{pmatrix}1&1.01\\1.01&1.0201\end{pmatrix} = \begin{pmatrix}2&2\\2&2.0002\end{pmatrix}.

Its trace is \(4.0002\), and its determinant is \(2(2.0002)-2^2=0.0004\). The eigenvalues solve \(\lambda^2-4.0002\lambda+0.0004=0\), giving approximately \(\lambda_{\max}=4.0001\) and \(\lambda_{\min}=0.0001000\). The matrix has full rank, but the small eigenvalue signals a weakly informed direction. The sensitivities are nearly parallel, so the observations respond similarly to the two parameters. This is not exact non-identifiability; it is poor separation in this design. These eigenvalue sizes are in the stated parameter coordinates and should not be compared across differently scaled parameterizations without scaling considerations.

Interpretation, Diagnostics, and Limits

A full-rank local information matrix supports regular, locally separate estimation at the nominal point, subject to the model and likelihood assumptions. It does not prove global identifiability: distinct parameter values far apart may still generate the same distribution. Conversely, singular information says that a first-order, information-based approximation cannot provide finite uncertainty estimates for every parameter direction. It does not by itself prove exact non-identifiability, because higher-order effects may distinguish parameter values.

For instance, let \(Y\sim N(\theta^2,1)\), with nominal value \(\theta_0=0\). The mean sensitivity is \(2\theta\), which is zero at the nominal value, so the expected information there is \(J(0)=(2\theta_0)^2=0\). Yet, up to a constant, the realized log likelihood is \(y\theta^2-\theta^4/2\), whose second derivative at zero is \(2y\). It is nonzero when \(y\ne0\). Thus zero expected information at a point does not imply zero curvature in the realized likelihood. This example also illustrates why regularity and the local nature of the diagnosis matter.

When a calculation indicates singularity or near-singularity, check that the information was evaluated at the intended nominal parameters, that the sensitivities and observation variances are correct, and that parameter units and numerical tolerances are sensible. Then inspect the null or weak eigenvectors and ask what combinations they represent. Possible responses include collecting observations under conditions that generate distinct sensitivities, removing a parameter that the scientific model does not need, or gathering external information. Each change requires a new model or design assessment; a numerical adjustment alone cannot establish identifiability.

This rank-and-eigenvalue diagnosis differs from examining individual parameter standard errors alone: a combination can be poorly informed even when the source of the problem is not obvious from separate summaries. It also differs from a global identifiability analysis, which asks whether distinct parameter values anywhere in the parameter space produce the same observation distribution. The local information calculation is especially useful during design and model checking, but it is not a substitute for global analysis when global uniqueness is essential.

Common pitfall: “The information matrix is singular, so the realized likelihood must be flat along its null direction.” This conclusion is too strong. Singularity of expected information establishes zero expected information curvature in that direction under the stated conditions. Realized likelihood curvature can still be nonzero, and higher-order changes in the model can distinguish parameters. The safe conclusion is that the usual first-order information approximation cannot provide finite uncertainty estimates for every parameter separately.

Check Your Understanding

Work through each question, then compare with the solutions.

For \(J=\begin{pmatrix}3&3\\3&3\end{pmatrix}\), give its rank and one null direction. What does that direction mean for first-order information?
A two-parameter model has one observation with sensitivity \((2,1)^{\mathsf T}\) and known variance \(4\). Calculate its information matrix and state its rank.
Does a small positive eigenvalue mean exact non-identifiability? What practical concern does it indicate?
In \(Y\sim N(\theta^2,1)\) at \(\theta_0=0\), why is the expected information zero, and what does that imply about realized likelihood curvature?

Solutions. (1) The determinant is \(9-9=0\) and the matrix is nonzero, so its rank is one. The vector \((1,-1)^{\mathsf T}\) is null; moving equally and oppositely in the two parameter coordinates has zero first-order information. (2) \(J=(1/4)(2,1)^{\mathsf T}(2,1)=\begin{pmatrix}1&0.5\\0.5&0.25\end{pmatrix}\). Its determinant is \(0.25-0.25=0\), and its nonzero entries make its rank one. (3) No. A small positive eigenvalue indicates weak information and potentially large uncertainty in the associated direction; exact singularity corresponds to a zero eigenvalue. (4) The mean derivative is \(2\theta\), hence zero at the nominal value, and the normal-mean information is its squared derivative divided by variance, also zero. Nevertheless, the realized log likelihood has second derivative \(2y\) at zero, which need not vanish.