Structural Scores And Fisher Information

Why Scores and Information Matter

A structural likelihood describes how a model’s parameters govern the distribution of observed data. Its score records how sensitive the log-likelihood is to changes in those parameters; Fisher information summarizes the expected sensitivity. These quantities connect the likelihood to two central questions: whether nearby parameter values imply distinguishable distributions, and how precisely regular estimators can estimate the parameters.

The calculations depend on the likelihood’s parameterization, including any restrictions or transformations that define the structural model. For a density obtained by transforming structural shocks, the likelihood must include the relevant change-of-variables terms before it is differentiated. Omitting such a term can alter both the score and the information.

Definitions and Regularity Conditions

Let \(Y_i\) denote observation \(i\), and let \(\theta\) be a \(k\)-dimensional structural parameter in an open parameter space. Suppose the observations are independent and identically distributed with density \(f(y;\theta)\), with respect to a measure that does not depend on \(\theta\). For observed data \(y_1,\ldots,y_n\), the log-likelihood is \(\ell_n(\theta)=\sum_{i=1}^n \log f(y_i;\theta)\).

Definition: The individual score is the \(k\)-vector \(s(Y_i;\theta)=\partial \log f(Y_i;\theta)/\partial\theta\). The sample score is \(S_n(\theta)=\partial\ell_n(\theta)/\partial\theta=\sum_{i=1}^n s(Y_i;\theta)\). The per-observation Fisher information is \(\mathcal I(\theta)=E_\theta[s(Y_i;\theta)s(Y_i;\theta)']\), and the information in \(n\) independent observations is \(n\mathcal I(\theta)\).

A prime denotes transpose. The score and information are evaluated at a parameter value; when they are random, their expectations are under the model at that value. For the standard information identities and asymptotic conclusions below, assume the model is correctly specified and sufficiently regular: the density is differentiable in \(\theta\), its support does not vary with \(\theta\), and differentiation can be interchanged with integration as needed. Also assume the relevant expectations exist and are finite. These conditions rule out important complications such as parameter-dependent support and non-differentiable likelihoods.

Under these conditions, the score has mean zero, \(E_\theta[s(Y_i;\theta)]=0\), and the information can also be written as the negative expected Hessian:

\mathcal I(\theta) =E_\theta[s(Y_i;\theta)s(Y_i;\theta)'] =-E_\theta\!\left[\frac{\partial^2\log f(Y_i;\theta)} {\partial\theta\,\partial\theta'}\right].

The first expression is an expected outer product; the second is expected curvature of the log-likelihood. They agree under the stated regularity conditions and correct specification. The matrix is positive semidefinite because for every vector \(a\), \(a'\mathcal I(\theta)a=E_\theta[(a's(Y_i;\theta))^2]\geq 0\).

Deriving the Information Identity

The key identities follow by differentiating the fact that a density integrates to one. The score’s mean is

E_\theta[s(Y_i;\theta)] =\int \frac{\partial \log f(y;\theta)}{\partial\theta}f(y;\theta)\,dy =\int \frac{\partial f(y;\theta)}{\partial\theta}\,dy =\frac{\partial}{\partial\theta}1=0.

For the second identity, differentiate this zero expectation with respect to \(\theta'\). Since \(\partial f/\partial\theta=f s\), differentiation of \(f s\) gives \(f ss'+f\,\partial s/\partial\theta'\). Integrating yields

0=E_\theta[ss']+E_\theta\!\left[\frac{\partial s}{\partial\theta'}\right] =E_\theta[ss']+E_\theta\!\left[\frac{\partial^2\log f} {\partial\theta\,\partial\theta'}\right].

Rearranging proves the identity. For independent observations, scores add and their cross-products have expectation zero because each score has mean zero. Consequently, the sample information is \(n\mathcal I(\theta)\).

Worked Calculations

Worked Example: Normal Structural Location Model

Suppose hypothetical independent outcomes follow \(Y_i=\theta+\varepsilon_i\), where \(\varepsilon_i\sim N(0,\sigma^2)\) and \(\sigma=0.5\) is known. Thus \(Y_i\sim N(\theta,\sigma^2)\); the structural parameter is the location \(\theta\). Assume the stated normal model and known, fixed variance are correct.

For one observation, the log-density, up to a constant not depending on \(\theta\), is \(- (y_i-\theta)^2/(2\sigma^2)\). Differentiating gives

s(y_i;\theta)=\frac{y_i-\theta}{\sigma^2}, \qquad \mathcal I(\theta)=E_\theta\!\left[\frac{(Y_i-\theta)^2}{\sigma^4}\right] =\frac{1}{\sigma^2}.

Take the hypothetical data \(2.1, 1.7, 2.4, 2.0\), so \(n=4\) and \(\bar y=8.2/4=2.05\). At the trial value \(\theta=2.00\), the sample score is

S_4(2.00)=\frac{(2.1-2.0)+(1.7-2.0)+(2.4-2.0)+(2.0-2.0)}{0.5^2} =\frac{0.20}{0.25}=0.800.

The positive score indicates that, at \(\theta=2.00\), the log-likelihood is increasing as \(\theta\) increases. The per-observation information is \(1/0.25=4\), and the sample information is \(4(4)=16\). Differentiating the sample log-likelihood shows its maximum occurs at \(\hat\theta=\bar y=2.05\): the score is zero there, and the log-likelihood has constant second derivative \(-4/0.25=-16\). These calculations use the specified normal location model and known variance; they do not establish that the model is appropriate for any real data.

Worked Example: Bernoulli Structural Probability

Suppose independent hypothetical outcomes \(Y_i\) are Bernoulli with success probability \(\theta\), where \(0<\theta<1\). The structural parameter directly determines the probability of success. For a single outcome, the log-likelihood is \(y_i\log\theta+(1-y_i)\log(1-\theta)\), so

s(y_i;\theta)=\frac{y_i}{\theta}-\frac{1-y_i}{1-\theta} =\frac{y_i-\theta}{\theta(1-\theta)}.

Because \(E_\theta[(Y_i-\theta)^2]=\theta(1-\theta)\), the information per observation is

\mathcal I(\theta) =E_\theta[s(Y_i;\theta)^2] =\frac{\theta(1-\theta)}{\{\theta(1-\theta)\}^2} =\frac{1}{\theta(1-\theta)}.

Suppose the hypothetical sample is \(1,1,0,1\). At \(\theta=0.60\), there are three successes, so

S_4(0.60)=\frac{3-4(0.60)}{0.60(0.40)} =\frac{0.60}{0.24}=2.500, \qquad 4\mathcal I(0.60)=\frac{4}{0.24}=16.67.

The positive score indicates that the likelihood is increasing at \(0.60\). Its maximum is at \(\hat\theta=3/4=0.75\), the observed success fraction. The information calculation is valid for interior values \(0<\theta<1\), as assumed; the formula is not finite at the boundary.

Information, Local Identification, and Precision

Identification concerns whether different parameter values imply different distributions of the observables. Information gives a local, derivative-based diagnostic. Under the regularity conditions above, a nonsingular information matrix is positive definite. In a differentiable model, the Kullback–Leibler divergence from the distribution at \(\theta\) to the distribution at a nearby value \(\theta+h\) has the local expansion

E_\theta\!\left[\log\frac{f(Y;\theta)}{f(Y;\theta+h)}\right] =\frac{1}{2}h'\mathcal I(\theta)h+o(\|h\|^2).

If \(\mathcal I(\theta)\) is positive definite, then \(h'\mathcal I(\theta)h>0\) for every nonzero \(h\). Nearby distinct parameter values therefore imply distinct distributions, giving local identification under these regularity conditions. This is a local conclusion: it does not rule out observationally equivalent parameter values far away, so it does not establish global identification.

For a correctly specified regular model with a locally identified interior parameter, standard maximum-likelihood asymptotics give an estimator whose approximate covariance is the inverse sample information:

\sqrt{n}(\hat\theta-\theta_0) \ \overset{d}{\longrightarrow}\ N\!\left(0,\mathcal I(\theta_0)^{-1}\right), \qquad \operatorname{Var}(\hat\theta)\ \approx\ \frac{1}{n}\mathcal I(\theta_0)^{-1}.

This statement requires additional standard conditions for maximum-likelihood consistency and asymptotic normality, including a well-behaved likelihood near the true interior value \(\theta_0\). It is not a guarantee for every structural model or estimator. When its conditions hold, greater information generally corresponds to lower asymptotic variance in the directions measured by the matrix.

Worked Example: A Singular Matrix Reveals an Unidentified Direction

Consider a hypothetical model with independent \(Y_i\sim N(a+b,1)\), where the structural parameter is \(\theta=(a,b)'\). The distribution depends on \(a\) and \(b\) only through their sum. Assume this normal model is correctly specified, with known variance. For one observation, writing \(\mu=a+b\), the score is

s(y;a,b)=(y-a-b) \begin{pmatrix}1\\1\end{pmatrix}, \qquad \mathcal I(a,b)= \begin{pmatrix}1&1\\1&1\end{pmatrix}.

For \(n=5\) hypothetical observations \(0.8,1.1,0.9,1.2,1.0\), the sample mean is \(1.0\). At \((a,b)=(0.4,0.6)\), each fitted mean is \(1.0\), so the sample score is

S_5(0.4,0.6) =\left(\sum_{i=1}^5(y_i-1.0)\right) \begin{pmatrix}1\\1\end{pmatrix} =0\begin{pmatrix}1\\1\end{pmatrix}.

The sample information is \(5\mathcal I\). Its eigenvalues are \(10\) and \(0\); the direction \((1,-1)'\) has zero information because changing \(a\) and \(b\) by opposite amounts leaves \(a+b\) unchanged. The data can inform the sum but cannot distinguish the two parameters individually. Here the singularity corresponds to a genuine observational equivalence.

Singularity is a warning, not a universal proof of nonidentification. For example, let a Bernoulli outcome have probability \(p(\theta)=1/2+\theta^3/4\), with \(-1<\theta<1\). This probability lies between \(1/4\) and \(3/4\), and is strictly increasing in \(\theta\), so different values of \(\theta\) imply different distributions. Yet \(p'(0)=0\), and the information per observation at zero is

\mathcal I(0)=\frac{\{p'(0)\}^2}{p(0)\{1-p(0)\}}=0.

Thus this parameter is identified even though the information is singular at zero: the parameter affects the distribution only at higher than first order there. Standard inverse-information approximations do not apply in that direction, and ordinary asymptotic behavior may fail. Identifiability and first-order information are related, but they are not interchangeable concepts.

Common Mistakes and Practical Checks

Treating zero score as evidence of identification. A zero score at a candidate value is a stationarity condition for the sample likelihood, not a guarantee of a unique maximum or of identification.
Calling singular information proof of nonidentification. It shows that some direction has no first-order information. Check whether the distribution is actually unchanged along that direction, or whether it changes at higher order.
Using the inverse information without checking its rank. If \(\mathcal I(\theta)\) is singular, its ordinary inverse does not exist. Do not report the usual inverse-information covariance for all parameter directions.
Confusing total and per-observation information. For \(n\) independent identically distributed observations, total information is \(n\mathcal I(\theta)\), not \(\mathcal I(\theta)\). The corresponding regular asymptotic covariance scales as \(1/n\).
Differentiating an incomplete likelihood. In structural models, parameter-dependent transformations can contribute Jacobian terms to the density. Construct the correct likelihood first, then differentiate it.

Key takeaway: Fisher information is the expected curvature of a regular, correctly specified likelihood and the expected outer product of its score. Positive-definite information supports local identification and regular precision calculations; singular information demands diagnosis of the particular model and parameter direction.

Check Your Understanding

Use the model assumptions stated in each question, and distinguish local information from identification itself.

For a regular model, define the individual score and explain why its expected value is zero.
In the normal location model with known variance \(\sigma^2\), what are the per-observation and \(n\)-observation information values?
Why does nonsingular information support local identification, but not necessarily global identification?
In the model \(Y_i\sim N(a+b,1)\), which parameter direction has zero information, and why?
Can a parameter be identified when information is singular? Explain using the cubic-probability example.