Expected Score and Population Fisher Information

From a Subject-Level Likelihood to Population Information

In population design, information is calculated from the distribution of the data contributed by a subject under a specified design. The relevant likelihood is the marginal subject-level likelihood: it describes the subject's observed data after unobserved individual effects have been integrated out. The Gaussian Approximation to the Subject-Level Likelihood provides one possible approximation to that likelihood; the definitions here apply to the likelihood itself, whether exact or approximated.

Let $\boldsymbol{\psi}$ be a $q$-dimensional vector of population parameters, $d$ a subject-level design, and $Y$ that subject's data. Write the marginal density or probability mass function as $p(y\mid\boldsymbol{\psi},d)$. When a model includes a random effect $\boldsymbol b$, this density is obtained by integrating over its distribution:

p(y\mid\boldsymbol{\psi},d) = \int p(y\mid\boldsymbol b,\boldsymbol{\psi},d) \,p(\boldsymbol b\mid\boldsymbol{\psi})\,d\boldsymbol b.

The score is the column vector of derivatives of the log marginal likelihood with respect to the population parameters. It depends on the observed data as well as on the parameter value and design.

\boldsymbol s(y;\boldsymbol{\psi},d) = \nabla_{\boldsymbol{\psi}}\log p(y\mid\boldsymbol{\psi},d).

Definition: The population Fisher information per subject at parameter value $\boldsymbol{\psi}$ and design $d$ is the expected outer product of the subject-level score, with expectation taken over $Y\sim p(\,\cdot\mid\boldsymbol{\psi},d)$: $$ \mathcal I(\boldsymbol{\psi};d) = \operatorname E_{\boldsymbol{\psi},d} \left[ \boldsymbol s(Y;\boldsymbol{\psi},d) \boldsymbol s(Y;\boldsymbol{\psi},d)^{\mathsf T} \right]. $$

This is a $q\times q$ matrix. Its diagonal entries describe information for individual parameters, while its off-diagonal entries describe how score components vary together. Because it is an expected outer product, it is positive semidefinite: for any vector $\boldsymbol a$, $\boldsymbol a^{\mathsf T}\mathcal I\boldsymbol a=\operatorname E[(\boldsymbol a^{\mathsf T}\boldsymbol s)^2]\geq 0$. The matrix is defined at a particular parameter value and design; it is not a universal property of the model independent of those choices.

Why the Score and Curvature Expressions Agree

Fisher information is also often expressed as the negative expected Hessian of the log likelihood. This is an equality only under appropriate regularity conditions. In particular, assume that the support of $Y$ does not depend on $\boldsymbol{\psi}$, that the density is sufficiently differentiable in the parameters, and that differentiation can be interchanged with integration. Also require the relevant derivatives and score products to be integrable, with the parameter value in the interior of the parameter space. These conditions ensure that the density remains normalized as it is differentiated.

For clarity, suppose first that the data are continuous and the density is with respect to a fixed measure. Since the density integrates to one, differentiating its normalization gives a zero vector. Using $\nabla_{\boldsymbol{\psi}}p=p\boldsymbol s$ then gives the zero-mean score result.

\int p(y\mid\boldsymbol{\psi},d)\,dy=1, \qquad \int \nabla_{\boldsymbol{\psi}}p(y\mid\boldsymbol{\psi},d)\,dy = \operatorname E[\boldsymbol s(Y;\boldsymbol{\psi},d)] = \boldsymbol 0.

For a discrete outcome, the same reasoning applies with a sum in place of the integral. To obtain the curvature identity, differentiate the log density twice. The Hessian is a matrix of second derivatives, and the product of score vectors supplies the other term in this calculation.

\nabla_{\boldsymbol{\psi}}^2\log p = \frac{\nabla_{\boldsymbol{\psi}}^2p}{p} - \boldsymbol s\boldsymbol s^{\mathsf T}.

Taking expectations, the first term on the right integrates to the second derivative of the normalization constant. Under the stated conditions, that derivative is zero. Therefore the expected Hessian is the negative expected score outer product.

\operatorname E\!\left[ \nabla_{\boldsymbol{\psi}}^2\log p(Y\mid\boldsymbol{\psi},d) \right] = \int \nabla_{\boldsymbol{\psi}}^2p(y\mid\boldsymbol{\psi},d)\,dy - \mathcal I(\boldsymbol{\psi};d) = -\mathcal I(\boldsymbol{\psi};d).

Theorem: Under the regularity conditions above, the population Fisher information per subject has two equivalent forms: $$ \mathcal I(\boldsymbol{\psi};d) = \operatorname E[\boldsymbol s\boldsymbol s^{\mathsf T}] = -\operatorname E\!\left[ \nabla_{\boldsymbol{\psi}}^2\log p(Y\mid\boldsymbol{\psi},d) \right], \qquad \operatorname E[\boldsymbol s]=\boldsymbol 0. $$

The conditions are substantive, not technical decoration. If the support changes with the parameter, differentiating the integral can produce boundary terms. If differentiation and integration cannot be interchanged, the calculation above is not justified. In either case, the expected score may not be zero and the two expressions need not agree.

Worked Examples

Worked Example: Information in a Bernoulli Response

Consider a hypothetical subject-level binary response $Y$, with $Y=1$ denoting response. Under a specified design, suppose $Y\sim\operatorname{Bernoulli}(p)$, where $p=\operatorname{logit}^{-1}(\eta)$ and the scalar population parameter is the log odds $\eta$. The support $\{0,1\}$ does not depend on $\eta$, and $0<p<1$, so the regularity conditions hold.

The log likelihood is $y\log(p)+(1-y)\log(1-p)$. Since $dp/d\eta=p(1-p)$, differentiating gives the score $s(y;\eta)=y-p$. For the hypothetical value $p=0.4$, the score is $0.6$ if $y=1$, and $-0.4$ if $y=0$. Thus the expected score is $0.4(0.6)+0.6(-0.4)=0$.

The expected outer product, which is a scalar in this example, is $0.4(0.6)^2+0.6(-0.4)^2=0.144+0.096=0.240$. The log-likelihood second derivative is $-p(1-p)$, so negative expected curvature is also $-[-0.4(0.6)]=0.240$. The information is $0.240$ per subject at this design and parameter value. The parameter $\eta$ is dimensionless, so this information is on the inverse-squared log-odds scale.

Worked Example: Information About a Gaussian Mean

Suppose, hypothetically, that a subject contributes one measurement $Y\sim N(\mu,\sigma^2)$, where $\mu$ is the population parameter of interest and the residual standard deviation $\sigma=2$ mg/L is known. The support is the real line for every $\mu$, and the normal density is smooth with finite score moments, so the regularity conditions hold.

The log likelihood, omitting terms not depending on $\mu$, is $-(y-\mu)^2/(2\sigma^2)$. Its score is $s=(y-\mu)/\sigma^2$. Under the model, $\operatorname E(Y-\mu)=0$ and $\operatorname E[(Y-\mu)^2]=\sigma^2$, giving $\operatorname E(s)=0$ and $\mathcal I(\mu)=\sigma^2/\sigma^4=1/\sigma^2$. Substituting $\sigma=2$ mg/L gives $1/4=0.2500\;(\mathrm{L}/\mathrm{mg})^2$ per subject.

The second derivative of the log likelihood is $-1/\sigma^2$, so negative expected curvature gives the same $0.2500\;(\mathrm{L}/\mathrm{mg})^2$. For illustration, if the hypothetical observed value is $y=12$ mg/L and the nominal mean is $\mu=10$ mg/L, the realized score is $(12-10)/4=0.5000\;\mathrm{L}/\mathrm{mg}$. That realized score is not the expected information; the information averages squared scores over all possible observations.

Failure When the Support Depends on the Parameter

Worked Example: A Uniform Model and Its Boundary Term

Let a hypothetical measurement satisfy $Y\sim\operatorname{Uniform}(0,\theta)$, where the unknown upper limit $\theta>0$ is measured in mg. Define its density pointwise as $f(y\mid\theta)=1/\theta$ for $0<y<\theta$ and $f(y\mid\theta)=0$ otherwise. The support changes with $\theta$, so the regularity conditions for the curvature identity do not hold.

For an interior observation, the score is $\partial\log(1/\theta)/\partial\theta=-1/\theta$. It is constant over the possible observations, so its expectation is $-1/\theta$, not zero, and the outer-product information is $1/\theta^2$. At the hypothetical value $\theta=5$ mg, these are $-0.2000\;\mathrm{mg}^{-1}$ and $0.04000\;\mathrm{mg}^{-2}$, respectively.

The log likelihood has second derivative $1/\theta^2$, making negative expected curvature $-1/\theta^2$, which is not the outer-product information. The reason is visible by differentiating the normalization integral. Its upper boundary contributes the interior, left-hand limit of the density:

\lim_{y\uparrow\theta} f(y\mid\theta)=\frac{1}{\theta}, \qquad \frac{d}{d\theta}\int_0^\theta \frac{1}{\theta}\,dy = \lim_{y\uparrow\theta}f(y\mid\theta) + \int_0^\theta\frac{\partial}{\partial\theta} \left(\frac{1}{\theta}\right)dy = \frac{1}{\theta}-\frac{1}{\theta}=0.

Under the stated pointwise definition, $f(\theta\mid\theta)=0$; the boundary contribution is the left-hand limit, not the density's assigned endpoint value. Omitting that contribution would incorrectly treat the derivative of the normalization integral as zero. This example shows why one must check support and boundary behavior before using the expected-curvature identity.

Diagnostics, Comparisons, and Limitations

For a regular model, a useful diagnostic is to approximate expectations by simulation at a fixed nominal $\boldsymbol{\psi}$ and design $d$. Simulate many subject-level data sets, calculate their marginal scores, and check whether the average score is near zero. Compare the average score outer product with the average negative Hessian. Small discrepancies can arise from finite simulation or numerical differentiation; persistent, substantial disagreement calls for checking the likelihood implementation, derivative calculations, integration over random effects, and regularity conditions.

Three quantities should not be confused. The score is the derivative for one realized subject. The observed information is the negative Hessian evaluated at that subject's observed data. The Fisher information is an expectation under the model. Under regularity, expected observed information equals Fisher information, but an individual observed-information matrix need not be positive semidefinite. The outer-product definition, by contrast, is positive semidefinite by construction.

The definition concerns the marginal data distribution. Differentiating a conditional likelihood while treating a subject's unobserved random effect as if it were observed generally does not produce the marginal score or population Fisher information. Likewise, information computed under a Gaussian approximation is information for that approximating distribution; its accuracy depends on the adequacy of the approximation.

Finally, Fisher information is local: it is evaluated at a chosen parameter value and design, and may change when either changes. It describes expected curvature or score variation, not guaranteed precision for a particular sample. It may be singular when some parameter combinations are not locally distinguished by the design. Parameter boundaries, changing support, or poorly behaved tails can invalidate the standard identities. These checks matter before using a Fisher matrix as a design criterion or interpreting its inverse as an approximation to parameter uncertainty.

Common Mistake

Wrong reasoning: “The negative expected Hessian is always Fisher information, so either expression can be used without checking the model.”
Correction: The equality follows from differentiating a normalized likelihood under regularity conditions. Parameter-independent support, valid differentiation under the integral, and finite expectations are among the required checks. If the support depends on the parameter, boundary terms can change the result, as in the uniform example.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What is the score for a subject-level marginal likelihood, and what expectation defines population Fisher information?
For a Bernoulli response with $p=0.25$ and logit parameter $\eta$, calculate the expected score and Fisher information per subject.
Why can the expected-score-zero result fail for a uniform distribution with an unknown upper endpoint?
Distinguish observed information from population Fisher information.

Solutions. (1) The score is $\nabla_{\boldsymbol{\psi}}\log p(Y\mid\boldsymbol{\psi},d)$, the derivative of the log marginal subject-level likelihood. Population Fisher information is the expected score outer product under that marginal distribution. (2) The Bernoulli logit score is $Y-p$, so its expectation is $p-p=0$, and the information is $p(1-p)=0.25(0.75)=0.1875$ per subject. (3) The support changes with the endpoint, so differentiating the normalization integral produces a boundary contribution; the usual interchange argument that yields zero expected score does not apply without that term. (4) Observed information is the negative log-likelihood Hessian at the realized data. Population Fisher information is an expectation under the model, conventionally defined as the expected score outer product; under regularity it also equals expected negative curvature.