Likelihood Scores And Fisher Information

Why Likelihood Scores and Information Matter

A likelihood describes how compatible possible parameter values are with observed data. Its local behavior is summarized by two related quantities: the score, which records the direction and rate of change of the log likelihood, and Fisher information, which describes the expected variability of that score. In design problems, information is useful because it characterizes how strongly a model is expected to distinguish nearby parameter values.

The score depends on the data actually observed. Fisher information is an expectation under a specified model and parameter value, so it is a property of the assumed sampling distribution at that value. Keeping these objects distinct is essential: one realized score, or its outer product, is not generally the Fisher information.

Definition: Let $Y$ have probability density or probability mass function $f(y;\theta)$, where $\theta=(\theta_1,\ldots,\theta_d)^\mathsf{T}$ is a $d$-dimensional parameter vector. The log likelihood for observed $y$ is $\ell(\theta;y)=\log f(y;\theta)$. The score vector is $$ U(\theta;Y)=\nabla_\theta \ell(\theta;Y). $$ The Fisher information matrix is $$ I(\theta)=E_\theta\left[U(\theta;Y)U(\theta;Y)^\mathsf{T}\right], $$ where the expectation is taken over $Y$ generated from $f(\,\cdot\,;\theta)$. For a scalar parameter, the score and information reduce to a scalar and its expected square.

The score has one component for each parameter. Its $j$th component is the derivative of the log likelihood with respect to $\theta_j$, evaluated at the observed data. A positive component means that a small increase in that parameter, holding the others fixed, would locally increase the log likelihood; a negative component means it would locally decrease it. The score has units reciprocal to the parameter units, and the $j,k$ entry of the information matrix has units reciprocal to the product of the units of $\theta_j$ and $\theta_k$.

Conditions for the Equivalent Curvature Formula

Fisher information can also be expressed using the curvature of the log likelihood. This second formula is useful only when the probability model is sufficiently regular. In particular, assume that the parameter lies in an open region, that the support of $Y$ does not depend on $\theta$, and that $f(y;\theta)$ is twice differentiable in $\theta$. Also assume that differentiation can be interchanged with integration or summation, the relevant derivatives are integrable, and the information entries are finite. These conditions allow derivatives of the normalization equation to be taken inside the expectation.

Theorem: Under these regularity conditions, $$ I(\theta)=E_\theta\left[U(\theta;Y)U(\theta;Y)^\mathsf{T}\right] =-E_\theta\left[\nabla_\theta^2\ell(\theta;Y)\right]. $$ The expectation of the score is zero. Consequently, the information matrix is also the covariance matrix of the score and is positive semidefinite.

The identity follows directly from the normalized density. First, since $\int f(y;\theta)\,dy=1$, differentiating once gives $E_\theta[U]=0$. To obtain the second formula, use the derivative of a logarithm: $$ \nabla_\theta^2\ell =\frac{\nabla_\theta^2 f}{f} -\frac{(\nabla_\theta f)(\nabla_\theta f)^\mathsf{T}}{f^2}. $$ Multiply by $f$ and take expectations. The first resulting term integrates to zero because it is the second derivative of the normalization integral. In the second term, $\nabla_\theta f/f=U$. Therefore, $$ E_\theta[\nabla_\theta^2\ell] =\int \nabla_\theta^2 f(y;\theta)\,dy -\int \frac{(\nabla_\theta f)(\nabla_\theta f)^\mathsf{T}}{f}\,dy =0-E_\theta[UU^\mathsf{T}]. $$ Rearranging gives the stated identity. Since $E_\theta[U]=0$, $E_\theta[UU^\mathsf{T}]=\operatorname{Cov}_\theta(U)$. For any vector $a$, $a^\mathsf{T}I(\theta)a=E_\theta[(a^\mathsf{T}U)^2]\geq 0$, which establishes positive semidefiniteness.

Worked Example: A Bernoulli Score and Its Information

Suppose, hypothetically, a single binary pharmacodynamic endpoint $Y$ is modeled as Bernoulli with response probability $p$, where $0<p<1$. Calculate its score and Fisher information at the nominal value $p=0.30$.

For $Y=y$, where $y$ is either 0 or 1, the log likelihood and score are $$ \ell(p;y)=y\log p+(1-y)\log(1-p),\qquad U(p;y)=\frac{y}{p}-\frac{1-y}{1-p}. $$

At $p=0.30$, a response ($y=1$) gives $U=1/0.30=3.333$, while no response ($y=0$) gives $U=-1/0.70=-1.429$. The expected score is $$ E_p[U]=0.30(3.333)+0.70(-1.429)=1.000-1.000=0. $$

The information is the expected squared score: $$ I(p)=0.30(3.333^2)+0.70((-1.429)^2) =3.333+1.429=4.762. $$ Equivalently, $I(p)=1/[p(1-p)]=1/(0.30\times0.70)=4.762$. This is information per binary observation at $p=0.30$; it is not the squared score for either possible response.

Vector Scores: Information Is a Matrix

For multiple parameters, the outer product in the definition matters: $UU^\mathsf{T}$ is a matrix, not a scalar. Its diagonal entries describe the expected squared score for individual parameters, while its off-diagonal entries describe expected products of score components. They reflect how score changes for different parameters vary together under the model.

Worked Example: Information for a Two-Parameter Normal Mean

Consider a hypothetical measurement vector $Y=(Y_1,Y_2)^\mathsf{T}$ with $Y\sim N_2(\theta,\Sigma)$, where $\theta=(\theta_1,\theta_2)^\mathsf{T}$ is unknown and $\Sigma=\begin{pmatrix}4&0\\0&1\end{pmatrix}$ is known. Calculate the score at $\theta=(2,2)^\mathsf{T}$ for observed data $y=(3,1)^\mathsf{T}$, and then calculate the Fisher information.

For a normal vector with known covariance, differentiating the log density gives $U(\theta;y)=\Sigma^{-1}(y-\theta)$. Here, $\Sigma^{-1}=\begin{pmatrix}0.25&0\\0&1\end{pmatrix}$, so the observed score is $$ U=\begin{pmatrix}0.25&0\\0&1\end{pmatrix} \begin{pmatrix}3-2\\1-2\end{pmatrix} =\begin{pmatrix}0.25\\-1\end{pmatrix}. $$

Under the model, $Y-\theta$ has mean zero and covariance $\Sigma$. Thus $E_\theta[UU^\mathsf{T}]=\Sigma^{-1}\Sigma\Sigma^{-1}=\Sigma^{-1}$, giving $$ I(\theta)=\begin{pmatrix}0.25&0\\0&1\end{pmatrix}. $$ For this one observed vector, the score outer product is $\begin{pmatrix}0.0625&-0.25\\-0.25&1\end{pmatrix}$, which differs from $I(\theta)$. The information is the expectation across possible data vectors, not the outer product calculated from the one observed vector.

Using Expected Negative Curvature

The alternative expression, $I(\theta)=-E_\theta[\nabla_\theta^2\ell]$, interprets information as the average negative curvature of the log likelihood. It can be convenient when differentiating the log likelihood twice is easier than calculating the score's distribution. The outer-product expression can be convenient when the score is simple. Under the stated conditions, both give the same expected matrix, and either can be used to check an analytic calculation.

The expectation is important. The Hessian at one observed data set is a random matrix; its negative is not generally equal to the expected negative Hessian. Nor is it generally equal to the score outer product at that data set. The equivalence proved above is between expectations. Replacing an expectation by a single realization changes the quantity being calculated.

Worked Example: Checking the Information Identity for Bernoulli Data

Suppose, hypothetically, that $n=5$ independent binary outcomes are summarized by $x=2$ responses. Evaluate the score and the expected information at $p=0.60$, and verify the negative expected Hessian formula.

Ignoring the binomial coefficient, which does not depend on $p$, the log likelihood is $\ell(p)=2\log p+3\log(1-p)$. Therefore, $$ U(p)=\frac{2}{p}-\frac{3}{1-p},\qquad U(0.60)=\frac{2}{0.60}-\frac{3}{0.40}=3.333-7.500=-4.167. $$ The observed score is negative at this data set and parameter value.

For $n$ independent Bernoulli outcomes, the expected score outer product is $I(p)=n/[p(1-p)]$. At the stated values, $$ I(0.60)=\frac{5}{0.60(0.40)}=\frac{5}{0.24}=20.83. $$ The second derivative for the observed data is $\ell''(p)=-2/p^2-3/(1-p)^2$. Its negative at $p=0.60$ is $2/0.36+3/0.16=24.31$, which is not the Fisher information. To calculate the expected negative second derivative, use $E_p[x]=np=3$ and $E_p[n-x]=n(1-p)=2$: $$ -E_p[\ell''(p)]=\frac{3}{0.36}+\frac{2}{0.16}=8.333+12.50=20.83=I(0.60). $$ This verifies the equality of the two expected definitions, while also showing why the observed-data curvature cannot be substituted for the expectation without qualification.

Diagnostics, Failure Modes, and Interpretation

A useful calculation check is to compute the score both by differentiating the log likelihood and, where practical, by differentiating the density and using $U=(\nabla_\theta f)/f$. Check that the expected score is zero and that the outer-product information is symmetric and positive semidefinite. If an analytic negative expected Hessian disagrees with the score outer product, first check signs, parameter ordering, and whether the expectation was taken under the same parameter value and model.

The regularity assumptions are substantive. For example, if the support depends on the parameter, differentiating the density alone may omit changes at the support boundary. At parameter boundaries, the open-region assumption can fail. If differentiation cannot be interchanged with integration, the derivation of zero expected score or the Hessian identity may fail. These are not merely algebraic inconveniences: the usual formulas need not hold in such models.

The expected information is a model-based quantity. If the assumed probability model does not describe the data-generating process, its expectation may not characterize the actual variability of the score. Also, positive semidefiniteness does not guarantee that the matrix is invertible. Information should therefore not be described automatically as an exact measure of parameter precision; translating it into uncertainty requires additional inferential assumptions and methods.

Common pitfall: “I calculated $U(\theta;y)U(\theta;y)^\mathsf{T}$, so I have the Fisher information.” That calculation gives the score outer product for one observed data set. Fisher information is its expectation under the model at $\theta$. Likewise, the negative Hessian at one data set is not the negative expected Hessian. Use either defining formula with the required expectation, and verify the regularity conditions before invoking their equivalence.

Key takeaway: The score is the data-dependent gradient of the log likelihood. Fisher information is the expected score outer product; under suitable differentiability, support, and integrability conditions, it is also the negative expected log-likelihood Hessian.

Check Your Understanding

Answer each question, then compare with the short solutions below.

For one Bernoulli observation with response probability $p$, write the score for an observed value $y$.
At $p=0.25$, calculate the Fisher information from one Bernoulli observation.
State two conditions used to justify the negative expected Hessian identity.
For a two-dimensional score, why is the outer product $UU^\mathsf{T}$ a matrix, and what does its expectation define?
Explain why a score outer product from one data set is not generally Fisher information.

Solutions. (1) $U=y/p-(1-y)/(1-p)$. (2) $I(p)=1/[p(1-p)]=1/(0.25\times0.75)=5.333$ per observation. (3) Suitable conditions include parameter-independent support, twice differentiability, and permission to interchange differentiation with integration; any two correctly stated conditions suffice. (4) If $U$ has two components, $UU^\mathsf{T}$ contains all pairwise component products; its expectation is the two-by-two Fisher information matrix. (5) The realized outer product depends on the particular observed data. Information is its expectation over data generated under the model at the specified parameter value.