Calculating Fisher Information in Common Models

Why Fisher Information Matters

An estimator’s sampling variability depends on the probability model and on the parameter being estimated. Fisher information quantifies how sharply the likelihood changes as that parameter changes: a likelihood that changes substantially near the true parameter carries more information about it. This tutorial develops the calculation directly from log-likelihoods for Bernoulli, binomial, Poisson, and normal models. These calculations are aligned with the SOA VEE Mathematical Statistics syllabus.

The main task is to differentiate the log-likelihood with respect to the parameter, then take an expectation under the stated model. Keep the model and parameterization explicit: information about a normal variance is not the same expression as information about its standard deviation, for example.

Definition: For a model with parameter $\theta$, density or probability mass function $f(x;\theta)$, and log-likelihood $\ell(\theta;x)=\log f(x;\theta)$, the score is $U(\theta)=\partial\ell(\theta;x)/\partial\theta$. The Fisher information in the observation is $$ I(\theta)=E_\theta\!\left[U(\theta)^2\right]. $$ When differentiation and expectation can be interchanged and the support does not depend on $\theta$, this is also $$ I(\theta)=-E_\theta\!\left[\frac{\partial^2\ell(\theta;X)}{\partial\theta^2}\right]. $$

Why Expected Curvature Gives the Same Information

The score measures the local change in log-likelihood. The second derivative measures its local curvature. For the standard fixed-support models considered here, their expected forms are connected by a short calculation. Write $f'=\partial f/\partial\theta$ and $f''=\partial^2 f/\partial\theta^2$. Since $\partial\log f/\partial\theta=f'/f$,

\frac{\partial^2\log f}{\partial\theta^2} =\frac{f''}{f}-\left(\frac{f'}{f}\right)^2.

Taking expectation under $f$ gives $\int f''(x;\theta)\,dx-E_\theta[U(\theta)^2]$. Because $\int f(x;\theta)\,dx=1$, differentiating twice gives $\int f''(x;\theta)\,dx=0$, under the stated interchange conditions. Thus the expected second derivative is $-E_\theta[U(\theta)^2]$, which proves the identity. For discrete models, the same argument uses sums instead of integrals.

The expectation in Fisher information is over all possible observations under the model, not just the observed data. The negative second derivative evaluated at the actual data is called observed information; it need not equal Fisher information.

Bernoulli and Binomial Models

Suppose $X\sim\operatorname{Bernoulli}(p)$, where $X=1$ denotes success and $X=0$ denotes failure, with $0<p<1$. Its probability mass function is $f(x;p)=p^x(1-p)^{1-x}$. The log-likelihood for one observation is

\ell(p;x)=x\log p+(1-x)\log(1-p).

Differentiating gives $U(p)=x/p-(1-x)/(1-p)$ and $\partial^2\ell/\partial p^2=-x/p^2-(1-x)/(1-p)^2$. Since $E[X]=p$, the expected negative second derivative is $1/p+1/(1-p)=1/[p(1-p)]$.

Result: For one Bernoulli observation, $I(p)=1/[p(1-p)]$. For $n$ independent Bernoulli observations with common success probability $p$, the log-likelihood contains $n$ Bernoulli terms and the information is $n/[p(1-p)]$.

A binomial observation $Y\sim\operatorname{Binomial}(m,p)$ counts successes among $m$ trials. Its probability mass function is $\binom{m}{y}p^y(1-p)^{m-y}$. The combinatorial factor does not depend on $p$, so it contributes nothing when differentiating the log-likelihood. The score and second derivative are

U(p)=\frac{Y}{p}-\frac{m-Y}{1-p}, \qquad \frac{\partial^2\ell}{\partial p^2} =-\frac{Y}{p^2}-\frac{m-Y}{(1-p)^2}.

Using $E[Y]=mp$ and $E[m-Y]=m(1-p)$, the expected negative second derivative is $m/p+m/(1-p)=m/[p(1-p)]$. A binomial count from $m$ trials therefore has the same information about $p$ as the $m$ individual Bernoulli outcomes.

Worked Example: Information From a Bernoulli Sample

A hypothetical quality check records whether each of 8 independently selected components triggers a warning. Assume each warning indicator is Bernoulli with common probability $p$, and suppose 3 warnings are observed. Find the maximum likelihood estimate of $p$, then calculate the Fisher information for the sample at that estimate.

Method. Use the Bernoulli sample likelihood to find the estimate, then evaluate the model’s expected information formula. The observed count identifies the estimate; it is not itself the definition of Fisher information.

The likelihood, apart from a factor that does not depend on $p$, is $p^3(1-p)^5$. Its log is $3\log p+5\log(1-p)$, so its score is

\frac{3}{p}-\frac{5}{1-p}=0 \quad\Longrightarrow\quad 3(1-p)=5p \quad\Longrightarrow\quad \hat p=\frac{3}{8}=0.375.

For $n=8$, the Fisher information function is $I(p)=8/[p(1-p)]$. Substituting $\hat p=3/8$ gives

I(\hat p)=\frac{8}{(3/8)(5/8)} =\frac{8}{15/64} =\frac{512}{15} \approx 34.133.

The estimated success probability is $0.375$, and the plug-in Fisher information is $34.133$, rounded to three decimal places. This calculation avoids treating the observed score as information: the information used here is the expectation under the Bernoulli model.

Worked Example: A Grouped Binomial Count

A hypothetical laboratory tests 20 independent samples under the same procedure and records only the total number of samples that fail a screen. Assume the failure count $Y\sim\operatorname{Binomial}(20,p)$; 7 failures are observed. Estimate $p$ and calculate the Fisher information at the estimate.

Method. Use the binomial likelihood, where $p$ is the failure probability and the number of trials $m=20$ is fixed. The maximum likelihood estimate is $\hat p=Y/m=7/20=0.35$.

For one binomial count, $I(p)=m/[p(1-p)]$. Therefore,

I(0.35)=\frac{20}{(0.35)(0.65)} =\frac{20}{0.2275} =\frac{8000}{91} \approx 87.912.

The plug-in information is $87.912$, rounded to three decimal places. The factor $m=20$ is essential even though the data are recorded as one count: that count summarizes 20 trials. Omitting $m$ would give the information for one Bernoulli trial, not for this binomial model.

Poisson Model

Let $X\sim\operatorname{Poisson}(\lambda)$, where $\lambda>0$ is the mean count in the specified observation period. For one count $x$, the log-likelihood is $\ell(\lambda;x)=x\log\lambda-\lambda-\log(x!)$. Its score is $x/\lambda-1$, and its second derivative is $-x/\lambda^2$. Since $E[X]=\lambda$, the Fisher information is

Result: One Poisson count with mean parameter $\lambda$ has $I(\lambda)=1/\lambda$. For $n$ independent counts $X_1,\ldots,X_n$ with the same $\lambda$, differentiating their joint log-likelihood gives $I(\lambda)=n/\lambda$.

Worked Example: Information in Repeated Poisson Counts

A hypothetical monitoring system records event counts over 5 equal-length shifts. Assume the counts are independent and each has a Poisson distribution with mean $\lambda$ events per shift. The recorded counts are 2, 0, 3, 1, and 4. Find the maximum likelihood estimate of $\lambda$ and the plug-in Fisher information.

Method. For independent Poisson counts, the joint log-likelihood, after dropping terms not involving $\lambda$, is $(\sum x_i)\log\lambda-5\lambda$. The score is $(\sum x_i)/\lambda-5$. The total count is $2+0+3+1+4=10$, so solving the score equation gives $\hat\lambda=10/5=2$ events per shift.

The Fisher information function for 5 counts is $I(\lambda)=5/\lambda$. At the estimate,

I(\hat\lambda)=\frac{5}{2}=2.5.

The plug-in information is $2.5$, in inverse units of the Poisson mean. As a check, the observed negative second derivative is $(\sum x_i)/\lambda^2=10/4=2.5$ at $\hat\lambda=2$. These values coincide here because the estimate sets $\sum x_i=5\hat\lambda$; they are not generally identical at arbitrary parameter values.

Normal Models: State Which Parameter Is Unknown

For a normal model, the information depends on whether the unknown parameter is the mean or the variance. If $X_1,\ldots,X_n$ are independent with $X_i\sim N(\mu,\sigma^2)$, and $\sigma^2$ is known while $\mu$ is unknown, the log-likelihood terms that depend on $\mu$ are $-\sum_i(x_i-\mu)^2/(2\sigma^2)$. Its first derivative is $\sum_i(x_i-\mu)/\sigma^2$, and its second derivative is $-n/\sigma^2$. Thus,

Result: In a normal sample with known variance $\sigma^2$ and unknown mean $\mu$, the Fisher information for $\mu$ is $I(\mu)=n/\sigma^2$.

If instead the mean $\mu$ is known and the unknown parameter is the variance $\theta=\sigma^2>0$, the log-likelihood is

\ell(\theta) =-\frac{n}{2}\log\theta -\frac{1}{2\theta}\sum_{i=1}^n(X_i-\mu)^2 +\text{a term not depending on }\theta.

Let $Q=\sum_i(X_i-\mu)^2$. The score is $-n/(2\theta)+Q/(2\theta^2)$, and the second derivative is $n/(2\theta^2)-Q/\theta^3$. Under this model $E[Q]=n\theta$; hence the expected second derivative is $n/(2\theta^2)-n/\theta^2=-n/(2\theta^2)$. Therefore the information for the variance parameter is $n/(2\theta^2)$.

Parameterization matters: If the unknown normal parameter is the variance $\theta=\sigma^2$, then $I(\theta)=n/(2\theta^2)$. If it is the standard deviation $\sigma$, then $I(\sigma)=2n/\sigma^2$. Do not substitute a standard deviation into a formula derived for a variance.

Worked Example: Information About a Normal Variance

Assume four independent measurements satisfy $X_i\sim N(4,\theta)$, where the mean $4$ is known and the unknown parameter $\theta$ is the variance. The observations are 2, 4, 4, and 6. Find the maximum likelihood estimate of $\theta$, then calculate the information for $\theta$ at that estimate.

Method. With known mean, the normal variance maximum likelihood estimate is the average squared deviation from that mean. Compute each deviation and square it: $2-4=-2$, with square $4$; $4-4=0$, with square $0$; $4-4=0$, with square $0$; and $6-4=2$, with square $4$. Thus $Q=4+0+0+4=8$, and

\hat\theta=\frac{Q}{n}=\frac{8}{4}=2.

The information formula for the variance parameter is $I(\theta)=n/(2\theta^2)$. Substituting $n=4$ and $\hat\theta=2$ gives

I(\hat\theta)=\frac{4}{2(2^2)} =\frac{4}{8} =0.5.

The plug-in information for the variance parameter is $0.5$. The estimate $2$ is a variance, not a standard deviation. Using the standard-deviation formula with $2$ would answer a different parameterization question.

Choosing the Right Calculation

The following summary gives the model and parameter for each formula. Sample size refers to the number of independent observations or, for the binomial model, the number of trials represented by the count.

Model and unknown parameter	Fisher information	Important condition
One Bernoulli observation, $p$	$1/[p(1-p)]$	$0<p<1$
Binomial count from $m$ trials, $p$	$m/[p(1-p)]$	Fixed $m$; success probability $p$
One Poisson count, $\lambda$	$1/\lambda$	$\lambda>0$
$n$ independent Poisson counts, common $\lambda$	$n/\lambda$	Same mean and exposure period for each count
Normal sample, unknown $\mu$	$n/\sigma^2$	$\sigma^2$ is known
Normal sample, unknown variance $\theta=\sigma^2$	$n/(2\theta^2)$	$\mu$ is known

A reliable calculation follows the same sequence: state the distribution and unknown parameter; write the log-likelihood; differentiate with respect to that parameter; take the model expectation of the squared score or negative second derivative; and only then substitute a parameter value if a numerical plug-in answer is requested. The resulting expression is a function of the parameter, even when the observed sample was used to choose a value at which to evaluate it.

Common Mistakes

Dropping the sample size. For $n$ independent Bernoulli observations the information is $n/[p(1-p)]$, not $1/[p(1-p)]$. The latter is for one observation.
Confusing binomial trials with binomial observations. A single binomial count can represent many trials, so its information includes the number of trials $m$.
Using the observed curvature as the definition. Fisher information is an expectation under the model. Observed information uses the realized data and can differ.
Mixing up variance and standard deviation. For a normal model, identify whether the unknown parameter is $\sigma^2$ or $\sigma$ before selecting a formula.
Taking an expectation under the wrong model. The Poisson calculation uses $E[X]=\lambda$, and the normal variance calculation uses $E[Q]=n\theta$. The expectation must match the stated distribution and parameter.

Key takeaway: Fisher information is calculated from the log-likelihood and averaged under the model. Differentiate with respect to the parameter actually being estimated, preserve the sample-size factor, and distinguish expected information from curvature at the observed data.

Check Your Understanding

For each problem, identify the model and unknown parameter before applying an information formula.

Ten independent Bernoulli trials have unknown success probability $p$. Calculate the Fisher information at $p=0.2$. (Answer: $62.5$)
A single count has distribution $\operatorname{Binomial}(8,p)$. Calculate the Fisher information at $p=0.5$. (Answer: $32$)
Six independent counts each have distribution $\operatorname{Poisson}(\lambda)$. Calculate the Fisher information at $\lambda=3$. (Answer: $2$)
Nine independent measurements have distribution $N(\mu,4)$, where the variance is known and $\mu$ is unknown. Calculate the Fisher information for $\mu$. (Answer: $9/4=2.25$)
Three independent observations have distribution $N(5,\theta)$, where the mean is known and the unknown parameter $\theta$ is the variance. Calculate the Fisher information at $\theta=2$. (Answer: $3/8=0.375$)

Model and unknown parameter	Fisher information	Important condition
One Bernoulli observation, \(p\)	\(1/[p(1-p)]\)	\(0<p<1\)
Binomial count from \(m\) trials, \(p\)	\(m/[p(1-p)]\)	Fixed \(m\); success probability \(p\)
One Poisson count, \(\lambda\)	\(1/\lambda\)	\(\lambda>0\)
\(n\) independent Poisson counts, common \(\lambda\)	\(n/\lambda\)	Same mean and exposure period for each count
Normal sample, unknown \(\mu\)	\(n/\sigma^2\)	\(\sigma^2\) is known
Normal sample, unknown variance \(\theta=\sigma^2\)	\(n/(2\theta^2)\)	\(\mu\) is known

What You'll Learn