Fisher Information in Bernoulli Samples

Why information depends on the probability being evaluated

For a Bernoulli sample, the likelihood is simple enough to calculate exactly, but its curvature depends on both the proposed success probability and the observed data. This makes it a useful setting for keeping three quantities distinct: the score, which is the first derivative of the log-likelihood; the observed information, which is its negative second derivative; and the expected Fisher information, which averages that curvature over possible samples at a specified probability.

The observed information is calculated from the data actually seen. Expected information is calculated under the model at a probability \(p\), before observing the sample. They need not agree at an arbitrary \(p\). For an interior maximum-likelihood estimate in the Bernoulli model, however, they do agree; the calculation also shows why special coincidences can occur away from that estimate.

Likelihood, score, and observed information

Let \(X_1,\ldots,X_n\) be independent and identically distributed Bernoulli random variables with success probability \(p\), where \(0<p<1\). Thus \(P(X_i=1)=p\) and \(P(X_i=0)=1-p\). Write \(x_1,\ldots,x_n\) for the observed values and \(S=\sum_{i=1}^n x_i\) for the number of successes. Since each \(x_i\) is either zero or one, the number of failures is \(n-S\).

For a fixed observed sample, its likelihood as a function of \(p\) is the product of the individual Bernoulli probabilities. Its log is therefore

L(p)=p^S(1-p)^{n-S}, \qquad \ell(p)=S\log p+(n-S)\log(1-p).

The score is the first derivative of \(\ell(p)\); the observed information is the negative of its second derivative. Both are functions of the observed count \(S\) and the probability at which they are evaluated.

Definition: For this Bernoulli sample, the score and observed information at \(p\in(0,1)\) are \(U(p)=\ell'(p)\) and \(J(p)=-\ell''(p)\), respectively. Expected Fisher information is \(I(p)=E_p[J(p)]\), where the expectation is over samples generated with success probability \(p\).

Differentiating the log-likelihood gives

U(p)=\frac{S}{p}-\frac{n-S}{1-p} =\frac{S-np}{p(1-p)}, \qquad J(p)=\frac{S}{p^2}+\frac{n-S}{(1-p)^2}.

The score is positive when \(S>np\), negative when \(S<np\), and zero when \(S=np\). Its sign indicates the direction in which a small change in \(p\) would increase the log-likelihood. The observed information is nonnegative, and for an interior \(p\) it is positive for every sample of size \(n>0\).

Worked Example: Score and Observed Information at an Arbitrary Probability

Suppose \(n=10\) Bernoulli trials produce \(S=6\) successes. Evaluate the score and observed information at \(p=0.4\), and compare them with the expected information there.

The score is \(U(0.4)=6/0.4-4/0.6=15-6.6667=8.3333\). Equivalently, \(S-np=6-10(0.4)=2\) and \(p(1-p)=0.24\), so \(U(0.4)=2/0.24=8.3333\).

The observed information is \(J(0.4)=6/(0.4)^2+4/(0.6)^2=6/0.16+4/0.36=37.5+11.1111=48.6111\). As derived below, expected information is \(I(0.4)=10/[0.4(0.6)]=10/0.24=41.6667\). The two information values differ at this probability for this sample.

Expected Fisher information

Under the model with probability \(p\), the success count has expectation \(E_p[S]=np\). Taking the expectation of the observed information therefore gives

\begin{aligned} I(p) &=E_p[J(p)]\\ &=\frac{E_p[S]}{p^2}+\frac{n-E_p[S]}{(1-p)^2}\\ &=\frac{np}{p^2}+\frac{n(1-p)}{(1-p)^2}\\ &=\frac{n}{p}+\frac{n}{1-p} =\frac{n}{p(1-p)}. \end{aligned}

This calculation establishes expected information directly from its definition. It can also be checked as the variance of the score. Indeed, \(U(p)=(S-np)/[p(1-p)]\); since \(S\) is a sum of \(n\) independent Bernoulli variables, \(\operatorname{Var}_p(S)=np(1-p)\). Consequently,

E_p[U(p)]=0, \qquad \operatorname{Var}_p(U(p)) =\frac{np(1-p)}{p^2(1-p)^2} =\frac{n}{p(1-p)} =I(p).

The equality of expected information and score variance is consistent with the expected information identity. Here the direct calculations also make clear which probability is used: both expectations are taken under the Bernoulli model at \(p\). Expected information depends on \(p\), even though it does not depend on the particular observed count.

Result: For an independent Bernoulli sample of fixed size \(n\), with \(0<p<1\), expected Fisher information is \(I(p)=n/[p(1-p)]\). It is the same whether calculated as the expected observed information or as the variance of the score.

What changes at the maximum-likelihood estimate?

When \(0<S<n\), the log-likelihood is strictly concave because \(\ell''(p)=-J(p)<0\). Setting the score equal to zero gives \(S-np=0\), so the unique maximum-likelihood estimate is \(\hat p=S/n\). At this value, the observed information is

\begin{aligned} J(\hat p) &=\frac{S}{(S/n)^2} +\frac{n-S}{(1-S/n)^2}\\ &=\frac{n^2}{S}+\frac{n^2}{n-S} =\frac{n^3}{S(n-S)}. \end{aligned}

Evaluating expected information at the same estimate gives

I(\hat p) =\frac{n}{(S/n)(1-S/n)} =\frac{n^3}{S(n-S)} =J(\hat p).

Thus, for an interior Bernoulli maximum-likelihood estimate, the observed information and expected information evaluated at that estimate agree. This is an equality of two numerical evaluations for this model, not a general rule that observed and expected information are identical for every sample and every probability.

Worked Example: Information at an Interior Maximum-Likelihood Estimate

A sample of \(n=10\) trials has \(S=6\) successes. Find the maximum-likelihood estimate, the score there, and both information values.

The estimate is \(\hat p=S/n=6/10=0.6\). The score is \(U(0.6)=6/0.6-4/0.4=10-10=0\), as required at an interior maximum.

The observed information is \(J(0.6)=6/(0.6)^2+4/(0.4)^2=6/0.36+4/0.16=16.6667+25=41.6667\). The expected information evaluated at the estimate is \(I(0.6)=10/[0.6(0.4)]=10/0.24=41.6667\). The two calculations agree. In the preceding example, \(J(0.4)\) was \(48.6111\), not \(I(0.4)=41.6667\); the agreement occurs at \(\hat p=0.6\), not at every value of \(p\).

When can the two information values agree?

For arbitrary \(p\in(0,1)\), subtract expected information from observed information. Putting the terms over a common denominator yields

\begin{aligned} J(p)-I(p) &=\frac{S}{p^2}+\frac{n-S}{(1-p)^2} -\frac{n}{p(1-p)}\\ &=\frac{S(1-p)^2+(n-S)p^2-np(1-p)} {p^2(1-p)^2}\\ &=\frac{(S-np)(1-2p)}{p^2(1-p)^2}. \end{aligned}

The denominator is positive for \(0<p<1\). Therefore \(J(p)=I(p)\) exactly when \(S=np\) or \(p=1/2\). The first condition is the score equation and, for \(0<S<n\), identifies the maximum-likelihood estimate. The second condition is a special feature of the Bernoulli calculation: at \(p=1/2\), observed and expected information agree for every possible count, whether or not that \(p\) is the estimate.

Worked Example: Agreement Away From the Estimate

Let \(n=20\), \(S=5\), and evaluate information at \(p=0.5\). The maximum-likelihood estimate is \(5/20=0.25\), so \(0.5\) is not the estimate.

At \(p=0.5\), the observed information is \(J(0.5)=5/0.25+15/0.25=20+60=80\). Expected information is \(I(0.5)=20/[0.5(0.5)]=20/0.25=80\). They agree despite the score being \(U(0.5)=(5-20(0.5))/0.25=-5/0.25=-20\), which is not zero. The difference formula explains this: its factor \(1-2p\) is zero at \(p=0.5\).

Boundary samples and a common mistake

If \(S=0\), the likelihood is \((1-p)^n\), which increases as \(p\) decreases toward zero; its maximum over the closed probability interval occurs at \(\hat p=0\). If \(S=n\), the likelihood is \(p^n\), maximized at \(\hat p=1\). Neither case has an interior maximum-likelihood estimate, so the interior calculation involving \(S\) and \(n-S\) in the denominator does not apply. The formula \(I(p)=n/[p(1-p)]\) is defined for \(0<p<1\), and it is not a finite information value at either boundary.

Common mistake: “Observed information equals expected information, so I can replace one by the other at any probability.” This is incorrect. For example, with \(n=10\), \(S=6\), and \(p=0.4\), the observed value is \(48.6111\) while the expected value is \(41.6667\). The correction is to distinguish the data-based \(J(p)\) from the model average \(I(p)\), and to evaluate each at the probability of interest. For Bernoulli data they agree at the interior estimate, and also in the special case \(p=1/2\); neither fact licenses replacing one by the other in general.

Key takeaway: For \(n\) Bernoulli observations, \(U(p)=(S-np)/[p(1-p)]\), \(J(p)=S/p^2+(n-S)/(1-p)^2\), and \(I(p)=n/[p(1-p)]\). At an interior estimate \(\hat p=S/n\), \(J(\hat p)=I(\hat p)\); at an arbitrary probability, they need not agree.

Check Your Understanding

Use the formulas and distinctions developed above to answer these questions.

For \(n=12\) and \(S=3\), write the score and observed-information formulas at an arbitrary \(p\).
At \(p=0.25\) in that sample, calculate the score, observed information, and expected information.
For which interior probability values do observed and expected information agree when \(n=12\) and \(S=3\)? Explain using the difference formula.
Why does the interior maximum-likelihood calculation require \(0<S<n\)? What estimate occurs if \(S=0\)?
Explain why equality of observed and expected information at \(p=1/2\) does not imply that \(p=1/2\) is the maximum-likelihood estimate.