Auditing Sufficiency and Information Calculations

Why Audit the Calculation?

Sufficiency and Fisher information are both read from the likelihood, but they answer different questions. A sufficient statistic retains all the sample’s information about the parameter; information quantifies how sensitive the likelihood is to that parameter. Errors often arise from a plausible-looking shortcut: a statistic may summarize the data without being sufficient, and the negative second derivative for one observed sample need not equal its expectation.

This tutorial checks these calculations for independent Bernoulli and Poisson observations. Throughout, the sample size \(n\) is fixed, and the observations are independent and identically distributed within each model. We restrict the Bernoulli parameter to \(0<p<1\) and the Poisson parameter to \(\lambda>0\), so the likelihoods have common support and are differentiable in the parameter.

Definition: A statistic \(T(X_1,\ldots,X_n)\) is sufficient for a parameter if, conditional on \(T\), the distribution of the full sample does not depend on that parameter. The score is \(U(\theta)=\partial\ell(\theta)/\partial\theta\). The observed information is \(J(\theta)=-\partial^2\ell(\theta)/\partial\theta^2\), evaluated at the observed data. The Fisher information is \(I(\theta)=E[J(\theta)]\), when this expectation exists.

For a regular model, with differentiation permitted under the sum or integral defining total probability, the expected information also equals the score variance: \(I(\theta)=\operatorname{Var}(U(\theta))\). In the discrete models below, the support does not depend on the parameter, and the relevant sums and derivatives can be evaluated directly. The examples derive the formulas rather than relying on the identity alone.

First Check: What Does the Likelihood Depend On?

A useful sufficiency audit begins by writing the joint likelihood and separating its parameter-dependent and data-only parts. In these fixed-size samples, the sum of the observations is sufficient. The sample mean contains exactly the same information as the sum when \(n\) is fixed, since each can be recovered from the other. A different summary should not be declared sufficient just because it seems natural or compact.

Worked Example: Bernoulli Data and the Success Count

Suppose five independent observations are \(x=(1,0,1,1,0)\), where each \(X_i\) is Bernoulli with success probability \(p\). The observed number of successes is \(S=\sum_i x_i=3\). Check the sufficiency claim and calculate the information quantities.

The joint likelihood and log-likelihood are

L(p;x)=\prod_{i=1}^5p^{x_i}(1-p)^{1-x_i} =p^S(1-p)^{5-S} =p^3(1-p)^2, \qquad \ell(p)=3\log p+2\log(1-p).

The likelihood depends on the data only through \(S\), so the factorization criterion gives sufficiency. The conditional argument is also direct: given \(S=s\), every binary sequence with \(s\) successes has probability \(p^s(1-p)^{5-s}\) before conditioning. There are \(\binom{5}{s}\) such sequences, so each has conditional probability \(1/\binom{5}{s}\), independent of \(p\).

Differentiating gives the score and observed information:

U(p)=\frac{3}{p}-\frac{2}{1-p}, \qquad J(p)=\frac{3}{p^2}+\frac{2}{(1-p)^2}.

At \(p=0.4\), \(J(0.4)=3/0.16+2/0.36=18.75+5.5556\approx24.3056\). The expected information for five observations is \(5/[p(1-p)]\), which at \(p=0.4\) is \(5/0.24\approx20.8333\). These are not equal for this data set and parameter value: observed information is a function of the realized sample, while expected information averages over possible samples.

Here is a direct verification of the Bernoulli expected-information formula. For a sample with \(n\) observations and \(S\) successes,

U(p)=\frac{S-np}{p(1-p)}, \qquad J(p)=\frac{S}{p^2}+\frac{n-S}{(1-p)^2}.

Since \(E[S]=np\), taking the expectation of \(J(p)\) gives \(n/p+n/(1-p)=n/[p(1-p)]\). Also, \(S\) has variance \(np(1-p)\), so the variance of \(U(p)\) is \(np(1-p)/[p^2(1-p)^2]=n/[p(1-p)]\). Thus both calculations give the same expected information. They do not imply that \(J(p)\) equals \(I(p)\) for each realized sample.

Poisson Samples: Check the Candidate Statistic

For independent Poisson observations with mean \(\lambda\), the likelihood has the form \(e^{-n\lambda}\lambda^S\) times a factor that does not involve \(\lambda\), where \(S=\sum_i X_i\). That parameter-dependent form identifies the sum as sufficient. To audit a competing summary, compare samples with the same value of that summary: if their likelihood ratio depends on \(\lambda\), the summary has not removed all information about \(\lambda\).

Worked Example: Poisson Sum, Product, and Information Scaling

Take four observations \(x=(0,1,2,3)\) from independent Poisson distributions with mean \(\lambda\). Their sum is \(S=6\). Check the sufficient statistic, then compare the observed and expected information at \(\lambda=2\).

The likelihood and log-likelihood are

L(\lambda;x) =\prod_{i=1}^4\frac{e^{-\lambda}\lambda^{x_i}}{x_i!} =\frac{e^{-4\lambda}\lambda^6}{0!\,1!\,2!\,3!} =\frac{e^{-4\lambda}\lambda^6}{12}, \qquad \ell(\lambda)=-4\lambda+6\log\lambda-\log 12.

All dependence on the sample in the parameter-dependent part is through \(S\), so \(S\) is sufficient. The sample mean \(S/4\) is equivalent because the sample size is fixed. Differentiating,

U(\lambda)=-4+\frac{6}{\lambda}, \qquad J(\lambda)=\frac{6}{\lambda^2}.

At \(\lambda=2\), the observed information is \(J(2)=6/4=1.5\). The expected information for four observations is \(4/\lambda\), hence \(I(2)=2\). Per observation, these values are \(1.5/4=0.375\) and \(2/4=0.5\), respectively. Dividing by \(n\) changes total information into information per observation; it does not change observed information into expected information.

The product is not a sufficient replacement for the sum. For two observations, the vectors \((1,4)\) and \((2,2)\) both have product \(4\), but their sums are \(5\) and \(4\). Their likelihood ratio, in that order, is

\frac{L(\lambda;(1,4))}{L(\lambda;(2,2))} = \frac{e^{-2\lambda}\lambda^5/(1!\,4!)} {e^{-2\lambda}\lambda^4/(2!\,2!)} =\frac{\lambda}{6}.

This ratio depends on \(\lambda\). Therefore, knowing the product does not make the conditional distribution of the data independent of \(\lambda\), and the product is not sufficient. A compact summary is not necessarily an informative-enough summary.

The Poisson information formulas can be checked in two ways. For a sample with sum \(S\), the score is \(U(\lambda)=S/\lambda-n\) and the observed information is \(J(\lambda)=S/\lambda^2\). Since \(E[S]=n\lambda\), \(E[J(\lambda)]=n/\lambda\). Also, \(\operatorname{Var}(S)=n\lambda\), so \(\operatorname{Var}(U(\lambda))=n/\lambda\). As with the Bernoulli model, this equality concerns an expectation over samples; it is not an equality between observed and expected information for every sample.

A Sample-Level Check of Observed Versus Expected Information

A second sample-level audit can make the distinction especially clear. The observed information uses the realized number of successes or events. Expected information substitutes its expectation, which averages over possible data sets under the model. Both are positive in these examples, but they need not have the same numerical value.

Worked Example: All Failures in a Bernoulli Sample

Suppose \(n=4\), \(p=0.25\), and the observed data are \((0,0,0,0)\), so \(S=0\). Calculate the score, observed information, and expected information, and check the score-variance calculation.

The log-likelihood is \(4\log(1-p)\). Thus

U(p)=-\frac{4}{1-p}, \qquad J(p)=\frac{4}{(1-p)^2}.

At \(p=0.25\), the score is \(-4/0.75=-16/3\), and the observed information is \(4/0.75^2=64/9\approx7.111\). By contrast, the expected information is

I(p)=\frac{4}{p(1-p)} =\frac{4}{0.25(0.75)} =\frac{64}{3} \approx21.333.

For the variance check, \(S\) has variance \(np(1-p)=0.75\), and the score for any outcome is \(U(p)=(S-np)/[p(1-p)]\). Hence

\operatorname{Var}(U(p)) =\frac{0.75}{[0.25(0.75)]^2} =\frac{64}{3} =I(p).

The all-failure sample has lower observed information than the model’s expected information at \(p=0.25\). The two values answer different questions, so the difference is not a contradiction.

A Reliable Audit Sequence

When checking a sufficiency or information calculation, separate the tasks rather than trying to infer everything from one expression. The following sequence catches most sign, statistic, and scaling errors.

Write the joint likelihood.
Include the full sample-dependent terms, even those that do not involve the parameter.

Audit the proposed statistic.
Identify the data summaries in the parameter-dependent likelihood. For a competing statistic, compare likelihoods for samples that share its value.

Differentiate before changing signs.
Compute the score \(U\), then the second derivative \(\ell''\), and only then set \(J=-\ell''\).

Label the information being reported.
Keep observed \(J\), expected \(I\), total-sample information, and per-observation information distinct.

Cross-check by expectation or variance.
Under the stated regularity conditions, verify \(E[J]=\operatorname{Var}(U)\); in these models, compute both directly.

Common Mistakes

Calling any summary sufficient. A product or average may look informative, but sufficiency depends on the likelihood or conditional distribution. In the fixed-\(n\) examples, the mean works because it is a one-to-one rescaling of the sum.
Dropping a minus sign. The log-likelihood often has negative second derivative, while observed information is its negative. For example, \(\ell''(\lambda)=-S/\lambda^2\), so \(J(\lambda)=S/\lambda^2\), not \(-S/\lambda^2\).
Equating observed with expected information. \(J\) depends on the realized data; \(I=E[J]\) averages over data generated under the model. The all-failure Bernoulli sample demonstrates that the values can differ substantially.
Mixing total and per-observation information. For \(n\) independent identical observations, total information is \(n\) times the one-observation information. State whether a reported quantity is total or per observation.
Assuming the score itself is information. The score can be positive or negative for a particular sample. Information here is defined by negative curvature or, in expectation, score variance; it is not the realized score.

Key takeaway: Check sufficiency by tracking parameter dependence in the full likelihood, and check information by keeping the score, negative curvature, expectation, and sample-size scale separate. For Bernoulli and Poisson samples, the sum is sufficient, but observed and expected information need not agree for the observed data.

Check Your Understanding

Use the likelihood and the distinctions in this tutorial to answer the following.

For \(n\) Bernoulli observations, explain why the sample mean and success count are equivalent sufficient statistics when \(n\) is fixed.
For independent Poisson observations, what feature of the likelihood shows that the sum is sufficient?
In the Poisson example with sum \(S=6\), calculate the observed information at \(\lambda=3\) and the expected information at \(\lambda=3\). Are they equal?
Why does a likelihood ratio that depends on the parameter for two data sets sharing a proposed statistic show that the statistic is not sufficient?
State one calculation that checks the Bernoulli Fisher information using the score, and one that checks it using observed curvature.