Bernoulli, Binomial, and Poisson Likelihoods

Why These Three Likelihoods Belong Together

Bernoulli, binomial, and Poisson models describe familiar forms of discrete data: a binary outcome, a count of successes in a fixed number of trials, and a count over an observation or time interval. Their likelihoods look different when written in their usual parameters, but each can be expressed in canonical exponential-family form. That common form makes it straightforward to identify the statistic carrying the parameter-dependent information and to derive the score and Fisher information.

Throughout, observations are independent when a sample is specified. For a binomial model, the number of trials \(m\) is fixed and known; for a Poisson model, the rate parameter is positive. We first use interior parameter values, so \(0<p<1\) and \(\lambda>0\).

Canonical Forms and Sufficient Statistics

The following representations use the usual probability parameter \(p\) for Bernoulli and binomial models, and rate \(\lambda\) for the Poisson model. The natural parameters are obtained by taking the coefficient of the statistic in the exponent.

For Bernoulli data, \(P(X=x)=p^x(1-p)^{1-x}\), for \(x\in\{0,1\}\). Since \(p=e^\eta/(1+e^\eta)\), this becomes \(\exp\{\eta x-\log(1+e^\eta)\}\). For a binomial count, the probability mass function is \(\binom{m}{x}p^x(1-p)^{m-x}\), so the same substitution gives \(\binom{m}{x}\exp\{\eta x-m\log(1+e^\eta)\}\).

For a Poisson count, \(P(X=x)=e^{-\lambda}\lambda^x/x!\). With \(\lambda=e^\eta\), this is \((1/x!)\exp\{\eta x-e^\eta\}\). Thus the binomial coefficient and the factorial are part of \(h(x)\), not of the natural parameter or log-partition function.

Proof. The factorization criterion says a statistic \(T\) is sufficient when the joint probability mass function can be factored into a function of the data through \(T\) and the parameter, multiplied by a factor that does not depend on the parameter. For independent Bernoulli observations \(X_1,\ldots,X_n\), let \(S=\sum_i X_i\). Their joint mass function is \(p^S(1-p)^{n-S}=\exp\{\eta S-nA(\eta)\}\), so its parameter dependence is through \(S\). For a single binomial observation, its likelihood is \(\binom{m}{x}\exp\{\eta x-A(\eta)\}\), and its parameter dependence is through \(x\). For independent Poisson observations, the joint mass function is \(\exp\{\eta\sum_i X_i-ne^\eta\}\prod_i(1/X_i!)\), whose parameter dependence is through \(\sum_iX_i\). In each case, the remaining factor is free of the parameter, establishing sufficiency. \(\square\)

Scores and Fisher Information

For a canonical model, the score is the derivative of the log-likelihood with respect to the natural parameter. In these three examples, differentiating the log-partition function produces the mean of the sufficient statistic, while its second derivative gives its variance. The next result records those calculations for the models at hand.

Proof. Ignoring terms independent of \(\eta\), the joint log-likelihood is \(\ell(\eta)=\eta T_\Sigma-nA(\eta)+C\), where \(C\) does not depend on \(\eta\). Differentiation gives \(U(\eta)=T_\Sigma-nA'(\eta)\), and a second differentiation gives \(\ell''(\eta)=-nA''(\eta)\). By definition, Fisher information is the negative expected second derivative, so \(I(\eta)=nA''(\eta)\).

For Bernoulli observations, \(A'(\eta)=e^\eta/(1+e^\eta)=p\) and \(A''(\eta)=p(1-p)\), which are the mean and variance of a Bernoulli variable. For one binomial observation, \(A'(\eta)=mp\) and \(A''(\eta)=mp(1-p)\), the binomial mean and variance. For a Poisson observation, \(A'(\eta)=e^\eta=\lambda\) and \(A''(\eta)=e^\eta=\lambda\), its mean and variance. Independence makes the information for \(n\) observations \(n\) times the one-observation information. \(\square\)

The canonical score equations are therefore \(T_\Sigma-np=0\) for Bernoulli data, \(x-mp=0\) for a single binomial count, and \(T_\Sigma-n\lambda=0\) for Poisson data. Solving these equations gives the familiar interior estimates: the sample proportion for Bernoulli observations, \(x/m\) for a binomial count, and the sample mean for Poisson observations. These are score-equation solutions when they lie in the respective parameter spaces.

Information depends on the parameter coordinate. For example, Bernoulli data have information \(n p(1-p)\) in the natural parameter \(\eta\), but \(n/\{p(1-p)\}\) in the probability parameter \(p\). The latter follows directly by differentiating the log-likelihood in \(p\); it is also the natural-coordinate information multiplied by \((d\eta/dp)^2=1/\{p^2(1-p)^2\}\). The parameterization must therefore be stated when reporting information.

Common Pitfall: Confusing the Natural Parameter With the Usual Parameter

A frequent error is to use \(p\) as though it were the canonical parameter in the Bernoulli and binomial exponential forms. The exponent contains \(x\log\{p/(1-p)\}\), not \(xp\); therefore the natural parameter is the log-odds \(\eta=\log\{p/(1-p)\}\). Similarly, the Poisson natural parameter is \(\log\lambda\), not \(\lambda\). The likelihood can of course be differentiated in the usual parameter, but the resulting score and information formulas must match that coordinate.

A related caution is to treat a score equation as automatically guaranteeing an interior solution. For example, if every Bernoulli outcome equals one, the natural-coordinate score is \(n(1-p)\), which is positive for every finite \(\eta\); it has no finite zero. Recognizing that the interior score equation lacks a solution is different from solving it incorrectly. The behavior of likelihood maxima at parameter-space boundaries requires separate attention.