Comparing Bernoulli, Binomial, and Poisson Representations

Three Discrete Models, One Canonical Pattern

Bernoulli, binomial, and Poisson distributions all describe discrete responses, but they represent different kinds of counts. A Bernoulli response records one success or failure. A binomial response records the number of successes in a fixed number of trials. A Poisson response records a nonnegative integer count without a fixed upper bound. Their canonical forms make these differences visible through the support, the natural parameter, the log-partition function, and the carrier term.

For a one-parameter exponential family, the form used here is

f(y;\eta)=h(y)\exp\{y\eta-A(\eta)\}.

Here \(y\) is the observed response, \(\eta\) is the natural parameter, \(A(\eta)\) is the log-partition function, and \(h(y)\) is the parameter-free carrier term. For the binomial distribution, the fixed trial count \(m\) is part of the specification, so both the support and \(A(\eta)\) depend on \(m\).

Canonical Forms Side by Side

Definition: In the canonical form \(f(y;\eta)=h(y)\exp\{y\eta-A(\eta)\}\), the natural parameter is \(\eta\), the log-partition function is \(A(\eta)\), and the carrier term is \(h(y)\). The carrier does not depend on the parameter. The support and any fixed design features, such as a binomial trial count, must be stated as part of the model.

Distribution	Support	Natural parameter \(\eta\)	Log-partition \(A(\eta)\)	Carrier \(h(y)\)
Bernoulli with success probability \(p\)	\(y\in\{0,1\}\)	\(\log\{p/(1-p)\}\)	\(\log(1+e^\eta)\)	1
Binomial with fixed trial count \(m\) and success probability \(p\)	\(y\in\{0,\ldots,m\}\)	\(\log\{p/(1-p)\}\)	\(m\log(1+e^\eta)\)	\(\binom{m}{y}\)
Poisson with rate \(\lambda\)	\(y\in\{0,1,2,\ldots\}\)	\(\log\lambda\)	\(e^\eta\)	\(1/y!\)

The Bernoulli and binomial models share the log-odds natural parameter, because both use a success probability \(p\). The Bernoulli model is the binomial model with \(m=1\): its trial-count factor in \(A(\eta)\) is one, and its carrier is \(\binom{1}{y}=1\) on its support. The Poisson model has a different natural parameter, the log-rate, and a different carrier.

A carrier is not an optional correction. For example, \(\binom{m}{y}\) counts how many ordered sequences of \(m\) binary outcomes produce a total of \(y\) successes. The factor \(1/y!\) is required for the Poisson probabilities to sum to one. In each case the carrier is independent of the parameter, but it can depend on the response and, for the binomial, on the fixed trial count.

Worked Example: Writing a Bernoulli Model in Canonical Form

Let \(Y\) be Bernoulli with \(p=0.8\), so \(P(Y=1)=0.8\) and \(P(Y=0)=0.2\). The natural parameter is

\eta=\log\left(\frac{p}{1-p}\right) =\log\left(\frac{0.8}{0.2}\right)=\log 4.

The log-partition function is

A(\eta)=\log(1+e^\eta)=\log(1+4)=\log 5.

Since \(h(y)=1\) for \(y=0,1\), the canonical mass is \(\exp\{y\log 4-\log 5\}\). At \(y=1\), this is \(4/5=0.8\); at \(y=0\), it is \(1/5=0.2\). Both probabilities agree with the specified distribution and sum to one.

Worked Example: A Binomial Count with a Fixed Trial Count

Suppose \(Y\sim\operatorname{Binomial}(m=4,p=0.25)\). Its natural parameter is

\eta=\log\left(\frac{0.25}{0.75}\right) =\log\left(\frac{1}{3}\right)=-\log 3.

The log-partition function is \(A(\eta)=4\log(1+e^\eta)=4\log(4/3)\). For the observed count \(y=2\), the carrier is \(\binom{4}{2}=6\), and the canonical probability is

\binom{4}{2}\exp\{2\eta-A(\eta)\} =6\left(\frac{1}{3}\right)^2\left(\frac{3}{4}\right)^4 =\frac{27}{128}.

The ordinary binomial expression gives the same check: \(\binom{4}{2}(1/4)^2(3/4)^2=6(1/16)(9/16)=27/128\). The value \(m=4\) appears in the support, in the carrier, and in the log-partition function. Omitting it from \(A\) would not give a normalized binomial model.

Worked Example: A Poisson Count

Let \(Y\sim\operatorname{Poisson}(\lambda=3)\). The natural parameter is \(\eta=\log 3\), and \(A(\eta)=e^\eta=3\). For an observed count \(y=2\), the canonical mass is

\frac{1}{2!}\exp\{2\log 3-3\} =\frac{9e^{-3}}{2}.

This is the usual Poisson probability \(e^{-3}3^2/2!\). Unlike the binomial case, there is no fixed maximum count: the support includes every nonnegative integer. The factorial carrier and the exponential log-partition together normalize these probabilities.

Why the Log-Partition Functions Differ

The log-partition function is chosen so that the canonical expression sums to one over the model’s support. This gives a direct way to verify each form and highlights why the fixed trial count changes the binomial expression.

Result: For \(\eta\in\mathbb{R}\), the canonical forms and normalizing identities are

\begin{aligned} \text{Bernoulli:}\quad& \sum_{y=0}^{1}\exp\{y\eta-\log(1+e^\eta)\}=1,\\ \text{Binomial}(m):\quad& \sum_{y=0}^{m}\binom{m}{y}\exp\{y\eta-m\log(1+e^\eta)\}=1,\\ \text{Poisson:}\quad& \sum_{y=0}^{\infty}\frac{1}{y!}\exp\{y\eta-e^\eta\}=1. \end{aligned}

Proof. For Bernoulli, the sum before dividing by its normalizer is \(1+e^\eta\): the \(y=0\) term is \(1\), and the \(y=1\) term is \(e^\eta\). Thus \(A(\eta)=\log(1+e^\eta)\) makes the sum equal to one.

For the binomial, factor out \(\exp\{-m\log(1+e^\eta)\}\). The remaining finite sum is \(\sum_{y=0}^{m}\binom{m}{y}(e^\eta)^y=(1+e^\eta)^m\), by the binomial theorem. The factor outside the sum is \((1+e^\eta)^{-m}\), so their product is one.

For the Poisson, factor out \(\exp\{-e^\eta\}\). The remaining series is \(\sum_{y=0}^{\infty}(e^\eta)^y/y!=\exp(e^\eta)\), by the power-series expansion of the exponential function. Multiplication gives one. All three normalizers are finite and positive for every real \(\eta\), so each has natural-parameter space \(\mathbb{R}\). \(\square\)

How Independent Bernoulli Responses Produce a Binomial Count

The binomial carrier term has a useful interpretation: it appears when a count combines a fixed number of individual binary outcomes. This connection requires a fixed number of trials and independent, identically distributed Bernoulli outcomes.

Theorem: If \(X_1,\ldots,X_m\) are mutually independent Bernoulli random variables with common success probability \(p\), then \(Y=\sum_{i=1}^{m}X_i\) has a \(\operatorname{Binomial}(m,p)\) distribution.

Proof. Fix an integer \(y\) with \(0\leq y\leq m\). For any particular sequence of \(m\) outcomes containing exactly \(y\) successes, independence gives its probability as \(p^y(1-p)^{m-y}\). There are \(\binom{m}{y}\) distinct sequences with exactly \(y\) successes: choose which \(y\) of the \(m\) positions are successes. These sequences are disjoint, so their probabilities add:

P(Y=y)=\binom{m}{y}p^y(1-p)^{m-y}.

For \(y\) outside \(0,\ldots,m\), the probability is zero because a sum of \(m\) binary outcomes cannot take such a value. The displayed mass function is the binomial distribution. Substituting \(p=e^\eta/(1+e^\eta)\) and \(1-p=1/(1+e^\eta)\) yields

\binom{m}{y}p^y(1-p)^{m-y} =\binom{m}{y}\exp\{y\eta-m\log(1+e^\eta)\},

which identifies its carrier and log-partition function in canonical form. \(\square\)

The result does not say that every count of successes is binomial. The fixed number of trials, independence, and common success probability are part of its assumptions. If the responses have different success probabilities, for instance, the simple binomial expression generally does not apply.

Reading the Comparison in Practice

When identifying one of these representations, check the support first, then the parameter mapping, and finally the carrier and normalizer. This order reduces common mistakes because the support often reveals whether the response is a single indicator, a bounded total, or an unbounded count.

Identify the response support.
A binary support suggests Bernoulli; a count from zero through a fixed \(m\) suggests binomial; an unbounded nonnegative integer count is compatible with Poisson.

Map the model parameter to \(\eta\).
For Bernoulli and binomial, use log-odds \(\log\{p/(1-p)\}\). For Poisson, use the log-rate \(\log\lambda\).

Check every parameter-free factor.
Retain the binomial coefficient or the Poisson factorial in \(h(y)\), and verify that \(A(\eta)\) normalizes the mass over the stated support.

A frequent error is to say that the binomial and Bernoulli forms have identical log-partition functions because they use the same log-odds. They share the same natural parameter, but for a fixed binomial trial count \(m\), \(A(\eta)=m\log(1+e^\eta)\). The Bernoulli expression is only the special case \(m=1\). Another error is to treat the Poisson model as a binomial with an unspecified or very large trial count. A Poisson support has no finite upper bound, while every fixed-\(m\) binomial count is at most \(m\); these are different probability models, even when they may be useful approximations in other settings.

Key takeaway: Bernoulli and binomial responses use log-odds, while Poisson counts use a log-rate. The binomial trial count multiplies the log-partition function and determines the support; the carrier terms preserve the correct count probabilities in all three canonical forms.

Check Your Understanding

Use the supports and canonical forms to answer each question.

For a binomial response with fixed trial count \(m\), state its support, natural parameter, log-partition function, and carrier term.
Why does a Bernoulli model have the same natural parameter as a binomial model but a different-looking log-partition function when \(m>1\)?
For a Poisson response with rate \(\lambda\), write \(\eta\), \(A(\eta)\), and \(h(y)\).
Which assumptions are needed for the sum of \(m\) Bernoulli variables to have a binomial distribution?
Explain why the binomial coefficient cannot be dropped from the canonical mass for a binomial count.