Canonical Forms for Bernoulli and Binomial Laws

Why These Two Laws Have Canonical Forms

Bernoulli and binomial laws are basic discrete models, but their canonical representations also make a broader point: a probability mass function can separate the parameter-dependent part from the part depending only on the observation. In these two cases, the canonical statistic is the observed number of successes, and the log-partition function gives the mean and variance by differentiation.

For a Bernoulli variable, the observation is either zero or one. For a binomial variable, it is a count from zero to a fixed number $n$ of trials. We derive both representations and keep track of a distinction that matters: the binomial coefficient is part of the carrier, not part of the canonical statistic.

Definition: A one-parameter canonical exponential-family mass function has the form $$ f(x\mid\eta)=h(x)\exp\{\eta T(x)-A(\eta)\}, $$ on a support that does not depend on $\eta$. Here $h(x)$ is the carrier, $T(x)$ is the canonical statistic, $\eta$ is the natural parameter, and $A(\eta)$ is the log-partition function. The natural parameter space consists of the values of $\eta$ for which the mass function can be normalized.

Bernoulli Law in Canonical Form

Let $X\sim\operatorname{Bernoulli}(p)$, with $0<p<1$. Its mass function is $f(x)=p^x(1-p)^{1-x}$ for $x\in\{0,1\}$. Taking logarithms of the two parameter factors separates the term multiplied by $x$ from the term that does not depend on $x$:

f(x)=\exp\{x\log p+(1-x)\log(1-p)\} =\exp\left\{x\log\frac{p}{1-p}+\log(1-p)\right\}.

Set $\eta=\log\{p/(1-p)\}$. Solving for $p$ gives $p=e^\eta/(1+e^\eta)$, and therefore $\log(1-p)=-\log(1+e^\eta)$. The canonical form is

f(x\mid\eta)=\exp\left\{\eta x-\log(1+e^\eta)\right\},\qquad x\in\{0,1\}.

Thus $h(x)=1$, $T(x)=x$, and $A(\eta)=\log(1+e^\eta)$. Every real $\eta$ corresponds to exactly one $p\in(0,1)$, so the natural parameter space is $\mathbb{R}$. The endpoints $p=0$ and $p=1$ are not represented by finite $\eta$; they are not included in this regular parameterization.

Worked Example: Rewriting a Bernoulli Mass Function

Suppose $X\sim\operatorname{Bernoulli}(0.3)$. Find its natural parameter and log-partition function, then verify the canonical mass at both possible observations.

The natural parameter is $$ \eta=\log\frac{0.3}{0.7}=\log\frac{3}{7}. $$ The log-partition function at this value is $$ A(\eta)=\log\left(1+\frac{3}{7}\right)=\log\frac{10}{7}. $$

For $x=1$, the canonical expression is $$ \exp\{\eta-A(\eta)\} =\frac{3}{7}\frac{7}{10}=0.3. $$ For $x=0$, it is $$ \exp\{-A(\eta)\}=\frac{7}{10}=0.7. $$ These are the original Bernoulli probabilities, and they sum to one.

Binomial Law: The Carrier Matters

Let $X\sim\operatorname{Binomial}(n,p)$, where $n\geq 1$ is a fixed integer and $0<p<1$. Its support is $\{0,1,\ldots,n\}$, and its mass function is $$ f(x)=\binom{n}{x}p^x(1-p)^{n-x}. $$

Using the same natural parameter $\eta=\log\{p/(1-p)\}$, the parameter-dependent part becomes $\eta x+n\log(1-p)$. Since $\log(1-p)=-\log(1+e^\eta)$, the canonical representation is

f(x\mid\eta)=\binom{n}{x}\exp\left\{\eta x-n\log(1+e^\eta)\right\},\qquad x=0,\ldots,n.

Here $h(x)=\binom{n}{x}$, $T(x)=x$, and $A_n(\eta)=n\log(1+e^\eta)$. The natural parameter space is all of $\mathbb{R}$: the support is finite, and the displayed masses are normalized for every finite real $\eta$. The index $n$ is fixed in this family; changing $n$ changes the support and the carrier.

The carrier is needed for the displayed binomial canonical representation. For $n>1$, omitting $\binom{n}{x}$ generally changes the masses, and the sum of the remaining terms is not the binomial expansion $(1+e^\eta)^n$ that gives the stated $A_n$. For $n=1$, however, $\binom{1}{x}=1$ at both support points, so the carrier is identically one and can be omitted without changing the masses. More generally, the carrier is not optional merely because it does not involve the parameter: it contributes to normalization.

Worked Example: A Binomial Probability in Canonical Coordinates

Let $X\sim\operatorname{Binomial}(5,0.4)$. Express the probability $P(X=2)$ using the canonical form and verify it numerically.

The natural parameter and log-partition function are $$ \eta=\log\frac{0.4}{0.6}=\log\frac{2}{3}, \qquad A_5(\eta)=5\log\left(1+\frac{2}{3}\right)=5\log\frac{5}{3}. $$ Therefore $$ P(X=2)=\binom{5}{2}\exp\{2\eta-A_5(\eta)\} =10\left(\frac{2}{3}\right)^2\left(\frac{3}{5}\right)^5. $$

Evaluating the factors gives $$ 10\cdot\frac{4}{9}\cdot\frac{243}{3125} =\frac{9720}{28125}=0.3456. $$ Directly from the binomial mass function, $$ \binom{5}{2}(0.4)^2(0.6)^3 =10(0.16)(0.216)=0.3456, $$ so the canonical and original calculations agree.

Derivatives of the Log-Partition Function

For a finite-support canonical family, the first two derivatives of the log-partition function give the mean and variance of the canonical statistic. Here the support is finite, so the normalizing sum is a finite sum and can be differentiated term by term without a limit-interchange condition.

Theorem: Suppose $f(x\mid\eta)=h(x)\exp\{\eta T(x)-A(\eta)\}$ is normalized on a finite support for every $\eta$ in an open interval. Then $$ A'(\eta)=E_\eta[T(X)], \qquad A''(\eta)=\operatorname{Var}_\eta(T(X)). $$

To prove the result, define the partition function $Z(\eta)=\sum_x h(x)e^{\eta T(x)}$, where the sum is over the finite support. Normalization gives $A(\eta)=\log Z(\eta)$. Differentiating finite sums,

Z'(\eta)=\sum_x T(x)h(x)e^{\eta T(x)}, \qquad Z''(\eta)=\sum_x T(x)^2h(x)e^{\eta T(x)}.

Dividing the first identity by $Z(\eta)$ gives the expectation under the normalized mass function. For the second derivative, use the derivative of a logarithm:

A'(\eta)=\frac{Z'(\eta)}{Z(\eta)}=E_\eta[T(X)], \qquad A''(\eta)=\frac{Z''(\eta)}{Z(\eta)} -\left(\frac{Z'(\eta)}{Z(\eta)}\right)^2 =E_\eta[T(X)^2]-E_\eta[T(X)]^2.

The final expression is the variance of $T(X)$, proving both identities. $\square$

Means and Variances for Bernoulli and Binomial Laws

For Bernoulli, differentiating $A(\eta)=\log(1+e^\eta)$ gives

A'(\eta)=\frac{e^\eta}{1+e^\eta}=p, \qquad A''(\eta)=\frac{e^\eta}{(1+e^\eta)^2}=p(1-p).

Because $T(X)=X$, these are $E[X]$ and $\operatorname{Var}(X)$. The second equality for the variance follows by substituting $p=e^\eta/(1+e^\eta)$ and $1-p=1/(1+e^\eta)$.

For binomial, $A_n(\eta)=n\log(1+e^\eta)$, so

A_n'(\eta)=n\frac{e^\eta}{1+e^\eta}=np, \qquad A_n''(\eta)=n\frac{e^\eta}{(1+e^\eta)^2}=np(1-p).

Again $T(X)=X$, giving the familiar mean $np$ and variance $np(1-p)$. The derivative calculation obtains both moments from the canonical representation, while the finite-support argument above justifies why these derivatives have that interpretation.

Worked Example: Recovering Binomial Moments

Let $X\sim\operatorname{Binomial}(12,0.25)$. Use the canonical parameterization to find its natural parameter, log-partition function, mean, and variance.

The odds are $p/(1-p)=0.25/0.75=1/3$, so $\eta=\log(1/3)=-\log 3$. Since $e^\eta=1/3$, $$ A_{12}(\eta)=12\log\left(1+\frac{1}{3}\right)=12\log\frac{4}{3}. $$

Differentiating and evaluating at this $\eta$ gives $$ E[X]=A_{12}'(\eta) =12\frac{1/3}{1+1/3} =12\cdot\frac14=3, $$ and $$ \operatorname{Var}(X)=A_{12}''(\eta) =12\frac{1/3}{(1+1/3)^2} =12\cdot\frac{3}{16}=\frac94=2.25. $$ These agree with $np=12(0.25)=3$ and $np(1-p)=12(0.25)(0.75)=2.25$.

Common Mistakes and a Normalization Check

Confusing $p$ with the natural parameter. The natural parameter is $\eta=\log\{p/(1-p)\}$, not $p$. The inverse map is $p=e^\eta/(1+e^\eta)$.
Using the Bernoulli log-partition function for a binomial count. The binomial function is $A_n(\eta)=n\log(1+e^\eta)$; omitting the factor $n$ gives the wrong normalization and moments unless $n=1$.
Dropping the binomial carrier without checking it. For $n>1$, the coefficient $\binom{n}{x}$ contributes to the normalized binomial masses. When $n=1$, it equals one everywhere and has no effect.
Reading $A'(\eta)$ as the mean of any quantity. It is the mean of the canonical statistic $T(X)$. In these representations $T(X)=X$, so it is the mean of the observation.

A useful check is to sum the canonical masses. For Bernoulli the sum is $e^{-A}(1+e^\eta)=1$. For binomial it is $$ e^{-n\log(1+e^\eta)} \sum_{x=0}^n\binom{n}{x}e^{\eta x} = \frac{(1+e^\eta)^n}{(1+e^\eta)^n}=1. $$ The binomial theorem verifies the normalizer directly and shows precisely where the carrier enters.

Key takeaway: Both laws use the count $x$ as their canonical statistic and the log-odds as their natural parameter. Their log-partition functions are $\log(1+e^\eta)$ for Bernoulli and $n\log(1+e^\eta)$ for binomial; the first and second derivatives give the corresponding mean and variance.

Check Your Understanding

Answer each question using the canonical form and its normalization.

For a Bernoulli law with success probability $p$, write the natural parameter and solve for $p$ in terms of that parameter.
For a binomial law with fixed $n$, identify the carrier, canonical statistic, and log-partition function.
Why can the binomial coefficient be omitted when $n=1$, and why does that reasoning not generally apply when $n>1$?
Differentiate the Bernoulli log-partition function twice and state the resulting mean and variance in terms of $p$.
For $X\sim\operatorname{Binomial}(8,0.5)$, use $A_8(\eta)$ to give $E[X]$ and $\operatorname{Var}(X)$ at the corresponding natural parameter.