The Bernoulli Distribution in Exponential-Family Form

Why Write a Bernoulli Model in Exponential-Family Form?

A Bernoulli random variable records one binary outcome: \(Y=1\) for the event designated as a success and \(Y=0\) otherwise. Its probability mass function is simple, but expressing it in exponential-family form makes its structure explicit. In particular, that form identifies a natural parameter and a log-partition function, whose derivatives give the mean and variance.

The success label is a coding choice, not a claim that the event is desirable. The calculations below assume the outcome has been coded as 1 for the event of interest. The Bernoulli distribution describes one binary response; it does not, by itself, specify how multiple responses are related or how predictors determine their probabilities.

Definition: A random variable \(Y\) has a Bernoulli distribution with success probability \(p\), written \(Y\sim\operatorname{Bernoulli}(p)\), if \(Y\) takes values in \(\{0,1\}\), with \(P(Y=1)=p\) and \(P(Y=0)=1-p\), where \(0<p<1\). Its probability mass function is \(f(y;p)=p^y(1-p)^{1-y}\) for \(y\in\{0,1\}\), and is zero outside that support.

Exponential-Family Representation

In the one-parameter exponential-family representation used here, a probability mass function is written \(f(y;\theta)=h(y)\exp\{T(y)\theta-A(\theta)\}\). The function \(T(y)\) is the sufficient statistic, \(\theta\) is the natural parameter, \(A(\theta)\) is the log-partition function, and \(h(y)\) is the base measure. For the Bernoulli distribution, the support is fixed at \(\{0,1\}\), and the representation can be obtained directly from its mass function.

\begin{aligned} f(y;p) &=p^y(1-p)^{1-y}\\ &=\exp\left\{y\log p+(1-y)\log(1-p)\right\}\\ &=\exp\left\{y\log\left(\frac{p}{1-p}\right)+\log(1-p)\right\}. \end{aligned}

Define \(\theta=\log\{p/(1-p)\}\). Solving for \(p\) gives \(p=e^\theta/(1+e^\theta)\), and therefore \(\log(1-p)=-\log(1+e^\theta)\). Substituting these identities yields the exponential-family form.

f(y;\theta)=\exp\{y\theta-\log(1+e^\theta)\}, \qquad y\in\{0,1\}.

Key identification: For \(Y\sim\operatorname{Bernoulli}(p)\), the sufficient statistic is \(T(Y)=Y\), the natural parameter is \(\theta=\log\{p/(1-p)\}\), the log-partition function is \(A(\theta)=\log(1+e^\theta)\), and the base measure is \(h(y)=1\) on the support \(\{0,1\}\). The natural parameter ranges over all real numbers when \(0<p<1\).

This representation is correctly normalized: summing the expression over its two possible outcomes gives \(e^{-A(\theta)}+e^{\theta-A(\theta)}=1\) when \(A(\theta)=\log(1+e^\theta)\). The function \(A\) is therefore not an arbitrary term; it is exactly the term required to make the two probabilities sum to one.

Mean and Variance From the Log-Partition Function

For a regular exponential family with form \(h(y)\exp\{T(y)\theta-A(\theta)\}\), the mean of the sufficient statistic is \(A'(\theta)\), and its variance is \(A''(\theta)\). Here \(T(Y)=Y\), so those derivatives give the mean and variance of the Bernoulli response itself. The following calculation makes the result explicit.

Differentiate the log-partition function.
For \(A(\theta)=\log(1+e^\theta)\), \(A'(\theta)=e^\theta/(1+e^\theta)\).

Recognize the success probability.
Because \(p=e^\theta/(1+e^\theta)\), the expected response is \(E(Y)=A'(\theta)=p\).

Differentiate again.
\(A''(\theta)=e^\theta/(1+e^\theta)^2\), which equals \(p(1-p)\). Thus \(\operatorname{Var}(Y)=p(1-p)\).

For completeness, the second derivative follows from the quotient rule: \(\{e^\theta/(1+e^\theta)\}'=\{e^\theta(1+e^\theta)-e^{2\theta}\}/(1+e^\theta)^2=e^\theta/(1+e^\theta)^2\). Since \(p=e^\theta/(1+e^\theta)\) and \(1-p=1/(1+e^\theta)\), this is \(p(1-p)\). This agrees with the direct Bernoulli calculation: \(E(Y)=0(1-p)+1p=p\), and \(E(Y^2)=p\), so \(\operatorname{Var}(Y)=E(Y^2)-\{E(Y)\}^2=p-p^2=p(1-p)\).

Result: If \(Y\sim\operatorname{Bernoulli}(p)\), then \(E(Y)=p\) and \(\operatorname{Var}(Y)=p(1-p)\). In exponential-family terms, these are \(A'(\theta)\) and \(A''(\theta)\), respectively. The variance is largest at \(p=1/2\), where it is \(1/4\), and decreases toward zero as \(p\) approaches either boundary.

Worked Calculations

Worked Example: Converting a Success Probability to Exponential-Family Form

Suppose a hypothetical binary response has success probability \(p=0.20\). Find its natural parameter and log-partition function, then use the exponential-family expression to recover both outcome probabilities.

The natural parameter is \(\theta=\log\{p/(1-p)\}=\log(0.20/0.80)=\log(0.25)\approx-1.3863\). The log-partition function is \(A(\theta)=\log(1+e^\theta)=\log(1+0.25)=\log(1.25)\approx0.2231\).

For a success, \(y=1\), so \(f(1;\theta)=\exp\{\theta-A(\theta)\}=\exp(-1.3863-0.2231)\approx0.2000\). For a failure, \(y=0\), so \(f(0;\theta)=\exp\{-A(\theta)\}=\exp(-0.2231)\approx0.8000\). The two probabilities sum to 1, as required. The mean is \(0.20\), and the variance is \(0.20(0.80)=0.1600\).

Worked Example: Recovering the Distribution From a Natural Parameter

For a hypothetical Bernoulli response, suppose the natural parameter is \(\theta=\log 3\). Find the success probability, log-partition function, mean, variance, and probability of observing a failure.

Since \(e^\theta=3\), the success probability is \(p=e^\theta/(1+e^\theta)=3/(1+3)=0.75\). The log-partition function is \(A(\theta)=\log(1+3)=\log 4\approx1.3863\). Therefore the mean is \(A'(\theta)=0.75\), and the variance is \(A''(\theta)=p(1-p)=0.75(0.25)=0.1875\).

For a failure, \(y=0\), so \(P(Y=0)=\exp\{0\theta-A(\theta)\}=\exp(-\log4)=1/4=0.25\). This is consistent with \(1-p=0.25\). A positive natural parameter here corresponds to success odds greater than 1: \(e^\theta=3\), so the success probability exceeds one half.

Worked Example: Changing Which Outcome Is Called a Success

Suppose a hypothetical event has probability \(0.64\), and initially \(Y=1\) denotes that event. Now define \(Y^*=1-Y\), so that \(Y^*=1\) denotes its complement. Find the natural parameter under each coding and check the variance.

Under the original coding, \(p=0.64\), so \(\theta=\log(0.64/0.36)=\log(16/9)\approx0.5754\). Under the reversed coding, the success probability is \(p^*=0.36\), giving \(\theta^*=\log(0.36/0.64)=\log(9/16)\approx-0.5754=-\theta\). The sign changes because the success and failure labels have been exchanged.

The variance under the original coding is \(0.64(0.36)=0.2304\). Under the reversed coding it is \(0.36(0.64)=0.2304\), the same value. This must hold because replacing \(Y\) by \(1-Y\) changes the mean from \(p\) to \(1-p\), but does not change the variance.

Assumptions, Boundaries, and Checks

The representation applies when the response has exactly two coded outcomes, 0 and 1, and a single success probability \(p\) strictly between 0 and 1. The chosen coding must be stated so that the meaning of \(p\) is unambiguous. The formula describes a single Bernoulli variable; if separate observations have different success probabilities, each probability mass function uses its own corresponding parameter.

The assumption \(0<p<1\) is equivalent to a finite natural parameter. At \(p=0\), the outcome is always 0; at \(p=1\), it is always 1. These are valid degenerate probability distributions, but they correspond to the limits \(\theta\to-\infty\) and \(\theta\to+\infty\), not to finite values of \(\theta\). The ordinary finite-parameter exponential-family representation therefore uses the open interval for \(p\).

Useful checks are to verify the support, confirm that the probabilities sum to one, and compare derivatives of \(A\) with direct Bernoulli moments. A mismatch usually points to an algebra or coding error. For example, the statistic must be \(y\), not \(1-y\), under the coding where \(y=1\) denotes success; using the latter statistic without changing the parameterization would reverse the interpretation.

Common Mistakes and Scope

Setting the natural parameter equal to \(p\). The natural parameter is \(\theta=\log\{p/(1-p)\}\), not the success probability. The probability is recovered by \(p=e^\theta/(1+e^\theta)\).
Dropping the log-partition term. The term \(A(\theta)\) normalizes the distribution. Without it, the two probabilities generally do not sum to 1.
Using \(A'(\theta)\) as the variance. The first derivative gives the mean of \(Y\); the second derivative gives its variance.
Ignoring outcome coding. Reversing which event is coded 1 changes \(p\) to \(1-p\) and \(\theta\) to \(-\theta\). It does not change the variance.
Extending the support without changing the model. This Bernoulli representation assigns probability only to 0 and 1. It is not the model for a count that can take values 0, 1, 2, and higher.

The direct probability parameterization \(p\) is often the most immediate way to describe a Bernoulli distribution. The exponential-family parameterization \(\theta\) is useful because its log-partition derivatives expose the response mean and variance in a form shared with other exponential-family distributions. Neither representation replaces the need to choose an appropriate response distribution. For instance, the Bernoulli support is specifically binary; a model for a total number of successes across trials requires a different distributional setup.

Key takeaway: A Bernoulli response has exponential-family form \(f(y;\theta)=\exp\{y\theta-A(\theta)\}\), with \(\theta=\log\{p/(1-p)\}\) and \(A(\theta)=\log(1+e^\theta)\). Consequently, \(E(Y)=A'(\theta)=p\) and \(\operatorname{Var}(Y)=A''(\theta)=p(1-p)\).

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For \(p=0.40\), calculate \(\theta\) and \(A(\theta)\). What probabilities does the exponential-family expression assign to 0 and 1?
If \(\theta=0\), find \(p\), the mean, and the variance.
Why does the Bernoulli model with \(p=0\) not correspond to a finite natural parameter?
If the success and failure labels are exchanged, what happens to \(p\), \(\theta\), and the variance?

Solutions. 1. \(\theta=\log(0.40/0.60)=\log(2/3)\approx-0.4055\), and \(A(\theta)=\log(1+2/3)=\log(5/3)\approx0.5108\). Thus \(f(1)=\exp(\theta-A)=0.40\) and \(f(0)=\exp(-A)=0.60\). 2. \(p=e^0/(1+e^0)=1/2\), so the mean is \(0.5\) and the variance is \(0.5(0.5)=0.25\). 3. \(\theta=\log\{p/(1-p)\}\) tends to \(-\infty\) as \(p\) tends to 0; \(p=0\) therefore has no finite \(\theta\). 4. The new probability is \(1-p\), the new natural parameter is \(\log\{(1-p)/p\}=-\theta\), and the variance stays \(p(1-p)\).