Writing The Bernoulli Distribution In Exponential Form

Why Write the Bernoulli Model in Exponential Form?

A Bernoulli response records whether an event occurs: a claim is reported, a policy lapses, or a scheduled payment is missed. Its probability mass function is simple, but expressing it in exponential-family form makes its parameter structure explicit. In particular, the same function that normalizes the probabilities also provides the mean and variance through its derivatives.

This representation is useful when recognizing a response distribution as an exponential-family model. The focus here is the Bernoulli distribution itself: how to rewrite its mass function and identify its natural parameter, cumulant function, mean, and variance.

Definition: A random variable \(Y\) has a Bernoulli distribution with success probability \(p\) if \(Y\) takes values in \(\{0,1\}\), with \(P(Y=1)=p\) and \(P(Y=0)=1-p\), where \(0<p<1\). Its probability mass function is \(P(Y=y)=p^y(1-p)^{1-y}\) for \(y\in\{0,1\}\).

Use the exponential-family form established in the earlier material:

f(y;\theta)=\exp\{y\theta-b(\theta)+c(y)\}.

For the Bernoulli distribution, the term \(c(y)\) will be zero. We will identify \(\theta\) and \(b(\theta)\) by rearranging the probability mass function, not by guessing them.

Rewriting the Bernoulli Probability Mass Function

Start with \(p^y(1-p)^{1-y}\). Taking its logarithm separates the terms involving the observed value \(y\) from the remaining term:

\begin{aligned} \log f(y;p) &= y\log p+(1-y)\log(1-p)\\ &= y\log\left(\frac{p}{1-p}\right)+\log(1-p). \end{aligned}

The coefficient of \(y\) is the log odds of success. Define the natural parameter \(\theta\) to be that coefficient. Since \(\log(1-p)=-\log(1+e^\theta)\), the remaining term is the negative of the cumulant function.

\theta=\log\left(\frac{p}{1-p}\right), \qquad b(\theta)=\log(1+e^\theta), \qquad c(y)=0.

Thus the Bernoulli mass function in exponential-family form is:

f(y;\theta)=\exp\{y\theta-\log(1+e^\theta)\}, \qquad y\in\{0,1\}.

The inverse relationship is \(p=e^\theta/(1+e^\theta)\). It follows because \(e^\theta=p/(1-p)\), so solving for \(p\) gives \(p=e^\theta(1-p)\), and hence \(p(1+e^\theta)=e^\theta\). The natural parameter can be any finite real number when \(0<p<1\).

Key identification: For a Bernoulli response with success probability \(p\), the natural parameter is the log odds \(\theta=\log\{p/(1-p)\}\), the cumulant function is \(b(\theta)=\log(1+e^\theta)\), and the remaining term is \(c(y)=0\).

Deriving the Mean and Variance

For an exponential-family mass function in the form above, the first derivative of the cumulant function gives the mean, and its second derivative gives the variance. For this Bernoulli model, both results can be checked directly from \(b(\theta)\).

b'(\theta)=\frac{e^\theta}{1+e^\theta}=p.

To differentiate again, apply the quotient rule. The numerator becomes \(e^\theta(1+e^\theta)-e^\theta e^\theta=e^\theta\), leaving:

b''(\theta)=\frac{e^\theta}{(1+e^\theta)^2} =\frac{e^\theta}{1+e^\theta} \left(1-\frac{e^\theta}{1+e^\theta}\right) =p(1-p).

Therefore, for \(Y\sim\operatorname{Bernoulli}(p)\), \(E[Y]=b'(\theta)=p\) and \(\operatorname{Var}(Y)=b''(\theta)=p(1-p)\). The variance also follows directly from \(Y^2=Y\) for a variable taking only 0 and 1: \(\operatorname{Var}(Y)=E[Y^2]-E[Y]^2=p-p^2\). The derivative calculation and the direct calculation agree.

Begin with the mass function.
Write \(f(y;p)=p^y(1-p)^{1-y}\), for \(y=0\) or \(y=1\).

Collect the term multiplying the outcome.
The coefficient of \(y\) in the log mass function is \(\log\{p/(1-p)\}\), which identifies \(\theta\).

Identify the normalizing term.
Rewrite the remaining log term as \(-\log(1+e^\theta)\), so \(b(\theta)=\log(1+e^\theta)\).

Differentiate to obtain the moments.
Compute \(b'(\theta)=p\) and \(b''(\theta)=p(1-p)\), giving the mean and variance.

Worked Representations

Worked Example: Convert a Success Probability to Exponential Form

A hypothetical indicator \(Y\) records whether a small equipment fault occurs during a one-day inspection. Assume \(Y\) is Bernoulli with \(p=0.2\). Find \(\theta\) and \(b(\theta)\), then verify the exponential-form probabilities and the mean and variance.

Method. Start from the Bernoulli model with \(0<p<1\); use the log-odds formula for \(\theta\), then substitute into the exponential-form mass function.

Natural parameter. The odds are \(p/(1-p)=0.2/0.8=0.25\). Therefore, \(\theta=\log(0.25)\), approximately \(-1.3863\).

Cumulant function. Since \(e^\theta=0.25\), \(b(\theta)=\log(1+0.25)=\log(1.25)\), approximately \(0.2231\).

Check both outcomes. For \(y=1\), the exponential-form probability is \(\exp(\theta-b(\theta))=0.25/1.25=0.2\). For \(y=0\), it is \(\exp(-b(\theta))=1/1.25=0.8\). The probabilities sum to \(1\).

Moments. The mean is \(b'(\theta)=p=0.2\). The variance is \(b''(\theta)=p(1-p)=0.2(0.8)=0.16\). These agree with the Bernoulli formulas.

Common mistake avoided: Using \(\log(p)\) as the natural parameter would give the wrong coefficient of \(y\). The coefficient is \(\log\{p/(1-p)\}\), here \(\log(0.25)\).

Worked Example: Use Log Odds to Recover the Bernoulli Model

A hypothetical claim indicator has natural parameter \(\theta=\log 3\). Find its success probability, cumulant value, mean, and variance. Then find the probability of observing no claim.

Method. Invert the natural-parameter relationship, then evaluate the cumulant and the mass function at \(y=0\).

Recover \(p\). Since \(e^\theta=3\), \(p=e^\theta/(1+e^\theta)=3/(1+3)=3/4\).

Evaluate \(b(\theta)\). \(b(\theta)=\log(1+e^\theta)=\log 4\).

Find the moments. The mean is \(b'(\theta)=3/4\). The variance is \(b''(\theta)=(3/4)(1/4)=3/16=0.1875\).

Find the no-claim probability. A no-claim outcome is \(y=0\), so its probability is \(\exp\{0\cdot\theta-b(\theta)\}=\exp(-\log 4)=1/4\). This is \(1-p\), as required.

Check: The success and no-claim probabilities are \(3/4\) and \(1/4\), and they sum to \(1\). The variance \(3/16\) is positive and equals \(p(1-p)\).

Worked Example: Verify a Sequence Likelihood

Suppose five hypothetical, independent Bernoulli indicators share the same probability \(p\). Their observed values are \(0,1,0,1,0\), and the model has natural parameter \(\theta=-\log 3\). Find \(p\), \(b(\theta)\), the likelihood of this sequence, and the single-indicator mean and variance.

Method and assumptions. The indicators are independent and have a common Bernoulli parameter. Their joint probability is the product of their five individual probabilities.

Recover the parameter. Here \(e^\theta=1/3\), so \(p=(1/3)/(1+1/3)=1/4\). The cumulant is \(b(\theta)=\log(1+1/3)=\log(4/3)\).

Write the sequence likelihood in exponential form. The observed outcomes sum to \(2\), and there are \(5\) observations. Multiplying the five masses gives

\exp\{2\theta-5b(\theta)\} =\exp\{-2\log 3-5\log(4/3)\} =\frac{27}{1024}.

Verify by direct multiplication. There are two ones and three zeros, so the probability is \(p^2(1-p)^3=(1/4)^2(3/4)^3=27/1024\), matching the exponential-form calculation.

Moments for one indicator. The mean is \(p=1/4\); the variance is \(p(1-p)=(1/4)(3/4)=3/16\).

Common mistake avoided: The cumulant term is included once per observation. Omitting the factor \(5\) in \(5b(\theta)\) would not give the joint probability.

Common Mistakes and Checks

The most reliable check is to substitute \(y=0\) and \(y=1\) into the proposed exponential form. It must return \(1-p\) and \(p\), respectively. The two values must add to \(1\). This catches sign errors in the cumulant term as well as errors in the natural parameter.

Using \(\log p\) instead of log odds. The natural parameter is \(\log\{p/(1-p)\}\), not \(\log p\). The term multiplying \(y\) includes both the success and failure probabilities after the log mass function is expanded.
Reversing the sign of the cumulant. The log mass has \(-b(\theta)\), so the cumulant itself is \(+\log(1+e^\theta)\).
Confusing the natural parameter with the probability. The natural parameter can be negative or greater than one. Recover the probability using \(p=e^\theta/(1+e^\theta)\).
Forgetting the number of observations in a joint likelihood. For \(n\) independent observations with a common \(\theta\), the cumulant contribution is \(-n b(\theta)\), not just \(-b(\theta)\).

The representation assumes a Bernoulli response: each observation is either 0 or 1. It does not describe a count taking values such as 0, 1, 2, and higher. Also, the finite natural parameter formula corresponds to \(0<p<1\); the cases \(p=0\) and \(p=1\) are degenerate distributions and do not have finite log odds.

Key takeaway: The Bernoulli mass function is \(\exp\{y\theta-b(\theta)\}\), with \(\theta=\log\{p/(1-p)\}\) and \(b(\theta)=\log(1+e^\theta)\). Differentiating \(b\) gives \(E[Y]=p\) and \(\operatorname{Var}(Y)=p(1-p)\).

Check Your Understanding

For each problem, identify the Bernoulli exponential-family quantities and check the probability or moment requested.

A hypothetical binary service indicator has \(p=0.3\). Find \(\theta\), \(b(\theta)\), the mean, and the variance. (Answer: \(\theta=\log(3/7)\approx-0.8473\), \(b(\theta)=\log(10/7)\approx0.3567\), mean \(0.3\), variance \(0.21\).)
A Bernoulli variable has natural parameter \(\theta=\log 5\). Recover \(p\), find \(b(\theta)\), and find the variance. (Answer: \(p=5/6\), \(b(\theta)=\log 6\), variance \(5/36\approx0.1389\).)
Four independent, identically distributed Bernoulli indicators have natural parameter \(\theta=-\log 4\), and their observed values are \(0,1,0,0\). Find \(p\), \(b(\theta)\), and the sequence likelihood. (Answer: \(p=1/5\), \(b(\theta)=\log(5/4)\), likelihood \((1/5)(4/5)^3=64/625=0.1024\).)
A Bernoulli response has \(p=0.65\). Write the log likelihood contribution for an observed \(y=1\) using \(\theta\) and \(b(\theta)\), then give its numerical probability. (Answer: \(\theta=\log(13/7)\), \(b(\theta)=\log(20/7)\), log likelihood \(\theta-b(\theta)\), probability \(0.65\).)
A Bernoulli model has \(e^\theta=4/3\). Find \(p\), the cumulant, the mean, the variance, and the probability of observing \(y=0\). (Answer: \(p=4/7\), \(b(\theta)=\log(7/3)\), mean \(4/7\), variance \(12/49\), and \(P(Y=0)=3/7\).)

What You'll Learn