Bernoulli Responses in Canonical Form

Why Write a Bernoulli Response in Canonical Form?

A binary response records one of two outcomes, conventionally coded \(Y=0\) or \(Y=1\). Its probability model is simple, but putting it in exponential-family form makes explicit how the outcome enters the likelihood and which function normalizes the probabilities. This representation is also the starting point for later GLM work with binary responses; here the focus is the probability model for one response, not a regression model.

Let \(Y\) have a Bernoulli distribution with success probability \(p\), where \(0<p<1\). Its mass is \(P(Y=1)=p\) and \(P(Y=0)=1-p\). The canonical parameter will be the log-odds, denoted by \(\eta\). The strict inequalities on \(p\) matter: they ensure that \(\eta\) is finite.

Deriving the Canonical Representation

For \(y\in\{0,1\}\), the Bernoulli mass function can be written in a single expression:

Take logarithms of the two factors, then collect the terms that depend on \(y\):

By definition, \(\eta=\log\!\left(\frac{p}{1-p}\right)\). Solving this equation for \(p\) gives \(p=e^\eta/(1+e^\eta)\), and therefore \(1-p=1/(1+e^\eta)\). Hence \(\log(1-p)=-\log(1+e^\eta)\). Substitution yields the canonical form:

This is the one-parameter exponential-family expression \(\exp\{T(y)\eta-A(\eta)\}\) on the Bernoulli support. Thus \(T(y)=y\) and \(A(\eta)=\log(1+e^\eta)\). There is no additional parameter-dependent dispersion factor in this single Bernoulli form.

In a discrete model, it is useful to keep two ideas separate. The counting measure specifies how probabilities are summed over possible outcomes. The carrier term records the support in the density expression relative to that measure. Here the carrier equals one at both allowed outcomes, so it does not change either mass; it makes explicit that the response can only be zero or one.

Normalization and the Natural Parameter

A log-partition function is the term that makes the exponential expression sum to one. For this Bernoulli model, normalization follows directly from the two possible outcomes.

Proof. Since \(e^\eta>0\), the denominator \(1+e^\eta\) is positive and \(p=e^\eta/(1+e^\eta)\) is strictly between zero and one. Evaluating the canonical expression at both possible outcomes gives

Each term is positive, so these are valid masses. The first is \(1-p\), and the second is \(p\), as claimed. \(\square\)

The finite natural-parameter space is therefore all of \(\mathbb{R}\): \(A(\eta)=\log(1+e^\eta)\) is finite for every real \(\eta\). Conversely, every \(p\in(0,1)\) gives a finite \(\eta\). The endpoints \(p=0\) and \(p=1\) correspond only to limits \(\eta\to-\infty\) and \(\eta\to+\infty\), not to finite values in this parameterization.

What the Log-Partition Function Encodes

For the Bernoulli model, \(A(\eta)\) does more than normalize the masses: its first two derivatives give the response mean and variance. These identities can be verified directly without appealing to a general result about exponential families.

Proof. Differentiate \(A(\eta)=\log(1+e^\eta)\):

Because \(Y\) is one for a success and zero otherwise, \(E[Y]=1\cdot p+0\cdot(1-p)=p\). Differentiating once more gives

Also \(Y^2=Y\) for both possible values, so \(E[Y^2]=p\). Therefore

This proves both identities for the Bernoulli response. \(\square\)

The variance is positive for every finite \(\eta\), since then \(0<p<1\). It approaches zero as \(p\) approaches either endpoint. These are properties of this particular response distribution; the calculation above does not require a general theorem about derivatives of log-partition functions.

Worked Representations

Common Mistakes and Interpretation

• Using \(p\) as the canonical parameter. The canonical parameter here is \(\eta=\log(p/(1-p))\), not \(p\). The probability is recovered from \(\eta\) by \(p=e^\eta/(1+e^\eta)\).
• Dropping the log-partition term. Writing \(\exp(y\eta)\) alone does not give probabilities that sum to one. The subtraction of \(A(\eta)\) supplies the required normalization.
• Putting the sign on the wrong term. The canonical exponent is \(y\eta-A(\eta)\), with \(A(\eta)=\log(1+e^\eta)\). For \(y=0\), this yields \(1/(1+e^\eta)\); for \(y=1\), it yields \(e^\eta/(1+e^\eta)\).
• Treating the base measure as the log-partition function. The base measure here is counting measure on the two-point sample space; the carrier is one on the support. The log-partition function is the parameter-dependent term \(A(\eta)\).
• Assuming the endpoints have finite log-odds. If \(p=0\) or \(p=1\), the logit formula is not finite. The regular canonical representation uses \(0<p<1\) and finite \(\eta\).

The canonical form separates the response-dependent statistic \(y\eta\) from the parameter-dependent normalizer \(A(\eta)\). For a single binary response, that separation is especially transparent because the statistic is simply the observed value itself. It also makes checking a proposed expression straightforward: evaluate it at zero and one, verify that the masses are \(1-p\) and \(p\), and confirm that they sum to one.