Exponential-Family Form For GLM Responses

Why the Exponential-Family Form Matters

A generalized linear model combines a response distribution with a structure for how its mean depends on predictors. The earlier material on the GLM mean structure introduced the linear predictor; here the focus is the response distribution’s mathematical form. Writing a distribution in exponential-family form makes its important ingredients visible and provides a common language for Bernoulli, binomial, Poisson, and normal responses.

The form also helps distinguish quantities that can look similar but play different roles. The natural parameter describes the response distribution on its exponential scale; the cumulant function normalizes that distribution; the dispersion term scales the response-dependent part; and a final term accounts for the observation and any fixed distributional details.

Definition: A response distribution is written in the common exponential-family form if its probability mass function or probability density can be expressed as

f(y;\theta,\phi) = \exp\left\{ \frac{y\theta-b(\theta)}{a(\phi)} +c(y,\phi) \right\}.

Here \(y\) is the observed response, \(\theta\) is the natural parameter, \(b(\theta)\) is the cumulant function (also called the log-partition function), and \(a(\phi)\) is the dispersion scaling term. The term \(c(y,\phi)\) contains the remaining dependence on the observed response and dispersion. The functions and parameter conventions must be stated for the particular distribution; they are not interchangeable across families.

For a discrete response, \(f\) is a probability mass function. For a continuous response, \(f\) is a density, so its value at a point is not the probability of observing exactly that point. Some response families have a fixed dispersion convention rather than a freely estimated \(\phi\). In those cases, writing \(a(\phi)=1\) records that convention.

Deriving the Bernoulli Representation

Suppose \(Y\) is a Bernoulli response, where \(Y=1\) denotes success and \(Y=0\) denotes failure, and \(P(Y=1)=p\), with \(0<p<1\). Its probability mass function is \(p^y(1-p)^{1-y}\). Taking logarithms and collecting the terms involving \(y\) gives the exponential-family representation.

\begin{aligned} f(y;p) &=p^y(1-p)^{1-y}\\ &=\exp\{y\log p+(1-y)\log(1-p)\}\\ &=\exp\left\{ y\log\left(\frac{p}{1-p}\right)+\log(1-p) \right\}. \end{aligned}

Set \(\theta=\log\{p/(1-p)\}\). Since \(p=e^\theta/(1+e^\theta)\), we have \(\log(1-p)=-\log(1+e^\theta)\). Therefore the response-dependent expression is \(y\theta-\log(1+e^\theta)\). Comparing it with the general form identifies the components.

Bernoulli representation: For \(Y\sim\operatorname{Bernoulli}(p)\), take \(\theta=\log\{p/(1-p)\}\), \(b(\theta)=\log(1+e^\theta)\), \(a(\phi)=1\), and \(c(y,\phi)=0\). The dispersion is fixed at one.

This derivation also explains why the cumulant function is needed: its contribution makes the two possible outcomes’ probabilities sum to one. It does not mean that \(e^{\theta-b(\theta)}\) is the probability of failure; for \(y=1\), it is the success probability.

Worked Example: Reading a Bernoulli Probability From the Form

A hypothetical policy screening outcome is recorded as \(Y=1\) for a flagged application and \(Y=0\) otherwise. Assume \(Y\sim\operatorname{Bernoulli}(p)\) with \(p=0.25\). Identify the exponential-family terms and calculate both outcome probabilities from the representation. State the method and a common mistake it avoids.

Method. Use the Bernoulli representation, with the fixed-dispersion convention \(a(\phi)=1\). The natural parameter is \(\theta=\log\{p/(1-p)\}=\log(0.25/0.75)=\log(1/3)\). The cumulant function is \(b(\theta)=\log(1+e^\theta)=\log(1+1/3)=\log(4/3)\), and \(c(y,\phi)=0\).

For a flagged outcome, \(y=1\), so \(f(1)=\exp\{\theta-b(\theta)\}\). Substitution gives \(\exp\{\log(1/3)-\log(4/3)\} =\exp\{\log(1/4)\}=0.25\).

For an unflagged outcome, \(y=0\), so \(f(0)=\exp\{-b(\theta)\} =\exp\{-\log(4/3)\}=3/4=0.75\). The check is \(f(1)+f(0)=0.25+0.75=1\), as required.

Common mistake avoided: Using \(\exp\{\theta-b(\theta)\}=0.75\) assigns the failure probability to \(y=1\). The calculation gives \(0.25\); the complementary probability \(0.75\) comes from \(y=0\).

Binomial Responses: The Trial Count Matters

Suppose \(Y\) counts successes in a fixed \(m\) independent Bernoulli trials, each with the same success probability \(p\). Then \(Y\sim\operatorname{Binomial}(m,p)\), where \(y\) can be any integer from \(0\) to \(m\). Its mass function includes a combinatorial factor because there are \({m\choose y}\) ways to place \(y\) successes among the trials.

\begin{aligned} f(y;p) &={m\choose y}p^y(1-p)^{m-y}\\ &=\exp\left\{ y\log\left(\frac{p}{1-p}\right) +m\log(1-p)+\log{m\choose y} \right\}. \end{aligned}

With \(\theta=\log\{p/(1-p)\}\), the identity \(\log(1-p)=-\log(1+e^\theta)\) gives \(m\log(1-p)=-m\log(1+e^\theta)\). Thus the binomial count has \(b(\theta)=m\log(1+e^\theta)\), \(a(\phi)=1\), and \(c(y,\phi)=\log{m\choose y}\). Under this convention the dispersion is fixed at one. In particular, the number of trials \(m\) appears in the cumulant function; it is not omitted just because the natural parameter is the same log odds used for Bernoulli responses.

Worked Example: A Grouped Binomial Likelihood

In a hypothetical audit, 2 of 5 independently sampled records meet a specified criterion. Assume a binomial model with fixed \(m=5\) and \(p=0.4\). Write its exponential-family terms and calculate the probability of exactly 2 qualifying records. State the method and the common mistake it avoids.

Method. Use the grouped binomial count representation. The natural parameter is \(\theta=\log(0.4/0.6)=\log(2/3)\). The cumulant function is \(b(\theta)=5\log(1+e^\theta)=5\log(5/3)\). The remaining data term is \(c(2,\phi)=\log{5\choose2}=\log 10\), and \(a(\phi)=1\).

Substitute \(y=2\) into the log mass: \(y\theta-b(\theta)+c(y,\phi) =2\log(2/3)-5\log(5/3)+\log 10\). Exponentiating gives \({5\choose2}(2/3)^2(3/5)^5\). Equivalently, using the original probability, \({5\choose2}(0.4)^2(0.6)^3 =10(0.16)(0.216)=0.3456\).

The result is a probability mass of \(0.3456\), which is between zero and one. Common mistake avoided: Using \(b(\theta)=\log(1+e^\theta)\) without the factor \(m=5\) would describe a single Bernoulli trial’s normalizing term, not the distribution of the count from five trials.

Poisson Responses

Suppose \(Y\) is a count with \(Y\sim\operatorname{Poisson}(\lambda)\), where \(\lambda>0\). Its mass function is \(e^{-\lambda}\lambda^y/y!\) for nonnegative integer \(y\). Setting \(\theta=\log\lambda\) gives \(\lambda=e^\theta\), so the log mass is \(y\theta-e^\theta-\log(y!)\).

Poisson representation: For \(Y\sim\operatorname{Poisson}(\lambda)\), take \(\theta=\log\lambda\), \(b(\theta)=e^\theta\), \(a(\phi)=1\), and \(c(y,\phi)=-\log(y!)\). The dispersion is fixed at one.

Worked Example: Expressing a Poisson Count

A hypothetical service desk records 2 requests in a fixed observation interval. Assume the count has a Poisson distribution with parameter \(\lambda=3\). Identify the exponential-family components and calculate the probability of the observed count. State the method and the mistake it avoids.

Method. Use the Poisson representation. The natural parameter is \(\theta=\log 3\), the cumulant function is \(b(\theta)=e^\theta=3\), the dispersion scaling is \(a(\phi)=1\), and \(c(2,\phi)=-\log(2!)=-\log 2\).

The log mass at \(y=2\) is \(2\log 3-3-\log 2\). Exponentiating, \(f(2)=e^{2\log3-3-\log2}=9/(2e^3)\approx0.2240\), to four decimal places. This is the probability mass for exactly 2 requests under the stated model.

Common mistake avoided: Omitting \(-\log(y!)\) drops the factorial term from the Poisson mass function. For \(y=2\), that would overstate the probability by a factor of \(2\).

Normal Responses

Let \(Y\sim N(\mu,\sigma^2)\), where \(\sigma^2>0\) is the variance, and treat \(\sigma^2\) as the dispersion parameter \(\phi\). The normal density can be rewritten by expanding the squared difference: \((y-\mu)^2=y^2-2y\mu+\mu^2\). This puts the terms involving \(y\) and \(\mu\) into the required form.

\begin{aligned} f(y;\mu,\sigma^2) &=\frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{(y-\mu)^2}{2\sigma^2}\right\}\\ &=\exp\left\{ \frac{y\mu-\mu^2/2}{\sigma^2} -\frac{y^2}{2\sigma^2} -\frac{1}{2}\log(2\pi\sigma^2) \right\}. \end{aligned}

Normal representation: For \(Y\sim N(\mu,\sigma^2)\), take \(\theta=\mu\), \(b(\theta)=\theta^2/2\), \(a(\phi)=\phi=\sigma^2\), and \(c(y,\phi)=-y^2/(2\phi)-\tfrac12\log(2\pi\phi)\), with \(\phi>0\).

Unlike the three fixed-dispersion representations above, this normal form has a dispersion parameter that is the variance. Since this is a continuous response, evaluating \(f\) at a particular \(y\) gives a density, not a probability at that exact value. A probability requires an interval and the area under the density over that interval.

Worked Example: Evaluating a Normal Density in Exponential Form

A hypothetical measurement has \(Y\sim N(10,4)\), where the second parameter is the variance, so the standard deviation is \(2\). Evaluate the density at \(y=12\) using the exponential-family form. State the method and the common mistake it avoids.

Method. Use the normal representation with \(\theta=\mu=10\), \(b(\theta)=10^2/2=50\), and \(\phi=\sigma^2=4\). The data term at \(y=12\) is \(c(12,4)=-12^2/(2\cdot4)-\tfrac12\log(2\pi\cdot4) =-18-\tfrac12\log(8\pi)\).

The exponential-family exponent is \((y\theta-b(\theta))/a(\phi)+c(y,\phi) =(12\cdot10-50)/4-18-\tfrac12\log(8\pi)\). The first term is \(70/4=17.5\), so the full exponent is \(17.5-18-\tfrac12\log(8\pi) =-0.5-\tfrac12\log(8\pi)\). Therefore \(f(12)=e^{-0.5}/\sqrt{8\pi}\approx0.1210\), to four decimal places.

Check. The standard normal density at two units above the mean, with standard deviation \(2\), is \(e^{-2^2/(2\cdot2^2)}/(2\sqrt{2\pi})=e^{-0.5}/\sqrt{8\pi}\), the same value. Common mistake avoided: The given \(4\) is the variance, not the standard deviation; using \(4\) as the standard deviation would produce a different density.

Compare the Four Representations

The table summarizes the conventions used here. For binomial responses, \(m\) is the fixed number of trials and \(y\) is the observed success count. For normal responses, \(\phi\) is the variance. A fixed dispersion of one is a convention for the listed families, not a claim that every possible response model must use that convention.

Response distribution	Natural parameter \(\theta\)	Cumulant function \(b(\theta)\)	Dispersion and data term
Bernoulli\((p)\)	\(\log\{p/(1-p)\}\)	\(\log(1+e^\theta)\)	\(a(\phi)=1;\ c=0\)
Binomial\((m,p)\)	\(\log\{p/(1-p)\}\)	\(m\log(1+e^\theta)\)	\(a(\phi)=1;\ c=\log{m\choose y}\)
Poisson\((\lambda)\)	\(\log\lambda\)	\(e^\theta\)	\(a(\phi)=1;\ c=-\log(y!)\)
Normal\((\mu,\sigma^2)\)	\(\mu\)	\(\theta^2/2\)	\(a(\phi)=\phi=\sigma^2;\ c=-y^2/(2\phi)-\tfrac12\log(2\pi\phi)\)

The representations are not merely a notation exercise. They make it possible to compare response families while keeping clear which part concerns the observed data, which part normalizes the distribution, and which part controls dispersion. This is useful when specifying or interpreting GLMs. It is also important not to infer that all families have the same dispersion behavior: the fixed-one convention for the Bernoulli, binomial, and Poisson forms differs from the normal variance parameter.

Common Traps

Reversing a Bernoulli outcome. For \(y=1\), the mass is \(e^{\theta-b(\theta)}=p\); for \(y=0\), it is \(e^{-b(\theta)}=1-p\).
Dropping the binomial trial count. A binomial count has \(m\log(1+e^\theta)\) in \(b(\theta)\), not just \(\log(1+e^\theta)\).
Omitting a data term. The binomial coefficient and the Poisson factorial are part of the mass function, even though they do not involve the natural parameter.
Confusing a variance and a standard deviation. In \(N(\mu,\sigma^2)\), the second argument is the variance; the normal dispersion \(\phi\) equals \(\sigma^2\).
Calling a density a probability. A normal density value at \(y\) can exceed a probability for a small interval and is not the probability of \(Y=y\).

Key takeaway: To verify an exponential-family representation, substitute the proposed \(\theta\), \(b(\theta)\), \(a(\phi)\), and \(c(y,\phi)\) into the general form and recover the original mass function or density. Check the response coding, trial count, dispersion convention, and any factorial or combinatorial term.

Check Your Understanding

For each problem, identify the requested exponential-family components or probability and check whether the stated response coding and dispersion convention are consistent.

A Bernoulli response has \(p=0.20\). Give \(\theta\) and \(b(\theta)\), then calculate the probability of \(y=0\) using the representation. (Answer: \(\theta=\log(1/4)\), \(b(\theta)=\log(5/4)\), \(P(Y=0)=0.8000\).)
A count is modeled as \(\operatorname{Binomial}(4,0.5)\). Use \(\theta=0\), \(b(\theta)=4\log 2\), and \(c(3)=\log{4\choose3}\) to calculate \(P(Y=3)\). (Answer: \(0.2500\).)
For \(Y\sim\operatorname{Poisson}(2)\), write \(\theta\) and \(b(\theta)\), and calculate \(P(Y=0)\) to four decimal places. (Answer: \(\theta=\log 2\), \(b(\theta)=2\), \(P(Y=0)=e^{-2}=0.1353\).)
A continuous response has \(Y\sim N(6,9)\). Identify \(\theta\), \(b(\theta)\), and \(\phi\), then evaluate the density at \(y=8\) to four decimal places. (Answer: \(\theta=6\), \(b(\theta)=18\), \(\phi=9\), \(f(8)\approx0.1065\).)
A grouped count has \(Y\sim\operatorname{Binomial}(3,0.25)\). Using \(\theta=\log(1/3)\), calculate the probability of exactly one success from the exponential-family representation, including the data term. (Answer: \(b(\theta)=3\log(4/3)\), \(c(1)=\log 3\), \(P(Y=1)=0.4219\).)

What You'll Learn