The Exponential-Family Probability Form

Why Write a Distribution in Exponential-Family Form?

The same probability model can be written in several algebraically equivalent ways. Exponential-family form makes the way a parameter enters the model explicit. In generalized linear models, this representation is useful because it separates the part involving the observed response from the part involving the model parameter. The task here is to identify that structure—not to infer a link function or derive moments.

The form applies to a probability mass function for a discrete response or a probability density function for a continuous response. The support of the response must be specified, and in the basic representation below it does not vary with the parameter. A function that is part of the base measure must also be free of the parameter being represented.

Definition: A one-parameter exponential-family model can be written as $$ f(y;\theta)=h(y)\exp\left\{\eta(\theta)T(y)-A\bigl(\eta(\theta)\bigr)\right\}. $$ Here, $y$ is an observed response; $T(y)$ is the response-dependent term in the exponent; $\eta(\theta)$ is the natural parameter; $A(\eta)$ is the cumulant function (also called the log-partition function); and $h(y)$ is the base measure. For discrete models, $f$ denotes a probability mass function; for continuous models, it denotes a density. The support and the functions $h$ and $T$ are fixed while $\theta$ varies.

The response-dependent part is not necessarily only $T(y)$: the base measure $h(y)$ can also depend on the observed response. The defining separation is that the parameter enters through the natural parameter and cumulant function, while $h$ and $T$ do not depend on that parameter. In particular, $T(y)$ is the response summary multiplied by the natural parameter in the exponent.

For models with more than one response summary, the corresponding form uses a vector of natural parameters and a vector-valued function $T(y)$. The one-parameter version is enough to practise identifying the components. Different algebraic arrangements may move a parameter-free factor between terms, so always check the complete expression rather than matching isolated symbols.

A Normalization Argument for the Cumulant Function

The cumulant function is not an arbitrary extra term. It is the term that makes the probabilities or densities add or integrate to one. This can be seen directly from the definition.

Start with an unnormalized kernel.
For a fixed natural parameter $\eta$, the response-dependent kernel is $h(y)\exp\{\eta T(y)\}$. It is nonnegative but does not generally sum or integrate to one.

Calculate its total mass.
For a discrete response, the total is $\sum_y h(y)\exp\{\eta T(y)\}$. For a continuous response, it is $\int h(y)\exp\{\eta T(y)\}\,dy$, with the integral over the support.

Divide by that total.
Writing the reciprocal normalizing constant as $\exp\{-A(\eta)\}$ gives a distribution whose total mass is one. Taking logarithms of the total gives $A(\eta)=\log\sum_y h(y)\exp\{\eta T(y)\}$ in the discrete case, or the corresponding log integral in the continuous case.

This argument also gives a practical check: substituting the proposed $A(\eta)$ should make the mass or density normalize correctly. It is not enough for the exponent to contain a parameter times the response; the normalizing term must also be correct.

Worked Examples: Identifying Each Component

Worked Example: A Hypothetical Bernoulli Claim Indicator

A hypothetical one-year insurance contract has a claim indicator $Y$, where $Y=1$ means at least one claim and $Y=0$ means none. Assume $Y\sim\operatorname{Bernoulli}(p)$, with $p=0.3$, and fixed support $\{0,1\}$. Write its mass function in exponential-family form, identify all four components, and check the probabilities for both outcomes.

Method. Start from the Bernoulli mass function and take logarithms of its factors:

f(y;p)=p^y(1-p)^{1-y} =\exp\left\{y\log\left(\frac{p}{1-p}\right)+\log(1-p)\right\}.

To match $\exp\{\eta T(y)-A(\eta)\}$, set $T(y)=y$ and $\eta=\log\{p/(1-p)\}$. Since $p=e^\eta/(1+e^\eta)$, the remaining term is $-A(\eta)=\log(1-p)=-\log(1+e^\eta)$. Thus $A(\eta)=\log(1+e^\eta)$, and $h(y)=1$.

For $p=0.3$, $\eta=\log(0.3/0.7)=\log(3/7)$, and $A(\eta)=\log(1+3/7)=\log(10/7)$. For $y=1$, the exponential-family expression is

1\cdot\exp\{\log(3/7)-\log(10/7)\} =\exp\{\log(3/10)\}=0.3.

For $y=0$, it is

1\cdot\exp\{0-\log(10/7)\}=\frac{7}{10}=0.7.

The probabilities sum to $0.3+0.7=1.0$, as required. The common mistake is to use $\log(p)$ as the natural parameter while retaining $A(\eta)=\log(1+e^\eta)$. That cumulant belongs with the log-odds $\eta=\log\{p/(1-p)\}$, not with $\log(p)$.

Worked Example: A Hypothetical Poisson Event Count

A hypothetical monitoring system records the number $Y$ of alerts during a fixed observation period. Assume $Y\sim\operatorname{Poisson}(\lambda)$, where $\lambda=2$, with support $y=0,1,2,\ldots$. Express the mass function in exponential-family form, identify the components, and evaluate the probability of exactly three alerts.

Method. Begin with $f(y;\lambda)=e^{-\lambda}\lambda^y/y!$. Use $\lambda^y=\exp\{y\log\lambda\}$, giving

f(y;\lambda)=\frac{1}{y!}\exp\{y\log\lambda-\lambda\}.

Therefore $T(y)=y$, $\eta=\log\lambda$, $A(\eta)=e^\eta$, and $h(y)=1/y!$. The support is nonnegative integer counts and does not depend on $\lambda$.

At $\lambda=2$, $\eta=\log 2$ and $A(\eta)=e^{\log 2}=2$. For $y=3$, the base measure is $h(3)=1/3!=1/6$, and the exponential factor is

\exp\{3\log 2-2\}=\exp\{\log 8-2\}=8e^{-2}.

Multiplying gives $f(3;2)=(1/6)(8e^{-2})=(4/3)e^{-2}$. Since $e^{-2}\approx0.1353353$, the probability is $0.1804$ to four decimal places. As a check, the original formula gives $e^{-2}2^3/3!= (4/3)e^{-2}$, the same result. Omitting $1/y!$ would give an incorrect probability: that factor is response-dependent and belongs in the base measure.

Worked Example: A Hypothetical Normal Measurement

Suppose a hypothetical instrument produces a measurement $Y\sim N(\mu,\sigma^2)$, with $\mu=4$ and known, fixed variance $\sigma^2=9$. For an observation $y=7$, write the density in exponential-family form, identify the components, and evaluate the density at $7$. The support is the real line.

Method. The normal density is

f(y;\mu)=\frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{(y-\mu)^2}{2\sigma^2}\right\}.

With $\sigma^2$ fixed, expand the square in the exponent:

-\frac{(y-\mu)^2}{2\sigma^2} =-\frac{y^2}{2\sigma^2}+\frac{\mu y}{\sigma^2}-\frac{\mu^2}{2\sigma^2}.

Thus $T(y)=y$, $\eta=\mu/\sigma^2$, $A(\eta)=\sigma^2\eta^2/2$, and $h(y)=(2\pi\sigma^2)^{-1/2}\exp\{-y^2/(2\sigma^2)\}$. The fixed variance is part of the base measure and cumulant expression; the natural parameter here represents the varying mean.

For the stated values, $\eta=4/9$, $A(\eta)=9(4/9)^2/2=8/9$, and

h(7)=\frac{1}{\sqrt{18\pi}}\exp\left\{-\frac{49}{18}\right\}.

The exponential-family density is $(18\pi)^{-1/2}\exp\{-49/18+7(4/9)-8/9\}$. The exponent simplifies to $-49/18+14/9-8/9=-49/18+12/18=-37/18$.

However, this simplification reveals a calculation error if compared to the original density: the linear term is $\mu y/\sigma^2=4(7)/9=28/9$, not $14/9$. Using the correct term, the exponent is $-49/18+28/9-8/9=-49/18+40/18=-9/18=-1/2$. Hence

f(7;4)=\frac{1}{\sqrt{18\pi}}e^{-1/2}\approx 0.0807.

The direct normal formula has $y-\mu=3$, so its exponent is $-3^2/(2\cdot9)=-1/2$, confirming the result. The displayed correction is a useful trap to watch for: the cross-product in $(y-\mu)^2$ is $2\mu y$, so after division by $2\sigma^2$ the linear term is $\mu y/\sigma^2$.

Common Checks and Exam Traps

When identifying a form, proceed in a fixed order: isolate the exponential, find the coefficient of the response summary, identify the remaining parameter-only term, and then collect parameter-free response factors into $h(y)$. Finally, verify the result against the original mass function or density.

Confusing the natural parameter with the original parameter. The natural parameter is the coefficient of $T(y)$ in the exponent. It may be a transformation of the original parameter, as $\log\lambda$ is for the Poisson model.
Dropping a response-dependent factor. A factorial or a response-only quadratic term may belong to $h(y)$; it cannot be discarded merely because it does not contain the parameter.
Using the wrong cumulant function. $A(\eta)$ must normalize the model in the chosen parameterization. If the natural parameter is changed, the matching cumulant term must be changed too.
Ignoring conditions. In the normal example, the variance was fixed. If additional parameters vary, the proposed one-parameter identification does not describe their full parameter dependence.
Changing the support unnoticed. The basic form above assumes fixed support. If the set of possible responses depends on the parameter, the representation needs additional care and cannot be accepted just by matching the exponent.

Key takeaway: A valid identification names $T(y)$, $\eta$, $A(\eta)$, and $h(y)$, states the parameter and support conditions, and reconstructs the original probability or density. The normalizing role of $A$ is the decisive check.

Check Your Understanding

For each problem, identify the exponential-family components and check the requested quantity using the stated model.

A hypothetical sensor records a binary fault indicator $Y\sim\operatorname{Bernoulli}(0.2)$. Write the natural parameter and cumulant function, then compute the mass at $y=0$. (Answer: $\eta=\log(1/4)$, $A(\eta)=\log(5/4)$, and $P(Y=0)=0.8000$.)
A hypothetical queue has count $Y\sim\operatorname{Poisson}(3)$. Give $h(y)$, $T(y)$, $\eta$, and $A(\eta)$, then find $P(Y=2)$. (Answer: $h(y)=1/y!$, $T(y)=y$, $\eta=\log 3$, $A(\eta)=3$, and $P(Y=2)=\frac{9}{2}e^{-3}\approx0.2240$.)
A hypothetical continuous measurement has $Y\sim N(2,4)$, where the variance $4$ is fixed. Identify $\eta$, $A(\eta)$, and $h(y)$. What exponent remains after multiplying $h(3)$ by the exponential-family factor? (Answer: $\eta=1/2$, $A(\eta)=1/2$, $h(y)=(8\pi)^{-1/2}e^{-y^2/8}$, and the combined exponent at $y=3$ is $-1/8$.)
A model is written as $f(y;\lambda)=\exp\{y\log\lambda-\lambda-\log(y!)\}$ on nonnegative integer support. Identify all components and verify the probability at $y=0$ when $\lambda=1$. (Answer: $T(y)=y$, $\eta=\log\lambda$, $A(\eta)=e^\eta=\lambda$, $h(y)=1/y!$, and $f(0;1)=e^{-1}\approx0.3679$.)
A hypothetical binary outcome has $P(Y=1)=0.6$. A candidate proposes $\eta=\log(0.6)$, $T(y)=y$, $A(\eta)=\log(1+e^\eta)$, and $h(y)=1$. Check the proposed mass at $y=1$, then give the correct natural parameter and cumulant. (Answer: the proposal gives $0.6/(1.6)=0.375$, so it is incorrect; $\eta=\log(0.6/0.4)=\log(3/2)$ and $A(\eta)=\log(5/2)$.)

What You'll Learn