Canonical Exponential-Family Form

Why Canonical Form Matters

Probability models are often introduced with different-looking formulas. The exponential density includes a rate in front of it, a normal density has a quadratic exponent, and a geometric mass function uses powers of a success probability. Canonical exponential-family form makes a shared structure visible: the observation enters through a statistic, the parameter enters through a natural parameter, and a log-partition function ensures the probabilities sum or integrate to one.

The aim here is to recognize and construct that representation. It is a useful language for likelihood theory, but identifying a canonical form does not by itself establish that a representation is unique, that a maximum likelihood estimate exists, or that a model has any particular regularity property.

Definition: Relative to a measure $\nu$ on the sample space, a family has exponential-family form if its density or mass function can be written $$ f(x;\theta)=h(x)\exp\!\left\{\eta(\theta)^{T}T(x)-A\bigl(\eta(\theta)\bigr)\right\}. $$ Here $h(x)$ is the carrier term, $T(x)$ is the (possibly vector-valued) statistic, $\eta(\theta)$ is the natural parameter, and $A$ is the log-partition function. The superscript $T$ denotes transpose, so $\eta(\theta)^T T(x)$ is an inner product. For a scalar statistic this product is ordinary multiplication.

The density or mass function is understood relative to a specified measure: commonly Lebesgue measure for a continuous observation and counting measure for a discrete one. The carrier can include an indicator of the support, and it must not depend on the model parameter. The natural parameter is a function of the original parameter $\theta$; it need not equal $\theta$. In a particular model, only natural-parameter values for which the expression is a probability distribution are allowed.

The Normalizing Role of the Log-Partition Function

For a representation to define a probability distribution, its density or mass function must integrate or sum to one. This requirement determines the log-partition function. Suppose $h(x)\geq 0$, and let $T(x)$ take values in $\mathbb{R}^d$. For any natural parameter $\eta$ for which the integral below is finite and positive, define

Z(\eta)=\int h(x)\exp\{\eta^T T(x)\}\,d\nu(x), \qquad A(\eta)=\log Z(\eta).

The integral notation includes a sum when $\nu$ is counting measure. The normalization claim follows directly:

Result: If $0<Z(\eta)<\infty$, then $$ f(x;\eta)=h(x)\exp\{\eta^T T(x)-A(\eta)\} $$ integrates to one with respect to $\nu$.

Indeed, because $A(\eta)=\log Z(\eta)$, we have $\exp\{-A(\eta)\}=1/Z(\eta)$. Therefore

\int h(x)\exp\{\eta^T T(x)-A(\eta)\}\,d\nu(x) = \frac{1}{Z(\eta)} \int h(x)\exp\{\eta^T T(x)\}\,d\nu(x) =1.

Conversely, if $h$ and $T$ are fixed and a density of this form is normalized, then its normalizing term must satisfy $\exp\{A(\eta)\}=Z(\eta)$, and hence $A(\eta)=\log Z(\eta)$. This identifies the role of $A$; it does not imply that every model has only one possible choice of carrier and statistic.

Worked Canonical Representations

Worked Example: Exponential Distribution in Rate Form

Let $X$ have an exponential distribution with rate $\lambda>0$, so its density is $f(x;\lambda)=\lambda e^{-\lambda x}$ for $x\geq 0$, and zero otherwise. Write it in canonical form.

First display the support explicitly: $$ f(x;\lambda)=\mathbf{1}_{\{x\geq 0\}}\lambda e^{-\lambda x}. $$ The parameter-dependent factor in the exponent is $-\lambda x$. Thus take the statistic $T(x)=x$ and the natural parameter $\eta=-\lambda$. Since $\lambda>0$, this parameterization has $\eta<0$.

The remaining factors identify the carrier and log-partition function: $$ h(x)=\mathbf{1}_{\{x\geq 0\}}, \qquad A(\eta)=-\log(-\eta). $$ To check, $\exp\{-A(\eta)\}=-\eta=\lambda$, so $$ h(x)\exp\{\eta T(x)-A(\eta)\} = \mathbf{1}_{\{x\geq 0\}}e^{-\lambda x}\lambda, $$ which is the given density. Equivalently, normalization gives $$ Z(\eta)=\int_0^\infty e^{\eta x}\,dx=-\frac{1}{\eta} \quad(\eta<0), \qquad A(\eta)=\log Z(\eta)=-\log(-\eta). $$

Worked Example: Geometric Distribution on Nonnegative Integers

Suppose $X$ counts the failures before the first success, with success probability $p$, where $0<p<1$. Its mass function is $P(X=x)=p(1-p)^x$ for $x=0,1,2,\ldots$. Identify its canonical components.

Write the power of $1-p$ as an exponential: $$ p(1-p)^x=p\exp\{x\log(1-p)\}. $$ Choose $T(x)=x$ and $\eta=\log(1-p)$, which is negative for $0<p<1$. Solving for $p$ gives $p=1-e^\eta$. Thus the carrier is $h(x)=1$ on the nonnegative integers, and the remaining parameter factor is $$ \exp\{-A(\eta)\}=p=1-e^\eta, \qquad A(\eta)=-\log(1-e^\eta). $$ The full representation is therefore $$ f(x;\eta)=\mathbf{1}_{\{x=0,1,2,\ldots\}} \exp\!\left\{\eta x+\log(1-e^\eta)\right\}. $$ Equivalently, the indicator can be included in the carrier. To verify the normalizer, use the geometric series for $\eta<0$: $$ Z(\eta)=\sum_{x=0}^{\infty}e^{\eta x} =\frac{1}{1-e^\eta}, \qquad \log Z(\eta)=-\log(1-e^\eta)=A(\eta). $$

Worked Example: Normal Distribution with Known Variance

Let $X\sim N(\mu,\sigma^2)$, with known $\sigma^2>0$ and unknown mean $\mu$. Expand the square in the density and read off the canonical form.

The density is $$ f(x;\mu)=\frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left\{-\frac{(x-\mu)^2}{2\sigma^2}\right\}. $$ Expanding the quadratic gives $$ -\frac{(x-\mu)^2}{2\sigma^2} = -\frac{x^2}{2\sigma^2} +\frac{\mu x}{\sigma^2} -\frac{\mu^2}{2\sigma^2}. $$ The coefficient of $x$ is the natural parameter, so set $\eta=\mu/\sigma^2$ and $T(x)=x$. Since $\mu=\sigma^2\eta$, the last term is $-\sigma^2\eta^2/2$. Consequently, $$ h(x)=\frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left\{-\frac{x^2}{2\sigma^2}\right\}, \qquad A(\eta)=\frac{\sigma^2\eta^2}{2}. $$ Substitution yields $h(x)\exp\{\eta x-A(\eta)\}$, which reproduces every term of the original density. The carrier depends on the known variance, but not on the unknown mean. Here the natural parameter differs from the original parameter by the factor $1/\sigma^2$.

Independent Observations Preserve the Form

A useful consequence of canonical form is that products of independent densities or mass functions remain in exponential-family form. This is an algebraic fact about the likelihood, not an assertion about the existence of an estimator.

Result: Suppose $X_1,\ldots,X_n$ are independent and each has density or mass function $$ f(x_i;\eta)=h(x_i)\exp\{\eta^T T(x_i)-A(\eta)\}. $$ Then their joint density or mass function is $$ f(x_1,\ldots,x_n;\eta) = \left\{\prod_{i=1}^n h(x_i)\right\} \exp\!\left\{ \eta^T\sum_{i=1}^n T(x_i)-nA(\eta) \right\}. $$

To prove the result, independence gives the joint density as the product of the individual densities. Substituting the assumed form and using the rules for products and exponentials,

\begin{aligned} \prod_{i=1}^n f(x_i;\eta) &= \prod_{i=1}^n \left[ h(x_i)\exp\{\eta^T T(x_i)-A(\eta)\} \right]\\ &= \left\{\prod_{i=1}^n h(x_i)\right\} \exp\!\left\{ \sum_{i=1}^n \eta^T T(x_i)-nA(\eta) \right\}\\ &= \left\{\prod_{i=1}^n h(x_i)\right\} \exp\!\left\{ \eta^T\sum_{i=1}^n T(x_i)-nA(\eta) \right\}. \end{aligned}

The statistic appearing in the joint exponential term is the sum $\sum_{i=1}^n T(X_i)$. The argument identifies the relevant data-dependent term in the likelihood; it does not require a separate theory of how to establish sufficiency.

Worked Example: Combining Exponential Observations

Let $X_1,\ldots,X_n$ be independent exponential observations with common rate $\lambda$. From the single-observation representation, $h(x_i)=\mathbf{1}_{\{x_i\geq 0\}}$, $T(x_i)=x_i$, $\eta=-\lambda$, and $A(\eta)=-\log(-\eta)$. The joint density is $$ \left\{\prod_{i=1}^n\mathbf{1}_{\{x_i\geq 0\}}\right\} \exp\!\left\{ -\lambda\sum_{i=1}^n x_i-n\log\lambda \right\}. $$ Indeed, $A(-\lambda)=-\log\lambda$, so the term $-nA(-\lambda)$ equals $n\log\lambda$, as required. The same joint density written in the original rate parameter is $$ \lambda^n\exp\!\left\{-\lambda\sum_{i=1}^n x_i\right\} \prod_{i=1}^n\mathbf{1}_{\{x_i\geq 0\}}. $$ The two expressions agree because $\exp\{n\log\lambda\}=\lambda^n$. Independence adds the individual statistics and multiplies the carriers; it does not change the underlying single-observation components.

Recognizing the Components Reliably

A practical way to rewrite a model is to expand its exponent and separate terms by what they depend on. Terms involving only the observation belong in the carrier. Terms that multiply a data-dependent statistic identify the natural parameter. The remaining parameter-only term is matched to $-A(\eta)$. Any support restriction must also be represented, usually through an indicator in the carrier.

Specify the measure and support.
Decide whether the model is a density or a mass function, and make any parameter-independent support indicator explicit.

Collect the data-dependent terms.
Rewrite the exponent so the data appear through one or more statistics $T(x)$.

Read off the natural parameter.
The coefficients of those statistics form $\eta(\theta)$. For vector statistics, collect the coefficients into a vector using the inner product $\eta(\theta)^T T(x)$.

Identify and check the remaining terms.
Put data-only factors in $h(x)$, assign the remaining parameter-only factor to $\exp\{-A(\eta)\}$, and verify normalization.

The check is essential: matching symbols by inspection is not enough if the resulting expression fails to integrate or sum to one. Also, the allocation is not always unique. For example, a parameter-independent multiplicative constant can sometimes be moved between the carrier and the normalizing term. A claimed representation should therefore be verified as an equality of densities or mass functions, rather than treated as the only possible decomposition.

Common Mistake: Confusing the Original and Natural Parameters

A frequent error is to copy the original parameter directly into the exponent as the natural parameter. In the exponential example, writing $e^{\lambda x}$ with natural parameter $\lambda$ would give the wrong sign: the density decays as $x$ increases, so its exponent contains $-\lambda x$, not $+\lambda x$. The correct natural parameter is $\eta=-\lambda$, and the corresponding log-partition function is $A(\eta)=-\log(-\eta)$.

A related error is to call every parameter-dependent factor a carrier. By definition, $h(x)$ cannot depend on the parameter. In the exponential density, $\lambda$ belongs to the natural parameter and log-partition terms; it cannot be placed in $h(x)$. In the normal example with known variance, the factor involving $x^2$ can be part of the carrier because the variance is fixed, while the unknown mean must enter through the canonical exponent.

Key takeaway: Canonical form separates a model into a parameter-independent carrier, a statistic of the observation, its natural parameter, and a log-partition function. Expand, collect, and then verify the representation by normalization.

Check Your Understanding

For each question, identify the relevant components or explain the reasoning in words.

For the exponential model with rate $\lambda$, what are $T(x)$ and $\eta$, and why must $\eta$ be negative?
In the geometric example, verify from $p=1-e^\eta$ that the parameter-only factor is $\exp\{-A(\eta)\}$.
For a normal model with known variance, which part of the expanded exponent identifies the natural parameter?
Why must an indicator for the support be included in the carrier when the support does not depend on the parameter?
For independent observations from one canonical family, what statistic appears in the joint exponential term?