Canonical Form of an Exponential Family

Why Canonical Form Matters

A density or mass function can hide its structure behind products, powers, and normalizing constants. The canonical exponential-family form makes that structure visible: it separates the part depending only on the observation from the part that pairs a statistic of the observation with a parameter. Reading this form correctly is useful because it gives a common language for comparing distribution models and locating the information about the parameter in the formula.

The key task here is algebraic but requires care. A factor belongs in the carrier only if it does not depend on the parameter; a factor involving the parameter but not the observation belongs in the log-partition term. The same distribution can also be written in different-looking ways, so a proposed representation must be checked by expanding its exponent and verifying its support.

Definition and Components

Fix a reference measure $\mu$ on the sample space, such as Lebesgue measure for a continuous density or counting measure for a discrete mass function. The representation below is relative to that measure and assumes the support can be expressed without changing with the parameter.

Definition: A one-parameter exponential-family representation has density or mass function $$ f_{\eta}(x)=h(x)\exp\{\eta T(x)-A(\eta)\}. $$ A multiparameter representation has the form $$ f_{\eta}(x)=h(x)\exp\{\eta^{\mathsf T}T(x)-A(\eta)\} =h(x)\exp\left\{\sum_{j=1}^{p}\eta_jT_j(x)-A(\eta)\right\}, $$ where $\eta=(\eta_1,\ldots,\eta_p)^{\mathsf T}$ and $T(x)=(T_1(x),\ldots,T_p(x))^{\mathsf T}$. The nonnegative, parameter-free function $h(x)$ is the carrier; $T(x)$ is the canonical statistic; $\eta$ is the natural parameter; and $A(\eta)$ is the log-partition function.

The expression is understood on a fixed support: outside it, the density or mass is zero. In the discrete case, $f_\eta(x)$ denotes a mass function with respect to counting measure. A model may have a parameter $\theta$ that is not itself the natural parameter; then the representation uses a map $\eta=\eta(\theta)$. The canonical statistic is a function of the observation, not of the parameter.

The terms are distinguished by what they depend on. The carrier can contain complicated functions of $x$, provided they are free of the parameter. The exponent’s parameter-observation interaction is written as $\eta^{\mathsf T}T(x)$. A term depending on the parameter alone is absorbed into $-A(\eta)$; a term depending on $x$ alone can be absorbed into $h(x)$, when it is a positive multiplicative factor.

Two Checks for a Proposed Representation

A representation should pass two direct checks. First, expanding the right-hand side must reproduce the given density or mass function, including any indicator of its support. Second, its integral or sum must be one. The log-partition function is precisely the parameter-only quantity that makes this normalization work.

Normalization identity: For a fixed reference measure $\mu$, suppose $h(x)\geq 0$ and $$ 0<Z(\eta):=\int h(x)\exp\{\eta^{\mathsf T}T(x)\}\,d\mu(x)<\infty. $$ Then the normalized density is obtained with $A(\eta)=\log Z(\eta)$. For a discrete model, the integral is the corresponding sum.

To verify this identity, substitute $A(\eta)=\log Z(\eta)$ into the proposed density and integrate:

\int h(x)\exp\{\eta^{\mathsf T}T(x)-A(\eta)\}\,d\mu(x) =e^{-A(\eta)}Z(\eta) =e^{-\log Z(\eta)}Z(\eta) =1.

The positivity and finiteness conditions matter: if $Z(\eta)=0$, there is no positive normalizing constant, and if $Z(\eta)=\infty$, no finite choice of $A(\eta)$ normalizes the expression. Determining all parameter values for which these conditions hold is a separate question; here they are used only to check a representation.

A Reliable Algebraic Extraction

A practical way to read a formula is to put it into the pattern “parameter-free factor times exponential of a parameter-weighted statistic, minus a parameter-only term.” The following elementary result formalizes that procedure.

Result: Suppose, on a fixed support, a normalized density or mass function can be written as $$ f_{\theta}(x)=c(\theta)h_0(x)\exp\left\{\sum_{j=1}^{p}q_j(\theta)S_j(x)\right\}, $$ where $c(\theta)>0$, $h_0(x)\geq 0$ is independent of $\theta$, and the displayed expression is integrable. Then it has canonical coordinates $$ T(x)=(S_1(x),\ldots,S_p(x))^{\mathsf T},\qquad \eta(\theta)=(q_1(\theta),\ldots,q_p(\theta))^{\mathsf T}, $$ with carrier $h_0(x)$ and $A(\eta(\theta))=-\log c(\theta)$.

Proof. Define $\eta_j(\theta)=q_j(\theta)$ for each $j$, and define $T_j(x)=S_j(x)$. Then the parameter-observation term is $$ \eta(\theta)^{\mathsf T}T(x)=\sum_{j=1}^{p}q_j(\theta)S_j(x). $$ Also, $-A(\eta(\theta))=\log c(\theta)$, so $\exp\{-A(\eta(\theta))\}=c(\theta)$. Substitution gives $$ h_0(x)\exp\{\eta(\theta)^{\mathsf T}T(x)-A(\eta(\theta))\} =h_0(x)\exp\left\{\sum_{j=1}^{p}q_j(\theta)S_j(x)\right\}c(\theta) =f_{\theta}(x). $$ Thus the proposed components reproduce the given normalized function. $\square$

This result identifies a representation from a factorization; it does not assert that the listed statistics are the only possible choice. In particular, one should not infer uniqueness merely because a formula has been successfully rearranged.

Worked Examples

Worked Example: Exponential Density

Let $X$ have density $f_\lambda(x)=\lambda e^{-\lambda x}$ for $x>0$, where $\lambda>0$, and density zero otherwise. Write it in canonical form.

The support is $x>0$. The factor $\lambda$ depends on the parameter, while the exponential contains the product $-\lambda x$. Set $h(x)=1$ on $x>0$, $T(x)=x$, and $\eta=-\lambda$. Since $\lambda>0$, this coordinate satisfies $\eta<0$.

To match the remaining factor, require $e^{-A(\eta)}=\lambda=-\eta$. Taking logarithms gives $A(\eta)=-\log(-\eta)$. Therefore $$ f_\eta(x)=\mathbf{1}_{\{x>0\}}\exp\{\,\eta x+\log(-\eta)\,\} =\mathbf{1}_{\{x>0\}}\exp\{\eta T(x)-A(\eta)\}. $$

Indeed, $-A(\eta)=\log(-\eta)=\log\lambda$, so the expression is $\lambda e^{-\lambda x}$, as required. The support indicator is parameter-free and may be included in the carrier.

Worked Example: A Bounded Continuous Exponential Tilt

Consider a density on $0\leq x\leq 1$ proportional to $e^{\eta x}$. Find its log-partition function and write the normalized density in canonical form.

Take $h(x)=1$ on $[0,1]$, $T(x)=x$, and calculate the normalizing integral. For $\eta\neq 0$, $$ Z(\eta)=\int_0^1 e^{\eta x}\,dx =\left[\frac{e^{\eta x}}{\eta}\right]_0^1 =\frac{e^\eta-1}{\eta}. $$ At $\eta=0$, direct integration gives $Z(0)=\int_0^1 1\,dx=1$. Thus $$ A(\eta)= \begin{cases} \log\left(\dfrac{e^\eta-1}{\eta}\right),&\eta\neq 0,\\ 0,&\eta=0. \end{cases} $$

For a numerical check, let $\eta=\log 2$. Then $Z(\log 2)=(2-1)/\log 2=1/\log 2$, so $A(\log 2)=-\log(\log 2)$. The density is $$ f_{\log 2}(x)=\exp\{(\log 2)x+\log(\log 2)\} =(\log 2)2^x,\qquad 0\leq x\leq 1. $$ Its integral is $(\log 2)\{1/\log 2\}=1$, verifying the normalization.

Worked Example: A Two-Parameter Discrete Model

On the support $\{0,1,2\}$, define $$ P_{\alpha,\beta}(X=x)=\frac{\exp\{\alpha x+\beta x^2\}}{1+e^{\alpha+\beta}+e^{2\alpha+4\beta}}. $$ Identify its canonical components and calculate the probabilities when $\alpha=\log 2$ and $\beta=\log 3$.

The denominator depends only on the parameters. The parameter-observation terms are $\alpha x$ and $\beta x^2$, so take $$ T(x)= \begin{pmatrix} x\\x^2 \end{pmatrix}, \qquad \eta= \begin{pmatrix} \alpha\\\beta \end{pmatrix}, \qquad h(x)=1\quad(x\in\{0,1,2\}). $$ The log-partition function is $$ A(\alpha,\beta)=\log\left(1+e^{\alpha+\beta}+e^{2\alpha+4\beta}\right). $$

At $\alpha=\log 2$, $\beta=\log 3$, the unnormalized weights at $x=0,1,2$ are respectively $$ e^0=1,\qquad e^{\log 2+\log 3}=6,\qquad e^{2\log 2+4\log 3}=2^2 3^4=324. $$ Their sum is $1+6+324=331$, so $A(\log 2,\log 3)=\log 331$. The probabilities are $1/331$, $6/331$, and $324/331$, which sum to one. The two coordinates of the canonical statistic are needed because the exponent contains both $x$ and $x^2$.

Common Mistakes and Why the Form Is Useful

Canonical form is a disciplined way to read a model, not a rule that every visible factor must be put in the same location. The roles follow from parameter dependence and the exact algebra of the expression.

Putting a parameter-dependent factor in the carrier. In the exponential-density example, $\lambda$ cannot be part of $h(x)$, because the carrier must be free of the parameter. It belongs in the parameter-only normalizing term.
Choosing a statistic that omits an exponent term. In the discrete example, using $T(x)=x$ alone would not account for $\beta x^2$. Both $x$ and $x^2$ appear in the canonical statistic.
Forgetting the support. A formula valid only for $x>0$ is not a complete density on the real line unless its zero values outside that support are understood. State the support or include its indicator in $h$.
Confusing the original parameter with the natural parameter. For the exponential density, the original rate is $\lambda$, while the natural parameter is $\eta=-\lambda$. The sign is essential to reproduce the exponent.
Checking the exponent but not the total mass. A correct-looking factorization is not a probability model unless its integral or sum equals one. The log-partition term is checked by normalization.

The practical payoff is that models with very different-looking formulas can be inspected through the same four components. Identifying $T(x)$ makes clear which features of the observation are paired with parameters, while $A(\eta)$ records the parameter-only adjustment required for a probability distribution. This representation also provides a starting point for later study of how the log-partition function and canonical statistics encode distributional properties.

Key takeaway: Expand the exponent, assign parameter-free observation factors to $h(x)$, collect each parameter-weighted observation function into $T(x)$, and place the remaining parameter-only normalization in $A(\eta)$. Then verify the support and total probability.

Check Your Understanding

For each item, identify the canonical components or explain the relevant check.

For a density proportional to $e^{-\theta x^2}$ on $x>0$, with $\theta>0$, what statistic and natural parameter appear in the exponent?
In a representation with exponent $\eta_1x+\eta_2\log x-A(\eta)$, what is the canonical statistic vector?
Why can a factor depending only on the parameter not be placed in the carrier?
For a proposed representation, what integral or sum must be checked to verify that it defines a probability distribution?
If a formula is written on a restricted support, how should values outside that support be handled in a complete density or mass function?