From Familiar Laws to Canonical Form
A probability law can look quite different in its usual parameters and in its canonical representation. Converting between the two is more than a change of notation: it identifies the statistic that carries the parameter dependence, the allowable natural parameters, and the function that normalizes the distribution. This tutorial carries out those reductions for a geometric law, a gamma law, and a normal law, keeping the support and parameterization explicit in each case.
We use the one-observation form of an exponential family. The reference measure is counting measure for a discrete distribution and Lebesgue measure for a continuous one. A carrier function records the part of the density or mass function that does not depend on the parameter; the log-partition function ensures that the resulting expression integrates or sums to one.
The canonical statistic need not be the observation itself. For a model with more than one natural parameter, it is a vector. We will find the normalizing integral directly in each example, then use derivatives of its logarithm to obtain moments.
A Moment Identity for Canonical Families
The log-partition function encodes the means and covariances of the canonical statistic. The identity is useful only when differentiation can be passed through the normalizing integral, so we state that condition explicitly.
Proof. Write the normalizing integral as $$ Z(\eta)=\int h(x)\exp\{\eta^\mathsf{T}T(x)\}\,d\nu(x), \qquad A(\eta)=\log Z(\eta). $$ By the assumed differentiation condition, $$ \frac{\partial Z}{\partial \eta_i} =\int T_i(x)h(x)e^{\eta^\mathsf{T}T(x)}\,d\nu(x). $$ Since \(f_\eta(x)=h(x)e^{\eta^\mathsf{T}T(x)}/Z(\eta)\), division by \(Z(\eta)\) gives \(\partial A/\partial\eta_i=E_\eta[T_i(X)]\). Differentiating once more, $$ \frac{\partial^2 A}{\partial\eta_i\partial\eta_j} =\frac{1}{Z}\frac{\partial^2 Z}{\partial\eta_i\partial\eta_j} -\frac{1}{Z^2}\frac{\partial Z}{\partial\eta_i}\frac{\partial Z}{\partial\eta_j} =E_\eta[T_i(X)T_j(X)]-E_\eta[T_i(X)]E_\eta[T_j(X)]. $$ The right-hand side is the \((i,j)\) entry of the covariance matrix. This proves both identities. \(\square\)
Worked Example: Geometric Failures Before the First Success
Worked Example: Geometric Failures Before the First Success
Let \(X\) count the failures before the first success, with success probability \(p\), where \(0<p<1\). Thus \(X\) takes values \(0,1,2,\ldots\) and \(P(X=x)=p(1-p)^x\). Find the canonical representation and its first two moments.
Carrier and statistic. Relative to counting measure on the nonnegative integers, take \(h(x)=1\) for \(x=0,1,\ldots\), and \(T(x)=x\). The mass function can be written $$ p(1-p)^x =\exp\{x\log(1-p)+\log p\}. $$ Set \(\eta=\log(1-p)\). Then \(\eta<0\), \(p=1-e^\eta\), and $$ f_\eta(x)=h(x)\exp\{\eta x-A(\eta)\}, \qquad A(\eta)=-\log(1-e^\eta). $$
Natural parameter space. The geometric series \(\sum_{x=0}^\infty e^{\eta x}\) is finite exactly when \(e^\eta<1\), or \(\eta<0\). In that case its sum is \(1/(1-e^\eta)\), so its logarithm is \(A(\eta)\), as stated.
Moments. Differentiation gives $$ A'(\eta)=\frac{e^\eta}{1-e^\eta}, \qquad A''(\eta)=\frac{e^\eta}{(1-e^\eta)^2}. $$ The moment identity yields \(E[X]=A'(\eta)=(1-p)/p\) and \(\operatorname{Var}(X)=A''(\eta)=(1-p)/p^2\). Therefore $$ E[X^2]=\operatorname{Var}(X)+(E[X])^2 =\frac{1-p}{p^2}+\frac{(1-p)^2}{p^2} =\frac{(1-p)(2-p)}{p^2}. $$
Worked Example: Gamma Shape and Rate
Worked Example: Gamma Shape and Rate
Let \(X\) have a gamma distribution with shape \(\alpha>0\) and rate \(\beta>0\), with density $$ f(x)=\frac{\beta^\alpha}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x}, \qquad x>0. $$ Here \(\Gamma(\alpha)=\int_0^\infty x^{\alpha-1}e^{-x}\,dx\). Find the canonical form and the first two moments of \(X\).
Carrier and statistic. Relative to Lebesgue measure, take \(h(x)=1\) on \(x>0\), with \(h(x)=0\) elsewhere, and \(T(x)=(\log x,x)^\mathsf{T}\). Set \(\eta_1=\alpha-1\) and \(\eta_2=-\beta\). The density becomes $$ f_\eta(x)=h(x)\exp\{\eta_1\log x+\eta_2x-A(\eta)\}, \qquad A(\eta)=\log\Gamma(\eta_1+1)-(\eta_1+1)\log(-\eta_2). $$
Natural parameter space. The normalizing integral is $$ \int_0^\infty x^{\eta_1}e^{\eta_2x}\,dx. $$ It is finite near zero exactly when \(\eta_1>-1\), and at infinity exactly when \(\eta_2<0\). Substituting \(u=(-\eta_2)x\) evaluates it as \(\Gamma(\eta_1+1)/(-\eta_2)^{\eta_1+1}\). Thus the natural parameter space is \(\eta_1>-1,\ \eta_2<0\), and the logarithm of this integral is the displayed \(A(\eta)\).
First two moments. Write \(\alpha=\eta_1+1\) and \(\beta=-\eta_2\). The identity for derivatives of \(A\) gives \(E[\log X]=\psi(\alpha)-\log\beta\), where \(\psi(\alpha)=d\log\Gamma(\alpha)/d\alpha\), and \(E[X]=-\partial A/\partial\eta_2=\alpha/\beta\). Also, \(\operatorname{Var}(X)=\partial^2 A/\partial\eta_2^2=\alpha/\beta^2\). Hence $$ E[X^2]=\operatorname{Var}(X)+(E[X])^2 =\frac{\alpha}{\beta^2}+\frac{\alpha^2}{\beta^2} =\frac{\alpha(\alpha+1)}{\beta^2}. $$ These moment calculations can also be checked directly: multiplying the density by \(x^k\) and using the gamma integral gives \(E[X^k]=\Gamma(\alpha+k)/(\Gamma(\alpha)\beta^k)\) for \(k=1,2\).
Worked Example: Normal Mean and Variance
Worked Example: Normal Mean and Variance
Let \(X\sim N(\mu,\sigma^2)\), with \(\mu\in\mathbb R\) and \(\sigma^2>0\). Find its canonical form, natural parameter space, and first two moments.
Carrier and statistic. Relative to Lebesgue measure on \(\mathbb R\), take \(h(x)=1\) and \(T(x)=(x,x^2)^\mathsf{T}\). Expanding the square in the density gives $$ f(x)=\exp\left\{ \frac{\mu}{\sigma^2}x-\frac{1}{2\sigma^2}x^2 -\frac{\mu^2}{2\sigma^2}-\frac12\log(2\pi\sigma^2) \right\}. $$ Thus \(\eta_1=\mu/\sigma^2\), \(\eta_2=-1/(2\sigma^2)\), and $$ A(\eta_1,\eta_2) =-\frac{\eta_1^2}{4\eta_2} +\frac12\log\left(\frac{-\pi}{\eta_2}\right). $$
Natural parameter space. Integrability requires a negative coefficient on \(x^2\), so \(\eta_2<0\); there is no restriction on \(\eta_1\). Conversely, for every \(\eta_1\in\mathbb R,\eta_2<0\), completing the square gives a finite Gaussian integral. The natural parameter space is therefore \(\mathbb R\times(-\infty,0)\). The inverse parameter relations are \(\sigma^2=-1/(2\eta_2)\) and \(\mu=-\eta_1/(2\eta_2)\).
Moments. Differentiating the log-partition function, $$ \frac{\partial A}{\partial\eta_1}=-\frac{\eta_1}{2\eta_2}=E[X], \qquad \frac{\partial A}{\partial\eta_2} =\frac{\eta_1^2}{4\eta_2^2}-\frac{1}{2\eta_2}=E[X^2]. $$ Substituting the inverse relations gives \(E[X]=\mu\) and \(E[X^2]=\mu^2+\sigma^2\), so \(\operatorname{Var}(X)=\sigma^2\). As a check, the second derivative with respect to \(\eta_1\) is \(-1/(2\eta_2)=\sigma^2\), the variance of the first canonical statistic \(X\).
What the Three Reductions Have in Common
In all three cases, the log-partition function is the logarithm of the integral or sum that normalizes the unnormalized canonical expression. Its derivatives provide moments of the canonical statistic, but the statistic differs across families: it is \(X\) for the geometric law, \((\log X,X)\) for the gamma law, and \((X,X^2)\) for the normal law. In particular, for the gamma and normal models, the second coordinate is not \(X^2\) in every case; it is \(X\) for gamma and \(X^2\) for normal.
The support and natural parameter space are part of the representation, not details to append later. For example, the gamma integral requires both a shape-related restriction at zero and a rate-related restriction at infinity. The normal integral requires a negative quadratic coefficient. These restrictions are what make the normalizing integral finite.
Common Mistakes
- Changing the geometric convention without changing the support. The law here counts failures before the first success, so it includes zero. A geometric law counting the trial of the first success has support \(1,2,\ldots\) and a different expression.
- Confusing gamma rate and scale. This tutorial uses the rate \(\beta\), which appears as \(e^{-\beta x}\). Replacing it with a scale parameter without changing the density and canonical parameter gives incorrect signs and moments.
- Using only one statistic for the two-parameter normal family. Both \(x\) and \(x^2\) carry parameter dependence after expanding the density; the canonical statistic is the pair \((x,x^2)\).
- Declaring a natural parameter valid from its algebraic appearance alone. A candidate canonical expression is not a probability law unless its normalizing integral is finite. Check the support and integrability conditions.
Check Your Understanding
Use the support and parameterizations stated in this tutorial.
- For the geometric failures-count law, what changes in the canonical expression if the support is changed to \(1,2,\ldots\) and the mass function is \(p(1-p)^{x-1}\)?
- Why must the gamma natural parameter satisfy both \(\eta_1>-1\) and \(\eta_2<0\)? Identify which endpoint each condition controls.
- For the gamma model, use \(E[X^2]=\alpha(\alpha+1)/\beta^2\) to recover \(\operatorname{Var}(X)\).
- Explain why the normal canonical statistic contains \(X^2\), and state the restriction on its natural parameter coefficient.
- In the general moment identity, what regularity condition allows differentiation of the normalizing integral under the integral sign?