Why These Canonical Forms Matter
The normal and gamma distributions appear throughout statistical modeling, but their familiar parameterizations do not always make their likelihood structure easy to compare. Writing a density in canonical exponential-family form makes explicit which functions of the data enter the exponent, how the ordinary parameters determine the natural parameters, and what restrictions the normalizing constant imposes.
This tutorial uses \(X\sim N(\mu,\sigma^2)\), with \(\sigma^2\) denoting the variance, and the gamma shape-rate parameterization. Both are represented relative to Lebesgue measure on their stated supports. The focus is the full two-parameter family in each case; a fixed-variance normal or fixed-shape gamma family imposes an additional restriction on the natural parameters.
A useful check on a proposed representation is to substitute the natural parameters and simplify the exponent. It must recover the original density, including its support and normalizing constant. The parameter-space restriction is not an optional add-on: it is what ensures that the exponential expression can be normalized.
The Gaussian Canonical Representation
For \(\mu\in\mathbb{R}\) and \(\sigma^2>0\), the normal density on \(\mathbb{R}\) is $$ f_{\mu,\sigma^2}(x)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left\{-\frac{(x-\mu)^2}{2\sigma^2}\right\}. $$ Expanding the square separates the data-dependent terms \(x\) and \(x^2\) from terms involving only the parameters: $$ -\frac{(x-\mu)^2}{2\sigma^2} =-\frac{x^2}{2\sigma^2}+\frac{\mu}{\sigma^2}x-\frac{\mu^2}{2\sigma^2}. $$ Thus the natural statistics are \(T(x)=(x,x^2)^\mathsf{T}\), and the natural parameters are \(\eta_1=\mu/\sigma^2\) and \(\eta_2=-1/(2\sigma^2)\). In particular, \(\eta_2\) must be negative.
The natural parameter space is all of \(\mathbb{R}\times(-\infty,0)\), not just the values from one particular normal distribution. Indeed, every pair in that set gives a positive variance through \(-1/(2\eta_2)\) and a real mean through \(-\eta_1/(2\eta_2)\). Conversely, every \(\mu\in\mathbb{R}\), \(\sigma^2>0\) produces such a pair.
Worked Example: Writing \(N(2,9)\) in Canonical Form
For \(X\sim N(2,9)\), the variance is \(9\), so $$ \eta_1=\frac{2}{9},\qquad \eta_2=-\frac{1}{18}. $$ The second natural parameter is negative, as required. The log-partition function is $$ A\left(\frac29,-\frac1{18}\right) =-\frac{(2/9)^2}{4(-1/18)} +\frac12\log\left(\frac{\pi}{1/18}\right) =\frac{2}{9}+\frac12\log(18\pi). $$ Consequently the canonical density is $$ f(x)=\exp\left\{\frac29x-\frac1{18}x^2-\frac29-\frac12\log(18\pi)\right\}. $$ The exponent’s quadratic terms combine as $$ \frac29x-\frac1{18}x^2-\frac29 =-\frac{(x-2)^2}{18}. $$ The remaining constant gives \(1/\sqrt{18\pi}=1/\sqrt{2\pi\cdot9}\), so this is exactly the \(N(2,9)\) density.
Deriving the Gaussian Log-Partition Function
The log-partition function is determined by requiring the canonical expression to integrate to one. For \(\eta_2<0\), complete the square: $$ \eta_1x+\eta_2x^2 =\eta_2\left(x+\frac{\eta_1}{2\eta_2}\right)^2-\frac{\eta_1^2}{4\eta_2}. $$ Therefore $$ \int_{\mathbb{R}}\exp(\eta_1x+\eta_2x^2)\,dx =\exp\left(-\frac{\eta_1^2}{4\eta_2}\right) \int_{\mathbb{R}}\exp\left\{\eta_2\left(x+\frac{\eta_1}{2\eta_2}\right)^2\right\}\,dx. $$ Because \(-\eta_2>0\), translating the integration variable and applying the Gaussian integral \(\int_{\mathbb{R}}e^{-a u^2}\,du=\sqrt{\pi/a}\), for \(a>0\), gives $$ \int_{\mathbb{R}}\exp(\eta_1x+\eta_2x^2)\,dx =\exp\left(-\frac{\eta_1^2}{4\eta_2}\right)\sqrt{\frac{\pi}{-\eta_2}}. $$ Taking the logarithm yields the stated \(A(\eta_1,\eta_2)\). If \(\eta_2=0\), the integrand is \(e^{\eta_1x}\), whose integral over \(\mathbb{R}\) is infinite for every \(\eta_1\). If \(\eta_2>0\), the positive quadratic term makes the integral infinite. Thus \(\eta_2<0\) is necessary as well as sufficient. \(\square\)
When the variance is fixed at a known value, \(\eta_2\) is fixed at \(-1/(2\sigma^2)\), while \(\eta_1\) varies with \(\mu\). That is a restricted subfamily of the full normal family, rather than a different full two-parameter natural space.
Worked Example: Checking \(N(-1,4)\) by Completing the Square
For \(\mu=-1\) and \(\sigma^2=4\), $$ \eta_1=-\frac14,\qquad \eta_2=-\frac18. $$ The formula for \(A\) gives $$ A\left(-\frac14,-\frac18\right) =-\frac{1/16}{-1/2}+\frac12\log(8\pi) =\frac18+\frac12\log(8\pi). $$ The canonical exponent is therefore $$ -\frac{x}{4}-\frac{x^2}{8}-\frac18-\frac12\log(8\pi) =-\frac{(x+1)^2}{8}-\frac12\log(8\pi). $$ This is the log density of \(N(-1,4)\): its quadratic term has denominator \(2\sigma^2=8\), and its normalizing factor is \(1/\sqrt{8\pi}\).
The Gamma Canonical Representation
Use the shape-rate convention: for shape \(\alpha>0\) and rate \(\beta>0\), the gamma density is $$ f_{\alpha,\beta}(x)=\frac{\beta^\alpha}{\Gamma(\alpha)}x^{\alpha-1}e^{-\beta x},\qquad x>0, $$ and it is zero for \(x\leq0\). Here \(\Gamma(\alpha)=\int_0^\infty t^{\alpha-1}e^{-t}\,dt\), defined for \(\alpha>0\).
The data-dependent terms in the log density are \(\log x\) and \(x\). The parameter-dependent remainder is \(\alpha\log\beta-\log\Gamma(\alpha)\). Setting \(\eta_1=\alpha-1\) and \(\eta_2=-\beta\) gives the canonical form below.
Worked Example: A Gamma Distribution with Shape 3 and Rate 2
For \(\alpha=3\), \(\beta=2\), the natural parameters are \(\eta_1=2\) and \(\eta_2=-2\). Both constraints hold: \(2>-1\) and \(-2<0\). Since \(\Gamma(3)=2\), $$ A(2,-2)=\log\Gamma(3)-3\log 2 =\log 2-3\log 2=-2\log 2. $$ The canonical density becomes, for \(x>0\), $$ \exp\{2\log x-2x+2\log2\}=4x^2e^{-2x}. $$ The original density is \(\frac{2^3}{\Gamma(3)}x^{3-1}e^{-2x}=4x^2e^{-2x}\), so the representation matches, including its normalizing constant.
Deriving the Gamma Log-Partition Function
For the canonical expression to define a probability density, its normalizing integral over the support must be finite. Write \(a=\eta_1+1\) and \(b=-\eta_2\). The integral is $$ \int_0^\infty x^{\eta_1}e^{\eta_2x}\,dx =\int_0^\infty x^{a-1}e^{-bx}\,dx. $$ When \(a>0\) and \(b>0\), the substitution \(t=bx\) gives \(x=t/b\), \(dx=dt/b\), and hence $$ \int_0^\infty x^{a-1}e^{-bx}\,dx =b^{-a}\int_0^\infty t^{a-1}e^{-t}\,dt =b^{-a}\Gamma(a). $$ Taking logarithms gives $$ A(\eta_1,\eta_2)=\log\Gamma(\eta_1+1)-(\eta_1+1)\log(-\eta_2). $$ The restrictions are necessary: near zero, \(x^{\eta_1}\) is integrable only if \(\eta_1>-1\); if \(\eta_1=-1\), its integral diverges logarithmically, and if \(\eta_1<-1\), it diverges more strongly. At infinity, the exponential factor decays only when \(\eta_2<0\). If \(\eta_2=0\), a power function has no finite integral over the whole positive half-line, and if \(\eta_2>0\), the integrand grows exponentially. Together these facts establish exactly \(\mathcal{H}_G\). \(\square\)
Worked Example: A Gamma Distribution with Shape One-Half
Take \(\alpha=1/2\) and \(\beta=3\). Then \(\eta_1=-1/2\), \(\eta_2=-3\), which satisfies \(\eta_1>-1\) and \(\eta_2<0\). Using \(\Gamma(1/2)=\sqrt{\pi}\), $$ A\left(-\frac12,-3\right) =\log\Gamma\left(\frac12\right)-\frac12\log3 =\frac12\log\left(\frac{\pi}{3}\right). $$ Thus, for \(x>0\), the canonical density is $$ \exp\left\{-\frac12\log x-3x-\frac12\log\left(\frac{\pi}{3}\right)\right\} =\sqrt{\frac{3}{\pi}}x^{-1/2}e^{-3x}. $$ The original gamma normalizing factor is \(3^{1/2}/\Gamma(1/2)=\sqrt{3/\pi}\), confirming the result.
Independent Samples and a Common Pitfall
For independent observations \(X_1,\ldots,X_n\) from either family, the joint density is the product of the individual densities. For the full normal family this gives $$ \prod_{i=1}^n f_\eta(x_i) =\exp\left\{\eta_1\sum_{i=1}^n x_i+\eta_2\sum_{i=1}^n x_i^2-nA(\eta_1,\eta_2)\right\}. $$ For the gamma family it gives $$ \mathbf{1}_{\{x_i>0\ \text{for all }i\}} \exp\left\{\eta_1\sum_{i=1}^n\log x_i+\eta_2\sum_{i=1}^n x_i-nA(\eta_1,\eta_2)\right\}. $$ The one-observation log-partition function is multiplied by \(n\); the natural parameter space itself does not change.
Common mistake: ignoring signs or support. One may see \(\eta_2=-1/(2\sigma^2)\) and mistakenly allow positive values, or write a gamma expression using \(\log x\) without stating that \(x>0\). Neither is harmless: for the normal family, a nonnegative quadratic coefficient fails to give a finite integral over \(\mathbb{R}\); for the gamma family, the statistic \(\log x\) is defined only on the positive support, and the integral is finite only for \(\eta_1>-1\) and \(\eta_2<0\). Always check the support and the entire natural parameter vector.
These representations make likelihood calculations and comparisons more transparent: the statistics identify how observations enter the exponent, while the log-partition function supplies the exact normalizer. They also provide a direct test of whether a proposed parameter value belongs to the family. For the full normal family, check \(\eta_2<0\); for the shape-rate gamma family, check both \(\eta_1>-1\) and \(\eta_2<0\).
Check Your Understanding
Use the canonical forms and parameter-space conditions to answer each question.
- For \(N(3,2)\), find \(\eta_1\) and \(\eta_2\), and state the log-partition function after substitution.
- Explain why \((\eta_1,\eta_2)=(1,0)\) cannot be a normal natural-parameter vector.
- For a gamma distribution with shape \(4\) and rate \(1/2\), find the natural parameters and the log-partition function.
- Why does the gamma natural parameter space require \(\eta_1>-1\), independently of the requirement \(\eta_2<0\)?
- For an independent normal sample, which two sample sums appear in the canonical exponent, and what happens to the log-partition term?