Tutorials › Statistical Theory I - Probability and Distribution Theory › The Log-Partition Function Normalizes the Family

Home  /  Tutorials  /  Exponential-Family Structure

Exponential-Family Structure · Tutorial 598 of 994

The Log-Partition Function Normalizes the Family

Learn how the carrier measure determines the normalizing integral or sum, and how to calculate the log-partition function and its domain in three standard families.

Advanced 10 min read

What You'll Learn

  • Define the normalizing integral and log-partition function relative to a carrier measure.
  • Prove why normalization uniquely determines the log-partition function.
  • Calculate the log-partition function for a finite-support family and track its domain.
  • Derive the Poisson log-partition function from its counting-measure representation.
  • Derive the Gaussian location-family log-partition function by completing the square.
  • Recognize how changing the carrier measure changes the displayed log-partition function.

Why Normalization Determines the Log-Partition Function

An exponential-family representation separates the part of a distribution that depends on the observation from a single parameter-dependent constant. That constant is not optional: it is exactly what makes the probabilities sum to one or the density integrate to one. Finding it is therefore a normalization problem, and the answer depends on both the canonical statistic and the carrier measure.

Write a one-parameter canonical family, relative to a fixed measure \(\nu\), as \(p_\eta(x)=\exp\{\eta T(x)-A(\eta)\}\). Here \(p_\eta\) is a density with respect to \(\nu\), not necessarily with respect to Lebesgue measure. If the carrier measure has a density or mass weight \(h\), that weight can instead be displayed explicitly; the corresponding normalization then includes \(h\).

Definition: Let \(T\) be measurable and let \(\nu\) be a nonzero measure that does not depend on \(\eta\). Define the normalizing integral (or sum) \(Z(\eta)=\int \exp\{\eta T(x)\}\,\nu(dx)\). The natural parameter domain is the set of \(\eta\) for which \(0<Z(\eta)<\infty\). On this domain, the log-partition function is \(A(\eta)=\log Z(\eta)\), and the normalized density relative to \(\nu\) is \(p_\eta(x)=\exp\{\eta T(x)-A(\eta)\}\). For discrete support, the integral is a sum.

The condition \(Z(\eta)<\infty\) ensures that the proposed normalizing constant is finite. The condition \(Z(\eta)>0\) ensures it is possible to normalize to total mass one. In the examples below, the carrier is nonzero and the integrand is strictly positive, so \(Z(\eta)>0\) whenever it is finite. The domain itself is found by checking this finiteness condition, not by assuming in advance that every real \(\eta\) is allowed.

Theorem: For a fixed carrier measure \(\nu\) and measurable statistic \(T\), the expression \(\exp\{\eta T(x)-A(\eta)\}\) is a probability density relative to \(\nu\) if and only if \(0<Z(\eta)<\infty\) and \(A(\eta)=\log Z(\eta)\).

Proof. For any finite real \(A(\eta)\), integrate the proposed density:

$$ \int \exp\{\eta T(x)-A(\eta)\}\,\nu(dx) =e^{-A(\eta)}\int \exp\{\eta T(x)\}\,\nu(dx) =e^{-A(\eta)}Z(\eta). $$

For this integral to equal one, \(Z(\eta)\) must be finite and strictly positive, and \(e^{-A(\eta)}Z(\eta)=1\). Solving gives \(A(\eta)=\log Z(\eta)\). Conversely, when \(0<Z(\eta)<\infty\), this choice is finite and the displayed integral equals one. Since the logarithm is single-valued on positive real numbers, no other finite value of \(A(\eta)\) can normalize the same expression. \(\square\)

Worked Normalizations

For a discrete family with carrier weights \(h(x)\), the normalizing sum is \(\sum_x h(x)e^{\eta T(x)}\). For a continuous family written with respect to Lebesgue measure, it is \(\int h(x)e^{\eta T(x)}\,dx\). These are the same construction with different ways of expressing the carrier.

Worked Example: A Finite-Support Family

Let \(X\) take values \(0,1,2\), use counting measure as the underlying measure, and give the carrier weights \(h(0)=1\), \(h(1)=2\), and \(h(2)=1\). Take \(T(x)=x\). Find the log-partition function and evaluate the family at \(\eta=\log 2\).

The normalizing sum is

$$ Z(\eta)=1e^{0\eta}+2e^\eta+1e^{2\eta} =1+2e^\eta+e^{2\eta} =(1+e^\eta)^2. $$

It is positive and finite for every finite real \(\eta\), so the natural parameter domain is \(\mathbb R\). Thus

$$ A(\eta)=\log\{1+2e^\eta+e^{2\eta}\} =2\log(1+e^\eta). $$

At \(\eta=\log 2\), \(e^\eta=2\), \(Z(\eta)=1+4+4=9\), and \(A(\eta)=\log 9\). The three probabilities, including their carrier weights, are

$$ P(X=0)=\frac{1}{9},\qquad P(X=1)=\frac{2(2)}{9}=\frac{4}{9},\qquad P(X=2)=\frac{2^2}{9}=\frac{4}{9}. $$

They sum to \(1/9+4/9+4/9=1\), as required. In particular, the weight \(h(1)=2\) belongs in the numerator as well as in the normalizing sum.

Worked Example: The Poisson Family

For \(x=0,1,2,\ldots\), write the Poisson mass function with rate \(\lambda>0\) in canonical form. The carrier weight is \(h(x)=1/x!\), the statistic is \(T(x)=x\), and the natural parameter is \(\eta=\log\lambda\), so \(\lambda=e^\eta\).

The normalizing sum at a finite real \(\eta\) is the exponential series:

$$ Z(\eta)=\sum_{x=0}^{\infty}\frac{e^{\eta x}}{x!} =\sum_{x=0}^{\infty}\frac{(e^\eta)^x}{x!} =\exp(e^\eta). $$

This is finite and positive for every \(\eta\in\mathbb R\). Therefore the natural parameter domain is all of \(\mathbb R\), and

$$ A(\eta)=\log Z(\eta)=e^\eta. $$

The resulting mass is \(p_\eta(x)=\exp\{\eta x-e^\eta\}/x!\). For example, at \(\lambda=3\), \(\eta=\log 3\) and \(A(\eta)=3\). The probability at \(x=2\) is

$$ p_{\log 3}(2) =\frac{\exp\{2\log 3-3\}}{2!} =\frac{9e^{-3}}{2}. $$

More generally, summing the mass function over all nonnegative integers gives \(e^{-e^\eta}Z(\eta)=e^{-e^\eta}e^{e^\eta}=1\). The rate must be positive in the usual Poisson parameterization, but \(\eta=\log\lambda\) ranges over every finite real number. The zero-rate limiting distribution corresponds to \(\eta\) tending to \(-\infty\), not to a finite point in this natural parameter domain.

Worked Example: A Gaussian Location Family With Fixed Variance

Let \(X\sim N(\mu,\sigma^2)\), where \(\sigma^2>0\) is known and fixed while \(\mu\) varies. Express the density in canonical form using \(T(x)=x\) and find the domain of its natural parameter.

Expanding the square in the Gaussian density gives

$$ -\frac{(x-\mu)^2}{2\sigma^2} =-\frac{x^2}{2\sigma^2} +\frac{\mu}{\sigma^2}x -\frac{\mu^2}{2\sigma^2}. $$

Set \(\eta=\mu/\sigma^2\), so \(\mu=\sigma^2\eta\). Taking the fixed carrier density to be \(h_\sigma(x)=(2\pi\sigma^2)^{-1/2}\exp\{-x^2/(2\sigma^2)\}\), the parameter-dependent terms have the form \(\eta x-A(\eta)\), with

$$ A(\eta)=\frac{\mu^2}{2\sigma^2} =\frac{\sigma^2\eta^2}{2}. $$

The normalizing integral confirms this result directly:

$$ \begin{aligned} Z(\eta) &=\int_{-\infty}^{\infty}h_\sigma(x)e^{\eta x}\,dx\\ &=\frac{1}{\sqrt{2\pi\sigma^2}} \int_{-\infty}^{\infty} \exp\left\{-\frac{x^2}{2\sigma^2}+\eta x\right\}\,dx\\ &=\exp\left\{\frac{\sigma^2\eta^2}{2}\right\} \frac{1}{\sqrt{2\pi\sigma^2}} \int_{-\infty}^{\infty} \exp\left\{-\frac{(x-\sigma^2\eta)^2}{2\sigma^2}\right\}\,dx\\ &=\exp\left\{\frac{\sigma^2\eta^2}{2}\right\}. \end{aligned} $$

The last integral, including its prefactor, is one: it is the integral of a normal density with mean \(\sigma^2\eta\) and variance \(\sigma^2\). Thus \(Z(\eta)\) is finite and positive for every finite real \(\eta\), and the natural parameter domain is \(\mathbb R\). For the concrete case \(\sigma^2=4\) and \(\mu=2\), \(\eta=1/2\) and \(A(\eta)=4(1/2)^2/2=1/2\), agreeing with \(\mu^2/(2\sigma^2)=4/8=1/2\).

Carrier Measures and the Meaning of the Constant

The log-partition function is determined by the whole normalizing integral, including the carrier. In a discrete representation, omitting a weight changes the sum; in a continuous representation, omitting the base density changes the integral. A useful check is to integrate or sum the final expression as written, with all carrier terms included.

The numerical value of \(A\) also depends on the chosen carrier representation. For example, replacing \(\nu\) by \(c\nu\), where \(c>0\), changes \(Z(\eta)\) to \(cZ(\eta)\). The corresponding log-partition function becomes \(A(\eta)+\log c\). This does not change the probability distribution: the carrier has been multiplied by \(c\), while the exponential factor \(e^{-A}\) has been divided by \(c\). The density relative to its specified carrier, rather than \(A\) in isolation, describes the distribution.

Common Mistakes

  • Using the wrong sign. If the canonical exponent is \(\eta T(x)-A(\eta)\), then \(A(\eta)=\log Z(\eta)\), not \(-\log Z(\eta)\). Indeed, the total mass is \(e^{-A(\eta)}Z(\eta)\).
  • Dropping the carrier weight. In the Poisson sum, \(1/x!\) is essential. Summing only \(e^{\eta x}\) would not give the Poisson normalizer.
  • Assuming every natural parameter is allowed. The domain consists of parameters for which the normalizing integral or sum is finite and positive. The three examples here all have domain \(\mathbb R\), but that is a property to verify, not a general rule.
  • Confusing the natural and original parameters. For Poisson, \(\eta=\log\lambda\), not \(\lambda\); for the fixed-variance Gaussian, \(\eta=\mu/\sigma^2\), not \(\mu\). The log-partition function is expressed in the natural parameter.
Key takeaway: First identify the carrier measure and the canonical statistic; then calculate \(Z(\eta)\), determine where it is finite and positive, and set \(A(\eta)=\log Z(\eta)\). This order makes the normalization and the natural parameter domain explicit.

Check Your Understanding

Try these questions by writing the normalizing integral or sum before identifying \(A(\eta)\).

  1. For support \(\{0,1\}\), counting measure, and \(T(x)=x\), write \(Z(\eta)\), find \(A(\eta)\), and determine its domain.
  2. In the Poisson representation, which part of the carrier produces the factor \(1/x!\) in the mass function?
  3. For a fixed-variance Gaussian location family, express \(\eta\) in terms of \(\mu\) and \(\sigma^2\). Why is every finite real \(\eta\) allowed?
  4. If a carrier measure is multiplied by a positive constant \(c\), how does the log-partition function change, and why does the represented probability distribution remain the same?
  5. A proposed canonical density has normalizing integral \(Z(\eta)=\infty\). Can any finite value of \(A(\eta)\) make it a probability density? Explain using the normalization equation.

Previous

Next

Back to Tutorials