Normalization and the Log-Partition Function

Why Normalization Determines the Log-Partition Function

An exponential-family representation separates a response model into a kernel and a normalizing term. The kernel contains the part that varies with the response and the parameter; the normalizing term makes the resulting probabilities sum to one, or the density integrate to one. The latter requirement is not an optional convention: once the kernel and its support are fixed, normalization determines the log-partition function.

This tutorial works directly from that requirement. It derives the normalizer for both discrete and continuous responses, then proves a useful consequence: under suitable conditions, the first derivative of the log-partition function is the mean of the statistic in the kernel, and its second derivative is that statistic's variance.

Definition: Let $Y$ take values in a support $\mathcal{Y}$, and consider a one-parameter kernel $h(y)\exp\{\eta T(y)\}$, where $h(y)\geq 0$ and $T(y)$ is a specified statistic. At a value of $\eta$ for which the following sum or integral is finite and positive, define the normalizing constant $Z(\eta)$ by $$ Z(\eta)=\sum_{y\in\mathcal{Y}}h(y)\exp\{\eta T(y)\} \quad\text{or}\quad Z(\eta)=\int_{\mathcal{Y}}h(y)\exp\{\eta T(y)\}\,dy, $$ as appropriate. The log-partition function is $A(\eta)=\log Z(\eta)$. The normalized probability mass function or density is $$ f(y;\eta)=h(y)\exp\{\eta T(y)-A(\eta)\}. $$

For a continuous response, the integral is with respect to the measure used to define the density; in the examples below it is ordinary integration with respect to $dy$. In either case, the defining equation is that the normalized model has total probability one. If the normalizer is finite and positive, taking logarithms gives $A(\eta)=\log Z(\eta)$. Thus $A$ is not chosen independently of the kernel.

Deriving the Normalizer by Summation

Worked Example: A Three-Point Discrete Response

Suppose $Y$ has support $\{0,1,2\}$, with kernel $\exp(\eta y)$. Find the normalizing constant and log-partition function, and verify the resulting probabilities.

Here $h(y)=1$ and $T(y)=y$, so the total mass of the kernel is the sum of its three values: $$ Z(\eta)=\sum_{y=0}^{2}\exp(\eta y)=1+\exp(\eta)+\exp(2\eta). $$ This is finite and positive for every real $\eta$. Therefore $$ A(\eta)=\log\left(1+\exp(\eta)+\exp(2\eta)\right). $$

The normalized probabilities are $$ P(Y=0)=\frac{1}{Z(\eta)},\qquad P(Y=1)=\frac{\exp(\eta)}{Z(\eta)},\qquad P(Y=2)=\frac{\exp(2\eta)}{Z(\eta)}. $$ Their sum is $$ \frac{1+\exp(\eta)+\exp(2\eta)}{Z(\eta)}=1, $$ as required. At $\eta=0$, $Z(0)=3$, $A(0)=\log 3$, and each of the three outcomes has probability $1/3$.

The calculation illustrates a general point about a discrete support: each allowed response contributes its kernel value to the sum. Omitting an allowed response changes the normalizer and hence changes every probability. The support is part of the model, not a detail to add after normalization.

Deriving the Normalizer by Integration

Worked Example: An Exponential Kernel on a Bounded Interval

Let $Y$ have support $[0,1]$ and kernel $\exp(\eta y)$, so $h(y)=1$. Integrate the kernel to find $Z(\eta)$ and $A(\eta)$.

For $\eta\ne 0$, $$ Z(\eta)=\int_0^1\exp(\eta y)\,dy =\left[\frac{\exp(\eta y)}{\eta}\right]_{0}^{1} =\frac{\exp(\eta)-1}{\eta}. $$ At $\eta=0$, the kernel is identically one, so $Z(0)=\int_0^1 1\,dy=1$. The expression for nonzero $\eta$ is positive: when $\eta>0$, both numerator and denominator are positive; when $\eta<0$, both are negative. Thus the log-partition function is $$ A(\eta)=\log\left(\frac{\exp(\eta)-1}{\eta}\right)\quad(\eta\ne 0), \qquad A(0)=0. $$

The normalized density is $f(y;\eta)=\exp\{\eta y-A(\eta)\}$ on $[0,1]$, and zero outside that support. Its integral is $$ \int_0^1 f(y;\eta)\,dy =\exp\{-A(\eta)\}Z(\eta)=1. $$ At $\eta=0$, this is the uniform density on $[0,1]$, whose integral is also one. The separate value at zero is important: substituting $\eta=0$ into the quotient for $Z$ would give $0/0$, not the normalizer.

Worked Example: A Continuous Kernel on the Positive Half-Line

Consider support $y>0$ and kernel $\exp(\eta y)$, with $h(y)=1$. Determine for which values used in this calculation the kernel has a finite integral, and normalize it when $\eta<0$.

If $\eta<0$, direct integration gives $$ Z(\eta)=\int_0^\infty \exp(\eta y)\,dy =\left[\frac{\exp(\eta y)}{\eta}\right]_0^\infty =0-\frac{1}{\eta}=-\frac{1}{\eta}. $$ Because $\eta<0$, this value is positive and finite. Hence $$ A(\eta)=\log\left(-\frac{1}{\eta}\right)=-\log(-\eta). $$ The normalized density is therefore $f(y;\eta)=(-\eta)\exp(\eta y)$ for $y>0$. Its integral checks directly: $$ \int_0^\infty (-\eta)\exp(\eta y)\,dy =(-\eta)\left(-\frac{1}{\eta}\right)=1. $$

For $\eta=0$, the integral of the kernel is $\int_0^\infty 1\,dy=\infty$; for $\eta>0$, the integrand grows and the integral is also infinite. Neither case supplies a finite normalizer. This example shows why the normalization calculation must use the actual support and parameter value rather than just manipulate the exponential expression formally.

Two Consequences of the Normalization Equation

The log-partition function does more than record a normalizing constant. When it is differentiable, its derivatives summarize the distribution of $T(Y)$. The result below states explicit sufficient conditions for differentiating the normalizing integral. It applies to sums as well by treating a discrete distribution as integration with respect to a measure that assigns mass to each support point.

Theorem: Let $\mu$ be a measure on the response space, let $h\geq 0$ and $T$ be measurable, and define $$ Z(\eta)=\int h(y)\exp\{\eta T(y)\}\,\mu(dy). $$ Suppose $\eta$ lies in an open interval $I$, and $Z(\xi)$ is finite and positive for every $\xi\in I$. For each $\eta\in I$, $$ A'(\eta)=E_\eta[T(Y)], \qquad A''(\eta)=\operatorname{Var}_\eta(T(Y)), $$ where the distribution with parameter $\eta$ has density $h(y)\exp\{\eta T(y)\}/Z(\eta)$ with respect to $\mu$.

Proof. Fix $\eta\in I$. Since $I$ is open, choose $\delta>0$ such that $[\eta-\delta,\eta+\delta]\subset I$. For $k=1,2$, the function $u\mapsto u^k\exp(-\delta u)$ is bounded for $u\geq 0$. Consequently, for some finite constant $C_k$, the integrand $|T(y)|^k\exp\{\eta T(y)\}$ is bounded by $C_k\exp\{(\eta+\delta)T(y)\}$ when $T(y)\geq0$, and by $C_k\exp\{(\eta-\delta)T(y)\}$ when $T(y)<0$. After multiplying by $h(y)$, these bounds are integrable because $Z(\eta+\delta)$ and $Z(\eta-\delta)$ are finite.

These bounds also dominate the difference quotients needed to differentiate $\exp\{\eta T(y)\}$ once and twice in a sufficiently small neighborhood of $\eta$, by the mean value theorem. The dominated convergence theorem therefore permits differentiation under the integral, giving $$ Z'(\eta)=\int T(y)h(y)\exp\{\eta T(y)\}\,\mu(dy), \qquad Z''(\eta)=\int T(y)^2h(y)\exp\{\eta T(y)\}\,\mu(dy). $$ Since $Z(\eta)>0$, differentiating $A(\eta)=\log Z(\eta)$ yields $$ A'(\eta)=\frac{Z'(\eta)}{Z(\eta)}=E_\eta[T(Y)] $$ and $$ A''(\eta)=\frac{Z''(\eta)}{Z(\eta)} -\left(\frac{Z'(\eta)}{Z(\eta)}\right)^2 =E_\eta[T(Y)^2]-E_\eta[T(Y)]^2 =\operatorname{Var}_\eta(T(Y)). $$ This proves both identities. $\square$

The variance identity implies $A''(\eta)\geq0$ wherever the theorem applies, so $A$ is convex there. If $T(Y)$ is not constant with probability one under the normalized distribution, its variance is positive and $A''(\eta)>0$. These conclusions come from the normalization integral; they do not require guessing a shape for $A$.

Worked Example: Checking the Derivative Identity in the Three-Point Model

Return to the support $\{0,1,2\}$. Write $Z(\eta)=1+\exp(\eta)+\exp(2\eta)$. Differentiation gives $$ A'(\eta)=\frac{\exp(\eta)+2\exp(2\eta)} {1+\exp(\eta)+\exp(2\eta)}. $$ The probabilities from the first example assign weights $1,\exp(\eta),\exp(2\eta)$ to $0,1,2$, respectively. Their weighted mean is $$ E_\eta[Y] =0\cdot\frac{1}{Z(\eta)} +1\cdot\frac{\exp(\eta)}{Z(\eta)} +2\cdot\frac{\exp(2\eta)}{Z(\eta)} =\frac{\exp(\eta)+2\exp(2\eta)}{Z(\eta)} =A'(\eta). $$ At $\eta=0$, this mean is $(0+1+2)/3=1$, agreeing with $A'(0)=1$. Differentiating again, or using the variance identity, gives $A''(0)=\operatorname{Var}(Y)=((0-1)^2+(1-1)^2+(2-1)^2)/3=2/3$. Thus both derivative identities can be checked from the probabilities themselves.

Common Mistake: Normalizing the Wrong Quantity

A frequent error is to treat the kernel as if it were already a probability distribution. For example, on $[0,1]$, $\exp(\eta y)$ does not generally integrate to one: its integral is $(\exp(\eta)-1)/\eta$ when $\eta\ne0$. Writing $f(y;\eta)=\exp(\eta y)$ as a density therefore gives total mass $Z(\eta)$, not one. The correction is to divide the kernel by $Z(\eta)$, equivalently subtract $A(\eta)=\log Z(\eta)$ in the exponent.

A related mistake is to normalize over a convenient support instead of the model's stated support. The positive-half-line example has a finite integral only for $\eta<0$; replacing its support with a bounded interval would produce a different normalizer and a different model. Always specify the support first, sum or integrate the kernel over that support, and only then take the logarithm.

Key takeaway: Given a fixed kernel and support, calculate $Z(\eta)$ by summing or integrating the kernel, and set $A(\eta)=\log Z(\eta)$ whenever $Z(\eta)$ is finite and positive. Under the stated differentiability conditions, $A'(\eta)$ is the mean of $T(Y)$ and $A''(\eta)$ is its variance.

Check Your Understanding

Use the normalization equation and the derivative identities to answer these questions.

For support $\{0,1\}$ and kernel $\exp(\eta y)$, write $Z(\eta)$, $A(\eta)$, and the two normalized probabilities.
For the kernel $\exp(\eta y)$ on $[0,2]$, set up the integral for $Z(\eta)$. What is $Z(0)$, and why must it be evaluated separately from a quotient formula that divides by $\eta$?
For the kernel $\exp(\eta y)$ on $y>0$, explain why $\eta=0$ cannot be normalized.
State the probabilistic meanings of $A'(\eta)$ and $A''(\eta)$, including the statistic whose mean and variance they describe.
In the three-point example, why does the fact that $A''(\eta)$ is a variance guarantee that it cannot be negative?