Log-Partition Gradients Give Mean Parameters

Why the Log-Partition Gradient Is a Mean

In a canonical exponential family, the parameter \(\eta\) enters the density through the term \(\eta^{\mathsf T}T(x)\), where \(T(X)\) is a vector of statistics. The log-partition function \(A(\eta)\) is chosen to make the density integrate to one. That normalization condition does more than define the family: when \(A\) can be differentiated, its gradient records the expected values of the statistics in \(T(X)\).

The key issue is not merely formal differentiation. The integral or sum being differentiated is infinite in many models, so the parameter must be in a region where differentiation under the integral sign is justified. We establish that condition first, then calculate the mean vector in several examples.

Canonical Form and the Differentiation Condition

Definition: Let \(\mu\) be a nonzero measure on the sample space, and let \(T(x)=(T_1(x),\ldots,T_d(x))^{\mathsf T}\) be a measurable, finite-valued vector. Define \(Z(\eta)=\int \exp\{\eta^{\mathsf T}T(x)\}\,\mu(dx)\) and the natural parameter space \(\Theta=\{\eta\in\mathbb R^d:0<Z(\eta)<\infty\}\). For \(\eta\in\Theta\), the canonical family has density relative to \(\mu\) \(f_\eta(x)=\exp\{\eta^{\mathsf T}T(x)-A(\eta)\}\), where \(A(\eta)=\log Z(\eta)\).

The measure \(\mu\) includes any carrier weight in the representation. For a discrete family it may assign non-unit mass to sample points; it need not be ordinary counting measure. This matters because changing \(\mu\) changes the normalizing integral and may change the distribution itself.

The theorem below applies at an interior point of the natural parameter space. Interior is important: it ensures that the integral remains finite not just at \(\eta\), but in a full neighborhood of \(\eta\). This neighborhood supplies an integrable bound for the derivatives.

Theorem: Suppose \(\eta\) is in the interior \(\Theta^\circ\) of the natural parameter space. Then each component of \(T(X)\) is integrable under \(f_\eta\), \(A\) is continuously differentiable in a neighborhood of \(\eta\), and \(\nabla A(\eta)=E_\eta[T(X)]\).

Proof by Differentiating the Normalizing Integral

Write \(t=T(x)\). Since \(\eta\in\Theta^\circ\), choose \(\delta>0\) small enough that every vertex \(\eta+2\delta s\), for \(s\in\{-1,1\}^d\), belongs to \(\Theta\). We may also take \(\delta\) small enough that the cube centered at \(\eta\) with coordinate radius \(\delta\) lies inside \(\Theta^\circ\).

For any \(\theta\) in that smaller cube, \(\theta^{\mathsf T}t\leq \eta^{\mathsf T}t+\delta\sum_{k=1}^d|t_k|\). Also, for some finite constant \(C_\delta\), \( |t_j|\leq C_\delta\exp(\delta|t_j|)\). Consequently,

|t_j|\exp\{\theta^{\mathsf T}t\} \leq C_\delta\exp\left\{\eta^{\mathsf T}t+2\delta\sum_{k=1}^d|t_k|\right\} \leq C_\delta\sum_{s\in\{-1,1\}^d} \exp\{(\eta+2\delta s)^{\mathsf T}t\}.

The last expression is integrable with respect to \(\mu\): it is a finite sum, and each vertex parameter is in \(\Theta\). Thus the displayed bound is an integrable dominating function, locally uniform in \(\theta\).

For the \(j\)th coordinate, consider the difference quotient of \(Z\). Its integrand converges pointwise to \(T_j(x)\exp\{\eta^{\mathsf T}T(x)\}\). For sufficiently small nonzero \(h\),

\left| \exp\{\eta^{\mathsf T}t\} \frac{\exp\{h t_j\}-1}{h} \right| \leq |t_j|\exp\{\eta^{\mathsf T}t+|h||t_j|\}.

Taking \(|h|\leq\delta\) makes the right-hand side bounded by the same type of integrable bound just established, with the cube or \(\delta\) reduced if necessary. The dominated convergence theorem therefore permits differentiation under the integral:

\frac{\partial Z(\eta)}{\partial\eta_j} = \int T_j(x)\exp\{\eta^{\mathsf T}T(x)\}\,\mu(dx).

The bound also proves absolute integrability of the derivative integrand. Applying dominated convergence to these derivative integrands as the parameter varies locally shows that the partial derivatives of \(Z\) are continuous. Hence \(Z\) is continuously differentiable there. Because \(Z(\eta)>0\), the chain rule gives

\frac{\partial A(\eta)}{\partial\eta_j} =\frac{1}{Z(\eta)}\frac{\partial Z(\eta)}{\partial\eta_j} =\int T_j(x)\frac{\exp\{\eta^{\mathsf T}T(x)\}}{Z(\eta)}\,\mu(dx) =E_\eta[T_j(X)].

This holds for every component \(j\), proving the vector identity. \(\square\)

Worked Examples

Worked Example: The Mean of a Bernoulli Variable

Let \(X\in\{0,1\}\), use counting measure on \(\{0,1\}\), and take \(T(X)=X\). The normalizing integral is a sum:

Z(\eta)=\sum_{x=0}^1 e^{\eta x}=1+e^\eta, \qquad A(\eta)=\log(1+e^\eta).

Differentiating gives

A'(\eta)=\frac{e^\eta}{1+e^\eta}.

The family assigns probability \(1/Z(\eta)\) to \(X=0\) and \(e^\eta/Z(\eta)\) to \(X=1\). Thus \(P_\eta(X=1)=e^\eta/(1+e^\eta)\), and direct expectation gives \(E_\eta[X]=0\cdot P_\eta(X=0)+1\cdot P_\eta(X=1)=e^\eta/(1+e^\eta)=A'(\eta)\).

For a numerical check, set \(p=0.3\), so \(\eta=\log(p/(1-p))=\log(3/7)\). Then \(e^\eta=3/7\), and \(A'(\eta)=(3/7)/(1+3/7)=(3/7)/(10/7)=0.3\).

Worked Example: The Mean of a Poisson Variable

Let \(X\) take values \(0,1,2,\ldots\), take \(T(X)=X\), and let \(\mu\) assign mass \(1/x!\) to the point \(x\). In particular, the carrier is not counting measure. The normalizer is

Z(\eta)=\sum_{x=0}^{\infty}\frac{e^{\eta x}}{x!} =\sum_{x=0}^{\infty}\frac{(e^\eta)^x}{x!} =\exp(e^\eta), \qquad A(\eta)=e^\eta.

The sum converges for every real \(\eta\), so the natural parameter space is all of \(\mathbb R\). Differentiating gives \(A'(\eta)=e^\eta\). The probability mass function confirms the interpretation:

P_\eta(X=x) =\frac{e^{\eta x}}{x!\exp(e^\eta)} =\exp(-e^\eta)\frac{(e^\eta)^x}{x!}.

This is the Poisson mass function with rate \(\lambda=e^\eta\), so its mean is \(e^\eta=A'(\eta)\). For example, if \(\eta=\log 4\), then \(Z(\eta)=e^4\), \(A'(\eta)=4\), and the distribution is Poisson with rate \(4\).

Worked Example: A Mean Vector on Three Support Points

Consider a family supported on \((0,0)\), \((1,0)\), and \((0,1)\), with counting measure and \(T(x)=x\). For \(\eta=(\eta_1,\eta_2)^{\mathsf T}\), the three unnormalized weights are \(1\), \(e^{\eta_1}\), and \(e^{\eta_2}\). Therefore

Z(\eta)=1+e^{\eta_1}+e^{\eta_2}, \qquad A(\eta)=\log(1+e^{\eta_1}+e^{\eta_2}).

Differentiate each coordinate:

\nabla A(\eta) = \begin{pmatrix} \dfrac{e^{\eta_1}}{1+e^{\eta_1}+e^{\eta_2}}\\[6pt] \dfrac{e^{\eta_2}}{1+e^{\eta_1}+e^{\eta_2}} \end{pmatrix}.

Take \(\eta_1=0\) and \(\eta_2=\log 2\). Then \(Z=1+1+2=4\), so the three support points have probabilities \(1/4\), \(1/4\), and \(1/2\), respectively. The first coordinate of \(T(X)\) is one only at \((1,0)\), hence its mean is \(1/4\). The second coordinate is one only at \((0,1)\), hence its mean is \(1/2\). The gradient gives exactly \((1/4,1/2)^{\mathsf T}\).

How to Use the Identity—and a Common Pitfall

The identity \(\nabla A(\eta)=E_\eta[T(X)]\) turns a potentially difficult expectation into a derivative of a normalizing function. It is especially useful when \(Z(\eta)\) can be summed or integrated directly, as in the examples above. For a vector statistic, each coordinate derivative gives the expectation of the corresponding statistic; the result is a mean vector, not a single scalar unless \(d=1\).

A common mistake is to differentiate a formal expression for \(Z\) without checking either the carrier measure or whether the parameter is in the interior of the natural parameter space. For instance, using counting measure on the nonnegative integers with \(T(x)=x\) would give \(Z(\eta)=\sum_{x=0}^{\infty}e^{\eta x}\), which diverges for \(\eta\geq0\) and, when finite, is a geometric-series normalizer—not the Poisson normalizer. The Poisson representation above requires the carrier mass \(1/x!\). A second caution is that at a boundary point of the natural parameter space the neighborhood argument in the proof may fail; the theorem does not assert differentiability there.

Key takeaway: At an interior natural parameter value, with a fixed carrier measure and finite normalizer in a neighborhood, differentiating the log-partition function is justified and gives the mean of the canonical statistic: \(\nabla A(\eta)=E_\eta[T(X)]\).

Check Your Understanding

Use the definitions and calculations above to answer these questions.

For the Bernoulli family with \(\eta=\log 2\), calculate \(A'(\eta)\) and identify the corresponding success probability.
In the Poisson example, what normalizing sum results from using counting measure instead of carrier mass \(1/x!\), and why does it not produce the Poisson family?
For the three-point family, calculate \(\nabla A(\eta)\) when \(\eta_1=\log 3\) and \(\eta_2=0\).
In the theorem, why is it useful to assume that \(\eta\) is in the interior of the natural parameter space?
If \(T(X)\) has three coordinates, how many mean identities are given by \(\nabla A(\eta)=E_\eta[T(X)]\)?