Deriving the Mean From the Log-Partition Function

Why a Normalizing Constant Contains a Mean

In an exponential-family model, the log-partition function is introduced so that the probabilities or densities sum or integrate to one. It can seem like a technical adjustment to make a model valid. But its derivative has a direct statistical meaning: under regularity conditions, it is the expected value of the sufficient statistic. This identity lets us read a model's mean from its normalizing function, without separately calculating the expectation from scratch.

The result applies to the conditional response distribution at a fixed value of its model parameters. It is not an approximation and does not require a large sample. The essential step is differentiating the normalization equation with respect to the natural parameter.

Definition: Consider a one-parameter exponential family written in canonical form as $$ f(y;\eta)=h(y)\exp\{\eta T(y)-A(\eta)\}. $$ Here $T(y)$ is the sufficient statistic, $\eta$ is its natural parameter, and $A(\eta)$ is the log-partition function. The function $h(y)$ does not depend on $\eta$. The identity developed here is $A'(\eta)=E_\eta[T(Y)]$.

The notation covers a probability mass function or a density. For a discrete response, normalization means summing over its possible values; for a continuous response, it means integrating over its support. In either case, the support must not change with $\eta$ for the elementary differentiation argument below. The parameter must also lie in an open range where the normalizing expression is finite and can be differentiated under the sum or integral.

In the course's exponential-family setup, the same identity may be written with a fixed dispersion factor. For example, if the exponent contains $\{\eta T(y)-b(\eta)\}/a(\phi)$, with $a(\phi)$ fixed as $\eta$ varies, the derivative of the log density is $\{T(y)-b'(\eta)\}/a(\phi)$, and normalization implies $b'(\eta)=E[T(Y)]$. The canonical form above keeps the main argument uncluttered.

Deriving the Identity

Begin with the fact that the model is normalized for every admissible $\eta$. In the continuous case this says that the integral of $f(y;\eta)$ over the response support is one. Differentiate both sides with respect to $\eta$. Under the conditions stated above, differentiation can be passed inside the integral. The derivative of the density is the density multiplied by the derivative of its log:

\frac{\partial}{\partial\eta} f(y;\eta) =f(y;\eta)\{T(y)-A'(\eta)\}.

The derivative of the constant one is zero, so integrating this expression gives

0=\int f(y;\eta)\{T(y)-A'(\eta)\}\,dy =E_\eta[T(Y)]-A'(\eta).

Rearranging proves the identity. For a discrete response, replace the integral by a sum; the same steps apply. This is a short derivation, but its assumptions matter: normalization alone is not enough to justify interchanging differentiation and integration unless the model permits that interchange.

Theorem: If $f(y;\eta)=h(y)\exp\{\eta T(y)-A(\eta)\}$ is normalized on a support that does not depend on $\eta$, and differentiation can be interchanged with the sum or integral defining normalization, then $$ A'(\eta)=E_\eta[T(Y)]. $$

An equivalent justification uses the normalizing integral or sum directly. Write $Z(\eta)=\int h(y)e^{\eta T(y)}dy$, or its discrete analogue. Then $A(\eta)=\log Z(\eta)$. Differentiating the logarithm gives $A'(\eta)=Z'(\eta)/Z(\eta)$. The numerator is the integral of $T(y)h(y)e^{\eta T(y)}$, while dividing by $Z(\eta)$ turns that integral into the model expectation of $T(Y)$. This version makes clear why the log of the normalizer, rather than the normalizer itself, has the expectation as its derivative.

Worked Examples

Worked Example: Bernoulli Response

Let $Y$ be Bernoulli with success probability $p$, where $0<p<1$. Its probability mass function is $P(Y=y)=p^y(1-p)^{1-y}$, for $y\in\{0,1\}$. Write it in canonical form and verify the derivative identity.

Rearrange the log probability: $$ \log P(Y=y)=y\log\left(\frac{p}{1-p}\right)+\log(1-p). $$ Set $\eta=\log\{p/(1-p)\}$. Solving gives $p=e^\eta/(1+e^\eta)$, and $\log(1-p)=-\log(1+e^\eta)$. Thus $T(y)=y$, $h(y)=1$, and $A(\eta)=\log(1+e^\eta)$.

Differentiate the log-partition function: $$ A'(\eta)=\frac{e^\eta}{1+e^\eta}. $$ Since $p=e^\eta/(1+e^\eta)$ and $T(Y)=Y$, the expected sufficient statistic is $E[Y]=P(Y=1)=p$. Therefore $A'(\eta)=E[T(Y)]=p$, as claimed.

For a numerical check, take $p=0.7$. Then $\eta=\log(7/3)$, so $e^\eta=7/3$ and $$ A'(\eta)=\frac{7/3}{1+7/3}=\frac{7}{10}=0.7=E[Y]. $$ Both quantities are in units of the Bernoulli response, whose values are zero and one.

Worked Example: A Three-Value Response

Consider a hypothetical discrete family with $Y\in\{0,1,2\}$, $T(Y)=Y$, and base weights $h(0)=1$, $h(1)=2$, and $h(2)=1$. These weights need not themselves sum to one; the log-partition function normalizes them after tilting by $\eta$.

The normalizing sum is $$ Z(\eta)=1+2e^\eta+e^{2\eta}=(1+e^\eta)^2, $$ so $A(\eta)=\log Z(\eta)=2\log(1+e^\eta)$. Differentiation gives $$ A'(\eta)=\frac{2e^\eta}{1+e^\eta}. $$ At $\eta=\log 2$, the three unnormalized weights are $1$, $4$, and $4$, so the normalized probabilities are $1/9$, $4/9$, and $4/9$. Directly, $$ E[Y]=0\left(\frac19\right)+1\left(\frac49\right)+2\left(\frac49\right)=\frac{12}{9}=\frac43. $$ The derivative gives $A'(\log 2)=2(2)/(1+2)=4/3$, in agreement.

Worked Example: A Symmetric Two-Value Response

Suppose a hypothetical response takes values $-1$ and $1$, with $T(Y)=Y$ and equal base weights $h(-1)=h(1)=1$. This is another canonical family on fixed support.

Its normalizing sum is $Z(\eta)=e^{-\eta}+e^\eta$, hence $A(\eta)=\log(e^{-\eta}+e^\eta)$. Differentiating, $$ A'(\eta)=\frac{e^\eta-e^{-\eta}}{e^\eta+e^{-\eta}}. $$ At $\eta=\log 3$, the unnormalized weights for $-1$ and $1$ are $1/3$ and $3$. The probabilities are therefore $1/10$ and $9/10$, and $$ E[Y]=(-1)\left(\frac{1}{10}\right)+(1)\left(\frac{9}{10}\right)=0.8. $$ The derivative is $(3-1/3)/(3+1/3)=0.8$ as well. The identity concerns the expected value of $T(Y)$, which here can be negative; it does not require the statistic to be an indicator or a count.

Using and Checking the Result

In a GLM, the conditional response distribution is indexed by parameters that may depend on covariates. The identity applies at each fixed covariate value: it relates the log-partition derivative at that observation's natural parameter to the conditional expectation of its sufficient statistic. When $T(Y)=Y$, this gives the conditional mean of the response. If $T(Y)$ is a different function, the result gives the expectation of that function, not automatically the mean of $Y$.

There are several useful checks when applying the formula:

Check the parameter. Differentiate with respect to the natural parameter $\eta$, not an arbitrary reparameterization. If $\eta$ is itself a function of another parameter $\beta$, the chain rule gives $\frac{dA}{d\beta}=A'(\eta)\frac{d\eta}{d\beta}$, which is not generally just $E[T(Y)]$.
Check the sufficient statistic. The right side is $E[T(Y)]$. It equals $E[Y]$ only when the statistic is $T(Y)=Y$.
Check normalization and support. Verify that $A$ actually normalizes the stated model and that the support does not vary with $\eta$, or use a result that justifies the more complicated case.
Check the parameter domain. The normalizing sum or integral must be finite at the parameter value, and differentiation must be valid there.

Directly summing or integrating $T(y)f(y;\eta)$ is a sound way to check a result, as the examples show. Differentiating $A$ is often shorter when $A$ is known. A numerical finite-difference approximation can be a diagnostic check, but it is not a proof and may be inaccurate if the step size is poorly chosen. Nor should $A'(\eta)$ be confused with the sample average of observed responses: the former is a model expectation at a parameter value; the latter is calculated from a particular sample.

Common Pitfalls and Scope

A frequent error is to differentiate only the term $\eta T(y)$ in the exponent and conclude that the derivative is $T(y)$. That is the derivative of part of the log density, not the expected statistic. The full derivative is $T(y)-A'(\eta)$; setting its model expectation to zero is what yields the identity. Another mistake is to omit the distinction between a statistic and the response itself. A family with $T(y)=y^2$, for example, yields $A'(\eta)=E[Y^2]$, not $E[Y]$.

A further caution is that parameter-dependent support can invalidate the simple differentiation argument. Boundary terms may appear when differentiating an integral whose limits depend on the parameter. Likewise, a parameter value at the boundary of the domain may not permit ordinary differentiation. In such cases, the identity must be checked using the specific model rather than applied mechanically.

The result is useful because it converts a normalizing function into a model-based expectation, and it supplies a consistency check on exponential-family algebra. It does not, by itself, establish that a chosen response distribution is appropriate for data, nor does it make the model's conditional mean a causal effect. The next step in the log-partition argument concerns higher derivatives; here the conclusion is limited to the first derivative and the expected sufficient statistic.

Key takeaway: For a regular canonical exponential family, differentiating the log-partition function with respect to its natural parameter gives $E[T(Y)]$. Identify the statistic, confirm the parameterization and regularity conditions, and only then interpret the derivative as a response mean when $T(Y)=Y$.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the canonical form $f(y;\eta)=h(y)\exp\{\eta T(y)-A(\eta)\}$, what quantity does $A'(\eta)$ equal?
For a Bernoulli response with $p=0.4$, calculate the natural parameter and verify that the derivative of its log-partition function is $0.4$.
For the three-value example, what are the probabilities at $\eta=0$, and what is $E[Y]$? Check the value against $A'(0)$.
Why is differentiating with respect to a parameter $\beta$ generally not the same as differentiating with respect to $\eta$?

Solutions. (1) $A'(\eta)=E_\eta[T(Y)]$, provided normalization can be differentiated under the sum or integral. (2) $\eta=\log(0.4/0.6)=\log(2/3)$, so $e^\eta=2/3$ and $A'(\eta)=(2/3)/(1+2/3)=0.4$. (3) At $\eta=0$, the weights $1,2,1$ give probabilities $1/4,1/2,1/4$; thus $E[Y]=0(1/4)+1(1/2)+2(1/4)=1$, and $A'(0)=2/(1+1)=1$. (4) If $\eta=\eta(\beta)$, the chain rule gives $dA/d\beta=A'(\eta)\,d\eta/d\beta$; the extra factor need not equal one.