Deriving Mean And Variance From The Log-Partition Function

Why Differentiate the Normalizing Identity?

In an exponential-family model, the normalizing term is chosen so that the probabilities or density integrate to one. That requirement is more than a technical detail: differentiating it reveals how the distribution's mean and variance change with the natural parameter. The same calculation applies across response families, provided differentiation under the integral or sum is justified.

For generalized linear model work, it is particularly important to keep the dispersion scale visible. In the common exponential-dispersion form, the first derivative of the cumulant function gives the mean, while its second derivative must be multiplied by the dispersion scale to give the variance.

Exponential-Dispersion Form and Notation

Fix a dispersion parameter \(\phi\), and let \(a(\phi)>0\). Consider a response \(Y\) with density or probability mass function, relative to a measure \(\nu\), of the form

Here \(\theta\) is the natural parameter in the usual exponential-dispersion convention, \(b(\theta)\) is the cumulant (or unscaled log-partition) function, and \(c(y,\phi)\) contains terms not involving \(\theta\). The parameter \(\theta\) ranges over an open interval on which the model is normalized. For each fixed \(\phi\), normalization means

For a discrete response, the integral denotes a sum. In this tutorial, \(b\) is called the cumulant function, following the usual exponential-dispersion convention. Some authors reserve “log-partition function” for the entire normalizing term \(b(\theta)/a(\phi)\). The parameterization used for differentiation determines whether the scale appears explicitly in the derivative identities; both conventions are related below.

The following regularity conditions suffice for the calculations. The support and measure \(\nu\) do not depend on \(\theta\); \(b\) is twice continuously differentiable on the open parameter interval; and, in a neighborhood of each \(\theta\), the integrand and its first two \(\theta\)-derivatives have integrable dominating functions. A convenient sufficient condition is that, on every sufficiently small closed interval \(K\) around \(\theta\),

is integrable with respect to \(\nu\). Since \(b'\) and \(b''\) are bounded on such an interval, this condition dominates the integrand and the derivatives needed below. It permits differentiation under the integral sign by the dominated convergence theorem. Parameter-dependent support requires separate care and is excluded from these identities as stated.

The First Derivative Gives the Mean

Write \(q_\theta(y)=f(y;\theta,\phi)\), holding \(\phi\) fixed. Direct differentiation of the log density gives

This derivative is the score with respect to \(\theta\). The normalization identity implies that its expectation is zero; the resulting equation identifies the mean.

Proof. Because \(q_\theta\) integrates to one for every \(\theta\), its derivative integrates to zero. Differentiation under the integral is permitted by the domination condition. Using the score expression,

Since \(a(\phi)>0\), multiplying by \(a(\phi)\) gives \(E[Y]=b'(\theta)\). \(\square\)

The Second Derivative Gives the Variance

The second derivative follows from differentiating normalization again. Differentiating the density itself, rather than only its logarithm, makes the squared deviation term visible. The first derivative of the density is

Differentiating once more yields

Proof. The normalization integral is identically one, so its second derivative is zero. The second-derivative domination condition allows the derivative to pass inside the integral. Substituting the expression above gives

The first result gives \(b'(\theta)=E[Y]\). Thus \(E[(Y-b'(\theta))^2]=\operatorname{Var}(Y)\), and the equation becomes

Multiplication by \(a(\phi)^2\) gives \(\operatorname{Var}(Y)=a(\phi)b''(\theta)\). \(\square\)

Which Log-Partition Derivative?

The scale factor can appear in different places depending on how the natural parameter and log-partition function are defined. In the exponential-dispersion convention above, the coefficient of \(y\) is \(\theta/a(\phi)\), while \(b(\theta)\) is the unscaled cumulant function. That is why its derivatives satisfy \(b'(\theta)=E[Y]\) and \(a(\phi)b''(\theta)=\operatorname{Var}(Y)\).

For fixed \(\phi\), define the coefficient parameter \(\eta=\theta/a(\phi)\) and the corresponding log-partition function

Then the model can be written with the parameter-dependent terms \(y\eta-A_\phi(\eta)\). Applying the chain rule gives

Thus, in the coefficient parameter \(\eta\), the first and second derivatives of the log-partition function are the mean and variance directly. In the usual dispersion convention, the same facts are expressed through \(b\), with the scale multiplying its second derivative. These are equivalent descriptions, not competing formulas.

Worked Examples

Common Mistake: Losing the Dispersion Scale

A frequent error is to read \(b''(\theta)\) as the response variance in every exponential-dispersion model. That is correct only when \(a(\phi)=1\). In the Gaussian example, \(b''(\mu)=1\), but the response variance is \(9\) when \(\sigma^2=9\). The correct calculation is \(a(\phi)b''(\theta)=9\cdot1=9\).

A related notational error is to differentiate \(b(\theta)/a(\phi)\) with respect to \(\theta\) and claim the result is the mean. Its derivative is \(b'(\theta)/a(\phi)\), not \(b'(\theta)\). That is the derivative of the scaled normalizing term in the \(\theta\) coordinate; the coefficient parameter \(\eta=\theta/a(\phi)\) instead gives the direct log-partition identities \(A_\phi'(\eta)=E[Y]\) and \(A_\phi''(\eta)=\operatorname{Var}(Y)\).