Deriving the Variance From the Log-Partition Function

Why the Curvature of the Log-Partition Function Matters

The earlier material on deriving the mean showed that the first derivative of a log-partition function gives the expected sufficient statistic. The second derivative has a similarly useful interpretation: it gives the variance of that statistic. This is more than a convenient shortcut. It connects the way the normalizing constant changes with the natural parameter to the spread of the modeled data.

The identity is especially useful when a distribution is written in exponential-family form. Rather than calculating moments from scratch for each response distribution, one can differentiate the same function that ensures the probabilities or density integrate to one. The result applies to the sufficient statistic in the exponent; it is the response variance only when that statistic is the response itself.

Definition: Consider a one-parameter exponential family written as \(f(y\mid\eta)=h(y)\exp\{\eta T(y)-A(\eta)\}\), with respect to a fixed measure. Here \(\eta\) is the natural parameter, \(T(Y)\) is the sufficient statistic, and \(A(\eta)\) is the log-partition function. The function \(A\) normalizes the distribution.

For the identity below, the support and base measure are fixed as \(\eta\) varies, the normalizing integral or sum is finite in a neighborhood of the parameter value, and differentiation can be passed under that integral or sum. These regularity conditions ensure the derivatives used in the calculation exist and are valid. In the examples, the distributions satisfy these conditions at the parameter values considered.

The Variance Identity

Write the normalizing constant before taking its logarithm as \(Z(\eta)=\int h(y)\exp\{\eta T(y)\}\,d\nu(y)\), where \(\nu\) denotes the reference measure. In a discrete model, the integral is a sum. Since \(A(\eta)=\log Z(\eta)\), differentiating \(Z\) reveals the moments of \(T(Y)\).

Differentiate the normalizer once.
Under the stated regularity conditions, \(Z'(\eta)=\int T(y)h(y)\exp\{\eta T(y)\}\,d\nu(y)\). Dividing by \(Z(\eta)\) gives the expectation under the normalized density.

Differentiate the log-normalizer.
Because \(A(\eta)=\log Z(\eta)\), the chain rule gives \(A'(\eta)=Z'(\eta)/Z(\eta)=E_\eta[T(Y)]\).

Differentiate once more.
Now \(Z''(\eta)=\int T(y)^2h(y)\exp\{\eta T(y)\}\,d\nu(y)\), so \(Z''(\eta)/Z(\eta)=E_\eta[T(Y)^2]\). Applying the quotient rule to \(A'=Z'/Z\) gives the variance.

A''(\eta) =\frac{Z''(\eta)}{Z(\eta)} -\left(\frac{Z'(\eta)}{Z(\eta)}\right)^2 =E_\eta[T(Y)^2]-\{E_\eta[T(Y)]\}^2 =\operatorname{Var}_\eta\{T(Y)\}.

Theorem: For a regular one-parameter exponential family in the form \(f(y\mid\eta)=h(y)\exp\{\eta T(y)-A(\eta)\}\), the first derivative satisfies \(A'(\eta)=E_\eta[T(Y)]\), and the second derivative satisfies \(A''(\eta)=\operatorname{Var}_\eta\{T(Y)\}\). In particular, \(A''(\eta)\geq 0\), so the log-partition function is convex wherever these derivatives exist.

The second derivative is with respect to the natural parameter \(\eta\), not with respect to the mean. Also, the theorem directly identifies the variance of \(T(Y)\). If \(T(Y)=Y\), it is the response variance; if \(T(Y)\) is a transformation or summary of the response, it is the variance of that transformed quantity instead.

Worked Examples

Worked Example: Bernoulli Response

Suppose, hypothetically, a single binary outcome \(Y\) records whether a participant experiences a specified event, with \(Y=1\) for an event and \(Y=0\) otherwise. Let the event probability be \(p=0.60\). In the Bernoulli canonical form, \(T(Y)=Y\), the natural parameter is \(\eta=\log\{p/(1-p)\}\), and \(A(\eta)=\log(1+e^\eta)\). The support is fixed at \(\{0,1\}\), and the finite normalizing sum is differentiable for every finite \(\eta\).

For these hypothetical data-generating parameters, \(\eta=\log(0.60/0.40)=\log(1.5)\approx0.4055\). Differentiate the log-partition function:

A'(\eta)=\frac{e^\eta}{1+e^\eta}, \qquad A''(\eta)=\frac{e^\eta}{(1+e^\eta)^2}.

Substitute \(e^\eta=1.5\):

A''(\eta)=\frac{1.5}{(1+1.5)^2} =\frac{1.5}{6.25} =0.2400.

Here \(A'(\eta)=1.5/2.5=0.60\), as expected, and \(T(Y)=Y\), so \(A''(\eta)=\operatorname{Var}(Y)=0.2400\) in squared binary-response units. Directly, a Bernoulli variable has variance \(p(1-p)=0.60(0.40)=0.2400\), confirming the result.

Worked Example: Poisson Response

Consider a hypothetical count \(Y\) of events in a fixed observation interval, modeled as Poisson with rate \(\lambda=2\) events per interval. Its canonical exponential-family form has \(T(Y)=Y\), \(\eta=\log\lambda\), and \(A(\eta)=e^\eta\). The count support is the fixed set of nonnegative integers, and the Poisson normalizer is finite for every finite \(\eta\).

The derivatives are

A'(\eta)=e^\eta, \qquad A''(\eta)=e^\eta.

For \(\lambda=2\), the natural parameter is \(\eta=\log 2\), so \(e^\eta=2\). Therefore,

A''(\log 2)=e^{\log 2}=2.

The first derivative gives a mean of 2 events per interval. Because the sufficient statistic is the response, the second derivative gives \(\operatorname{Var}(Y)=2\) squared event-count units. This agrees with the direct Poisson moment result that the variance equals the rate.

Worked Example: When the Sufficient Statistic Is Not the Response

For a hypothetical finite-support model, let the response \(Y\) take values \(0,1,2\), with probability mass function \(P_\eta(Y=y)=\exp\{\eta y^2-A(\eta)\}\) for each of those values. Thus \(T(Y)=Y^2\), which is not the response \(Y\). The support is fixed, and the normalizing sum is finite for every finite \(\eta\). At \(\eta=0\), each response value has probability \(1/3\), since \(A(0)=\log(1+1+1)=\log 3\).

The log-partition function is

A(\eta)=\log(1+e^\eta+e^{4\eta}).

By the variance identity, \(A''(0)\) is the variance of \(T(Y)=Y^2\), not the variance of \(Y\). Calculate the needed moments directly at \(\eta=0\). The possible values of \(T(Y)\) are \(0,1,4\), each with probability \(1/3\), so

E[T(Y)]=\frac{0+1+4}{3}=\frac{5}{3}, \qquad E[T(Y)^2]=\frac{0^2+1^2+4^2}{3}=\frac{17}{3}.

Consequently,

A''(0)=\operatorname{Var}\{Y^2\} =\frac{17}{3}-\left(\frac{5}{3}\right)^2 =\frac{51-25}{9} =\frac{26}{9}\approx2.889.

For comparison, the response variance is \(E[Y^2]-\{E[Y]\}^2=5/3-1^2=2/3\), because \(E[Y]=(0+1+2)/3=1\). The two variances differ, which confirms why the theorem must be applied to the statistic appearing in the exponent.

How to Check a Derivative-Based Variance

A reliable calculation has two independent checks. First, verify that the exponent has been written as \(\eta T(y)-A(\eta)\), so the derivative is taken with respect to the same natural parameter that multiplies \(T(y)\). Second, when the response distribution is familiar or has a small support, calculate the moments directly and compare them with \(A''(\eta)\). Agreement checks both the algebra and the identification of the sufficient statistic.

The derivative must produce a nonnegative value, since variance cannot be negative. A negative second derivative usually signals an algebra or sign error, a mistaken parameterization, or a calculation outside the conditions under which the derivatives are valid. A zero value is possible in a degenerate case where \(T(Y)\) is constant; it is not a negative variance.

Approach	What it calculates	When it is useful
Differentiate \(A(\eta)\) twice	The variance of the sufficient statistic \(T(Y)\)	When the log-partition function is known and its derivatives are manageable
Calculate moments from the distribution	\(E[T(Y)^2]-\{E[T(Y)]\}^2\), or the response variance if using \(Y\)	As a direct calculation or an independent check, especially for small finite support

The derivative method is not a substitute for identifying the model correctly. It does not imply that the variance of \(Y\) equals \(A''(\eta)\) when \(T(Y)\ne Y\). Nor does it remove the need to check regularity: if support or integrability conditions prevent differentiating the normalizer in the required way, the derivation cannot simply be assumed to hold. This tutorial uses the unscaled canonical form displayed above; additional scale or dispersion conventions require care about which parameter and log-partition function are being differentiated.

Common Pitfalls

Using the wrong random quantity. The theorem gives \(\operatorname{Var}\{T(Y)\}\), not automatically \(\operatorname{Var}(Y)\). Identify \(T(Y)\) from the exponent before interpreting the result.
Differentiating with respect to a different parameter. The stated identity uses derivatives with respect to \(\eta\), the coefficient of \(T(Y)\). A derivative with respect to a mean parameter is a different calculation.
Dropping the normalization term. The log-partition function is what makes the distribution sum or integrate to one. Its curvature carries the variance information.
Treating a negative result as a possible variance. Since \(A''(\eta)=\operatorname{Var}\{T(Y)\}\), a negative value indicates an error or a failed condition, not a valid model variance.

Key takeaway: In the canonical one-parameter form, the first derivative of the log-partition function is the mean of the sufficient statistic and its second derivative is that statistic’s variance. Always check whether the statistic is the response before reporting a response variance.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For \(A(\eta)=\log(1+e^\eta)\), differentiate twice and state what \(A''(\eta)\) represents when \(T(Y)=Y\).
A Poisson response has \(\eta=\log 3\) and \(A(\eta)=e^\eta\). Find \(A'(\eta)\) and \(A''(\eta)\), with their interpretations.
In the finite-support example, why is \(A''(0)\) not the variance of \(Y\)? Give both variances.
Name two conditions used in the derivation that make differentiation of the normalizer legitimate.

Solutions. (1) \(A'(\eta)=e^\eta/(1+e^\eta)\) and \(A''(\eta)=e^\eta/(1+e^\eta)^2\). When \(T(Y)=Y\), the second derivative is the response variance. (2) \(A'(\log 3)=e^{\log 3}=3\), the mean count per interval; \(A''(\log 3)=3\), the variance in squared count units. (3) The exponent contains \(T(Y)=Y^2\), so \(A''(0)=\operatorname{Var}(Y^2)=26/9\), whereas \(\operatorname{Var}(Y)=2/3\). (4) For example, the support and base measure are fixed as \(\eta\) varies, and the normalizing integral or sum is finite in a neighborhood with differentiation permitted under it.