From a Log-Partition Function to Moments
A log-partition function is more than a device for writing a probability model compactly: its derivatives encode the response’s moments. This tutorial works through that calculation for Bernoulli, Poisson, normal, and gamma responses. The central bookkeeping point is that the second derivative gives the variance function, while a dispersion factor may still be needed to obtain the response variance.
Use the exponential-dispersion form
where \(b(\theta)\) is the log-partition function, \(\theta\) is the canonical parameter, and \(a(\phi)>0\) is the dispersion scaling factor. The support of \(Y\) does not depend on \(\theta\). In each calculation below, \(\theta\) is in the interior of its natural parameter space, and the normalizing integral or sum can be differentiated twice under the integral or sum. These conditions hold for the four families considered here at the stated parameter values.
Why the Derivatives Give the Moments
The moment calculation follows directly from normalization. It is useful to establish both the moment-generating-function identity and the resulting mean-variance formula; together they show exactly where the dispersion factor enters.
Derivation. Multiply the density by \(e^{tY}\). The part of the exponent involving \(Y\) becomes \(Y[\theta+a(\phi)t]/a(\phi)\). The remaining terms are unchanged, so the integral or sum is the normalizing constant at parameter \(\theta+a(\phi)t\), divided by the original normalizing constant. In the stated form, the normalizing constant for the part involving \(Y\) is \(\exp\{b(\theta)/a(\phi)\}\). Therefore,
Differentiating the logarithm of this expression at zero gives the mean and variance. The differentiations are valid on a neighborhood of zero because the shifted parameter remains in the natural parameter space there.
Derivation. Write \(K(t)=\log M_Y(t)\). The moment-generating-function identity gives
Since \(K'(0)=E[Y]\) and \(K''(0)=\operatorname{Var}(Y)\), evaluation at zero yields \(E[Y]=b'(\theta)\) and \(\operatorname{Var}(Y)=a(\phi)b''(\theta)\). Finally, substituting \(\mu=b'(\theta)\) expresses the second derivative as \(V(\mu)\), giving \(\operatorname{Var}(Y)=a(\phi)V(\mu)\). This last step requires expressing \(b''(\theta)\) in terms of \(\mu\), which is possible on the mean’s parameter domain. \(\square\)
Bernoulli and Poisson Responses
For Bernoulli and Poisson responses, \(a(\phi)=1\). Thus their variance functions are also their response variances. In both cases, the canonical parameter ranges over the real line, but the resulting means have family-specific domains.
Worked Example: Bernoulli Log-Partition Function
Let \(Y\) be Bernoulli with success probability \(p\). Its log-partition function is \(b(\theta)=\log(1+e^\theta)\), with \(\theta=\log\{p/(1-p)\}\) and \(a(\phi)=1\). Differentiate:
The first derivative is \(p\), so \(\mu=p\). Rewriting the second derivative in terms of \(p\) gives \(V(\mu)=p(1-p)=\mu(1-\mu)\). Because \(a(\phi)=1\), this is also the response variance.
For a numerical check, take \(p=0.3\). Then \(\theta=\log(0.3/0.7)\), \(b'(\theta)=0.3\), and \(b''(\theta)=0.3(0.7)=0.21\). A Bernoulli variable with success probability \(0.3\) has variance \(0.3(1-0.3)=0.21\), as required.
Worked Example: Poisson Log-Partition Function
For a Poisson response with mean \(\lambda\), \(b(\theta)=e^\theta\), \(\theta=\log\lambda\), and \(a(\phi)=1\). The derivatives are
Thus \(\mu=e^\theta=\lambda\), and replacing \(e^\theta\) by \(\mu\) gives \(V(\mu)=\mu\). Since the dispersion factor is one, \(\operatorname{Var}(Y)=\mu\).
For \(\lambda=2.5\), \(\theta=\log(2.5)\), so \(b'(\theta)=2.5\) and \(b''(\theta)=2.5\). The derived mean and variance are both \(2.5\), agreeing with the Poisson mean-variance relationship. The mean domain is \(\mu>0\), not including zero for a finite canonical parameter.
Normal and Gamma Responses
For normal and gamma responses, dispersion scales the response variance. Therefore \(b''(\theta)\) alone is not the full variance unless the dispersion factor equals one. In each case, first identify \(\mu=b'(\theta)\), then express \(b''(\theta)\) in terms of that mean.
Worked Example: Normal Log-Partition Function
A normal response \(Y\sim N(\mu,\phi)\) has \(a(\phi)=\phi\), \(b(\theta)=\theta^2/2\), and \(\theta=\mu\). The derivatives are
Therefore the mean is \(\mu=\theta\), the variance function is \(V(\mu)=1\), and the response variance is \(\phi V(\mu)=\phi\). The mean domain is the whole real line.
For example, if \(\mu=4\) and \(\phi=1.44\), then \(\theta=4\), \(b'(4)=4\), and \(b''(4)=1\). The response variance is \(1.44\times1=1.44\). This constant variance function does not mean that every normal response has variance one: the dispersion factor determines its response variance.
Worked Example: Gamma Log-Partition Function
Use the gamma exponential-dispersion parameterization with \(a(\phi)=\phi\), \(b(\theta)=-\log(-\theta)\), and canonical parameter \(\theta=-1/\mu<0\). Differentiation gives
The mean is \(-1/\theta=\mu\). Since \(1/\theta^2=\mu^2\), the variance function is \(V(\mu)=\mu^2\), and the response variance is \(\phi\mu^2\). Here \(\mu>0\), matching the canonical domain \(\theta<0\).
Take \(\mu=3\) and \(\phi=0.2\). Then \(\theta=-1/3\), \(b'(\theta)=3\), and \(b''(\theta)=1/(-1/3)^2=9\). The variance function is \(9\), while the response variance is \(0.2\times9=1.8\). The distinction is important: reporting \(9\) as the response variance would omit the dispersion scaling.
Comparing the Four Derivations
The calculations can be checked side by side. The column headed \(V(\mu)\) contains the second derivative after expressing it in terms of the mean; the final column includes the dispersion factor needed for the response variance.
| Family | \(b(\theta)\) | Mean \(\mu=b'(\theta)\) | \(V(\mu)=b''(\theta)\) | Response variance |
|---|---|---|---|---|
| Bernoulli | \(\log(1+e^\theta)\) | \(e^\theta/(1+e^\theta)\) | \(\mu(1-\mu)\) | \(\mu(1-\mu)\) |
| Poisson | \(e^\theta\) | \(e^\theta\) | \(\mu\) | \(\mu\) |
| Normal | \(\theta^2/2\) | \(\theta\) | \(1\) | \(\phi\) |
| Gamma | \(-\log(-\theta)\) | \(-1/\theta\) | \(\mu^2\) | \(\phi\mu^2\) |
For Bernoulli, the mean range is \(0<\mu<1\); for Poisson, it is \(\mu>0\); for normal, it is \(\mu\in\mathbb{R}\); and for gamma, it is \(\mu>0\). These domains are useful checks: a formula derived from \(b\) should only be interpreted for means attainable by a parameter in the family.
Common Mistake: Dropping or Adding Dispersion
A frequent error is to treat \(b''(\theta)\) as the response variance in every exponential-dispersion model. The calculation above shows why that fails: \(b''(\theta)\) is the variance function, and the response variance is \(a(\phi)b''(\theta)\). For the normal and gamma models, omitting \(\phi\) gives the wrong answer unless \(\phi=1\).
The reverse error is to multiply by a dispersion factor when the chosen family form has \(a(\phi)=1\). In the Bernoulli and Poisson parameterizations used here, no such extra factor appears. Always read the scale factor from the specified model form rather than assuming it is present or absent.
Check Your Understanding
Use the displayed log-partition functions and the stated dispersion factors to answer each question.
- For a Bernoulli response with \(\mu=0.6\), calculate the variance function and the response variance.
- For a Poisson response with canonical parameter \(\theta=\log(4)\), calculate \(b'(\theta)\), \(b''(\theta)\), and the response variance.
- For a normal response with dispersion \(\phi=2.5\), what are the variance function and response variance?
- For a gamma response with \(\mu=5\) and \(\phi=0.1\), calculate \(V(\mu)\) and \(\operatorname{Var}(Y)\).
- Explain why \(b''(\theta)\) is not, by itself, the response variance for the normal and gamma parameterizations used here.