Why Derivatives Give Moments
In a generalized linear model, the response distribution is described conditionally on the predictors. A useful way to obtain its mean and variance is to differentiate the function that makes the exponential-family probability model sum or integrate to one. This connects a model's parameterization to quantities that are directly interpretable on the response scale.
The key is to differentiate with respect to the natural parameter while holding the dispersion parameter fixed. The first derivative of the cumulant function gives the conditional mean; its second derivative gives the conditional variance after multiplication by the dispersion factor. The factor matters: omitting it gives an incorrect variance whenever the dispersion is not one.
The assumptions ensure that differentiating the probability model does not introduce changing integration limits or invalid exchanges of differentiation and integration. In the standard one-parameter form, \(Y\) itself is the statistic multiplied by \(\theta\). Other exponential-family forms can involve a statistic such as \(T(Y)\); the formulas above apply to the stated form with \(Y\) as that statistic.
The notation \(b'(\theta)\) and \(b''(\theta)\) means the first and second derivatives with respect to \(\theta\). In particular, \(\phi\) is held fixed during these derivatives. If \(a(\phi)=1\), the variance is simply \(b''(\theta)\); otherwise, it is \(a(\phi)b''(\theta)\).
Deriving the Mean and Variance
The formulas follow from normalization. Since the probabilities or density integrate to one for every allowed \(\theta\), differentiating that condition gives an identity involving the first moment. Differentiating a second time gives an identity involving the variance.
For fixed \(\phi\), the derivative of the log density with respect to \(\theta\) is \([y-b'(\theta)]/a(\phi)\). Thus the derivative of the density itself is the density multiplied by this expression.
The integral or sum of the density is one, so its derivative is zero. This yields \(E[Y\mid x]-b'(\theta)=0\), giving the conditional mean.
Differentiating again produces a term involving \([Y-b'(\theta)]^2\) and a term involving \(b''(\theta)\). Their expectations must balance, giving the conditional variance.
To see the details, write \(a=a(\phi)\) temporarily. The derivative of the log density is \([y-b'(\theta)]/a\). Since the integral or sum of \(f(y\mid x)\) is one, its derivative is zero:
For a discrete response, replace the integral by a sum. Since \(a>0\), this proves \(E[Y\mid x]=b'(\theta)\). For the second derivative, differentiating the density twice gives
The integral or sum of this second derivative is also zero. Therefore,
The last equality uses the mean already derived: \(b'(\theta)=E[Y\mid x]\), so the expected squared deviation from \(b'(\theta)\) is the conditional variance. This derivation also shows why \(b''(\theta)\) must be nonnegative wherever the model is valid: \(a(\phi)>0\) and a variance cannot be negative.
Worked Example: Recovering the Mean and Variance of a Poisson Response
Suppose a hypothetical claim-count response \(Y\), conditional on a predictor value \(x\), has a Poisson distribution with parameter \(\lambda=2\). Derive its mean and variance from the cumulant function, then check them directly from the probability mass function.
Method and parameterization. For a Poisson response, the mass at a nonnegative integer \(y\) is \(e^{-2}2^y/y!\). Write \(\theta=\log 2\), so \(e^\theta=2\). The exponential-family form is \(f(y\mid x)=\exp\{y\theta-e^\theta-\log(y!)\}\). Thus \(a(\phi)=1\), \(b(\theta)=e^\theta\), and the support is \(y=0,1,2,\ldots\).
Mean. Differentiate the cumulant function:
Variance. Differentiate again and include \(a(\phi)=1\):
Direct check. The exponential-series identity \(\sum_{y=0}^{\infty}2^y/y!=e^2\) gives \(E[Y]=e^{-2}\sum_{y=1}^{\infty}y2^y/y! =2e^{-2}\sum_{z=0}^{\infty}2^z/z!=2\). Likewise, \(E[Y(Y-1)]=4\). Hence \(E[Y^2]=E[Y(Y-1)]+E[Y]=4+2=6\), and \(\operatorname{Var}(Y)=E[Y^2]-E[Y]^2=6-4=2\).
The derivative method and the direct calculation agree. A common mistake avoided here is to confuse \(b(\theta)\) with the mean: the mean is \(b'(\theta)\), not \(b(\theta)\), even though they happen to be equal for this particular Poisson parameterization.
Worked Example: A Bernoulli Response With Success Probability One-Quarter
A hypothetical indicator \(Y\) records whether a small portfolio segment has a specified event, with \(P(Y=1\mid x)=1/4\) and \(P(Y=0\mid x)=3/4\). Find its natural parameter, cumulant function, conditional mean, and conditional variance. Check the moments using the two possible outcomes.
Method and parameterization. The Bernoulli mass is \(p^y(1-p)^{1-y}\), for \(y=0,1\). Rearranging it gives \(\exp\{y\log[p/(1-p)]+\log(1-p)\}\). Therefore,
Differentiate. The first two derivatives are
At \(e^\theta=1/3\), the conditional mean is \(b'(\theta)=(1/3)/(4/3)=1/4\). The conditional variance is \(b''(\theta)=(1/3)/(4/3)^2=(1/3)(9/16)=3/16=0.1875\).
Direct check. Since \(Y\) is one with probability \(1/4\) and zero otherwise, \(E[Y]=(1)(1/4)+(0)(3/4)=1/4\). Also \(E[Y^2]=1/4\), because \(Y^2=Y\). Thus \(\operatorname{Var}(Y)=1/4-(1/4)^2=1/4-1/16=3/16\). The derivative results match both direct moments.
The method is to express the probability mass in canonical exponential form before differentiating. A common mistake is to use \(\log p\) as the natural parameter; the coefficient of \(y\) after rearranging is instead \(\log[p/(1-p)]\).
Worked Example: A Geometric Response and the Dispersion Factor
Suppose \(Y\) is the number of failures before the first success in a hypothetical sequence of independent trials, each with success probability \(p=0.4\). Derive its mean and variance from the cumulant function and find \(P(Y\geq 3)\). Use the convention that \(Y\) takes values \(0,1,2,\ldots\).
Method and parameterization. Put \(q=1-p=0.6\). The probability mass is \(p q^y\), which can be written \(\exp\{y\log q+\log p\}\). Consequently, \(\theta=\log q=\log(0.6)\), \(b(\theta)=-\log(1-e^\theta)\), and \(a(\phi)=1\).
Differentiate and evaluate.
Using \(e^\theta=0.6\) and \(1-e^\theta=0.4\), \(E[Y\mid x]=0.6/0.4=1.5\), and \(\operatorname{Var}(Y\mid x)=0.6/(0.4)^2=0.6/0.16=3.75\). The tail event \(Y\geq3\) requires the first three trials to be failures, so \(P(Y\geq3)=0.6^3=0.2160\), to four decimal places.
Direct check. For this geometric convention, the mean and variance are \(q/p\) and \(q/p^2\), respectively. Substitution gives \(0.6/0.4=1.5\) and \(0.6/0.16=3.75\), agreeing with the derivatives. This check also depends on the stated convention: a geometric variable that counts the trial number of the first success has support \(1,2,\ldots\) and a mean larger by one.
The common mistake avoided is overlooking the definition of the count. Confirming whether the response counts failures or the trial of the first success prevents a one-unit error in the mean and a misread tail probability.
Dispersion, Conditional Interpretation, and Common Traps
The result is conditional when the distribution is specified given predictors. If \(\theta\) depends on \(x\), the equations are \(E[Y\mid x]=b'(\theta(x))\) and \(\operatorname{Var}(Y\mid x)=a(\phi)b''(\theta(x))\). These are not generally the unconditional mean and variance across a population with varying predictor values. In particular, the unconditional variance can also reflect variation in the conditional means.
The dispersion factor is part of the variance formula, not an optional adjustment. For example, in a normal response model written with \(a(\phi)=\phi\), the second derivative of \(b\) alone is not the full conditional variance. The factor \(\phi\) must multiply it. Conversely, setting \(a(\phi)=1\) is appropriate only for a parameterization or model that actually has that factor.
The derivatives are with respect to the natural parameter, not automatically with respect to a probability, a mean, or a linear predictor. If a problem gives \(p\) but asks for a derivative with respect to \(\theta\), first express \(\theta\) and \(b(\theta)\) in the chosen exponential-family form. Also check whether the stated response uses \(Y\) itself as the statistic; if the exponent contains a different statistic, the direct moment interpretation must be adjusted.
Check Your Understanding
For each model, identify the relevant derivatives and retain any dispersion factor. Answers are included for checking.
- A Bernoulli response has \(b(\theta)=\log(1+e^\theta)\), \(a(\phi)=1\), and \(\theta=\log 2\). Find its conditional mean and variance. (Answer: mean \(2/3\); variance \(2/9\).)
- A Poisson response has \(b(\theta)=e^\theta\), \(a(\phi)=1\), and \(\theta=\log 3\). Find its conditional mean, variance, and probability of zero. (Answer: mean \(3\); variance \(3\); probability of zero \(e^{-3}\approx0.0498\).)
- A model has \(b(\theta)=\theta^2/2+4\theta\), \(a(\phi)=2\), and \(\theta=1\). Use the derivative formulas to find its conditional mean and variance. (Answer: mean \(5\); variance \(2\).)
- A geometric variable counts failures before the first success, with success probability \(0.8\). Find its mean, variance, and probability of at least two failures. (Answer: mean \(0.25\); variance \(0.3125\); probability \(0.04\).)
- A normal response is parameterized with \(a(\phi)=\sigma^2=4\), \(b(\theta)=\theta^2/2\), and \(\theta=-1\). Find its conditional mean and variance, taking care to use the stated dispersion. (Answer: mean \(-1\); variance \(4\).)