Deriving The Poisson Mean And Variance

Why the Poisson Log-Partition Function Matters

A Poisson regression model specifies a count response through its conditional distribution given covariates. The mean is central to that specification, but the Poisson model also imposes a particular relationship between the mean and variance. Both quantities can be derived from one function: the log-partition function. Differentiating it shows exactly how the natural parameter determines the response moments.

This connection also explains the log link. For a Poisson response, the natural parameter is the logarithm of the mean. Thus, when the natural parameter is represented by a linear predictor, the conditional mean is the exponential of that predictor. The discussion below focuses on deriving these facts, not on fitting a regression model.

The Poisson Exponential-Family Form

Let \(Y\) be a count taking values in \(\{0,1,2,\ldots\}\). For a finite real parameter \(\theta\), its Poisson probability mass function can be written

P(Y=y)=\frac{\exp(y\theta)}{y!}\exp\{-\exp(\theta)\},\qquad y=0,1,2,\ldots

Equivalently, this has the canonical exponential-family form \(P(Y=y)=\exp\{y\theta-b(\theta)+c(y)\}\), where

b(\theta)=\exp(\theta),\qquad c(y)=-\log(y!).

The function \(b(\theta)\) is the log-partition function: it is the term that makes the probabilities sum to one. To check the form, set \(\lambda=\exp(\theta)>0\). Then \(P(Y=y)=\exp(-\lambda)\lambda^y/y!\), the mass function of a Poisson random variable with mean parameter \(\lambda\). In this parameterization, \(\theta=\log(\lambda)\) is called the natural parameter.

Definition: For the Poisson model, the natural parameter is \(\theta=\log(\lambda)\), and the log-partition function is \(b(\theta)=\exp(\theta)\). The canonical form is \(P(Y=y)=\exp\{y\theta-b(\theta)-\log(y!)\}\) for nonnegative integer \(y\).

Deriving the Mean and Variance

For an exponential-family distribution, derivatives of the log-partition function encode moments. In this Poisson case, the identities are

E[Y]=b'(\theta),\qquad \operatorname{Var}(Y)=b''(\theta).

Here is a direct justification for the Poisson model. Its normalizing sum is

Z(\theta)=\sum_{y=0}^{\infty}\frac{\exp(y\theta)}{y!} =\exp\{\exp(\theta)\}.

The equality follows from the power series for the exponential function. The series may be differentiated twice with respect to \(\theta\): on any bounded interval \(|\theta|\leq M\), the absolute values of the terms in the series and its first two derivative series are bounded respectively by \(e^{yM}/y!\), \(y e^{yM}/y!\), and \(y^2 e^{yM}/y!\). Each bounding series converges by the ratio test. Thus the series and its first two derivative series converge uniformly on that interval, justifying term-by-term differentiation.

Theorem: If \(Y\) has the Poisson distribution in canonical form with finite natural parameter \(\theta\), then \(E[Y]=b'(\theta)\) and \(\operatorname{Var}(Y)=b''(\theta)\), where \(b(\theta)=\exp(\theta)\).

To prove the first identity, differentiate \(Z\). By the justified term-by-term differentiation,

Z'(\theta)=\sum_{y=0}^{\infty}\frac{y\exp(y\theta)}{y!}.

The probability at \(y\) is \(\exp(y\theta)/(y!Z(\theta))\), so dividing the last sum by \(Z(\theta)\) gives

E[Y]=\sum_{y=0}^{\infty}yP(Y=y)=\frac{Z'(\theta)}{Z(\theta)} =\frac{d}{d\theta}\log Z(\theta)=b'(\theta).

For the second identity, differentiating twice gives \(Z''(\theta)=\sum_y y^2\exp(y\theta)/y!\), and hence \(E[Y^2]=Z''(\theta)/Z(\theta)\). Therefore

\begin{aligned} \operatorname{Var}(Y) &=E[Y^2]-\{E[Y]\}^2\\ &=\frac{Z''(\theta)}{Z(\theta)} -\left\{\frac{Z'(\theta)}{Z(\theta)}\right\}^2\\ &=\frac{d^2}{d\theta^2}\log Z(\theta) =b''(\theta). \end{aligned}

Since \(\log Z(\theta)=\exp(\theta)=b(\theta)\), direct differentiation yields

b'(\theta)=\exp(\theta),\qquad b''(\theta)=\exp(\theta).

Consequently, both the mean and variance equal \(\exp(\theta)\). This proves the stated identities and the Poisson mean-variance relationship. \(\square\)

From the Natural Parameter to a Regression Mean

Now suppose the distribution of a count response \(Y\) is specified conditionally on a covariate vector \(x\). If the conditional distribution is Poisson with natural parameter \(\theta(x)\), the result just derived gives

E[Y\mid X=x]=\exp\{\theta(x)\},\qquad \operatorname{Var}(Y\mid X=x)=\exp\{\theta(x)\}.

Write the conditional mean as \(\mu(x)=E[Y\mid X=x]\). Since \(\mu(x)=\exp\{\theta(x)\}\), taking the logarithm gives \(\log\{\mu(x)\}=\theta(x)\). In a Poisson regression model with a linear predictor \(\eta(x)=x^{\mathsf T}\beta\), the canonical specification is \(\theta(x)=\eta(x)\). It follows that

\log\{\mu(x)\}=x^{\mathsf T}\beta,\qquad \mu(x)=\exp\{x^{\mathsf T}\beta\},\qquad \operatorname{Var}(Y\mid X=x)=\mu(x).

The logarithm is therefore the canonical link for the Poisson response: it maps the positive conditional mean to the natural parameter. The exponential of the linear predictor ensures that the resulting conditional mean is positive for every finite predictor value.

Worked Example: A Natural Parameter Gives the Mean and Variance

Suppose \(Y\) has a Poisson distribution with natural parameter \(\theta=\log(3)\). Find its mean and variance using the log-partition function.

The log-partition function and its derivatives are \(b(\theta)=\exp(\theta)\), \(b'(\theta)=\exp(\theta)\), and \(b''(\theta)=\exp(\theta)\). At \(\theta=\log(3)\),

b'(\log 3)=\exp\{\log(3)\}=3,\qquad b''(\log 3)=\exp\{\log(3)\}=3.

Thus \(E[Y]=3\) and \(\operatorname{Var}(Y)=3\). The corresponding Poisson mean parameter is \(\lambda=\exp(\theta)=3\), consistent with both derivative calculations.

Worked Example: A Linear Predictor Determines the Conditional Moments

Consider a Poisson regression mean with predictor \(x\) and linear predictor \(\eta(x)=-0.4+0.3x\). Find the conditional mean and variance at \(x=2\).

First evaluate the linear predictor:

\eta(2)=-0.4+0.3(2)=-0.4+0.6=0.2.

For the canonical log link, \(\theta(2)=\eta(2)=0.2\). Therefore the conditional mean is \(\mu(2)=\exp(0.2)\), approximately \(1.2214\). The conditional variance is the same:

E[Y\mid X=2]=\exp(0.2)\approx1.2214,\qquad \operatorname{Var}(Y\mid X=2)=\exp(0.2)\approx1.2214.

The value \(0.2\) is the natural parameter, not the mean. Exponentiating it gives the positive mean and variance.

Conditional and Marginal Variability

The equality of mean and variance applies conditional on the covariates. It does not generally say that the unconditional variance of \(Y\) equals its unconditional mean when covariates vary across observations. The law of total expectation and the law of total variance give

E[Y]=E\{\mu(X)\},\qquad \operatorname{Var}(Y)=E\{\operatorname{Var}(Y\mid X)\} +\operatorname{Var}\{E(Y\mid X)\} =E\{\mu(X)\}+\operatorname{Var}\{\mu(X)\}.

Worked Example: A Poisson Conditional Model Can Have Larger Marginal Variance

Let \(X\) take two values with equal probability. Given \(X=0\), let \(Y\) be Poisson with mean \(1\); given \(X=1\), let \(Y\) be Poisson with mean \(4\). Find the unconditional mean and variance.

The conditional mean and variance are \(1\) when \(X=0\), and \(4\) when \(X=1\). Thus

E[Y]=\frac{1+4}{2}=2.5.

For the variance, the expected conditional variance is \((1+4)/2=2.5\). The variance of the two conditional means is

\operatorname{Var}\{\mu(X)\} =\frac{(1-2.5)^2+(4-2.5)^2}{2} =\frac{2.25+2.25}{2}=2.25.

Therefore \(\operatorname{Var}(Y)=2.5+2.25=4.75\), which is greater than the unconditional mean \(2.5\). This does not contradict the Poisson conditional relationship: at each value of \(X\), the conditional variance equals the conditional mean.

Common Mistake and Key Takeaway

A frequent error is to identify the linear predictor itself as the Poisson mean. For example, if \(\eta(x)=0.2\), it is incorrect to say the conditional mean is \(0.2\) or that the conditional variance is \(0.2\). The canonical Poisson parameterization sets \(\theta(x)=\eta(x)\), while the mean is \(b'\{\theta(x)\}=\exp\{\eta(x)\}\). At \(\eta(x)=0.2\), both conditional moments are \(\exp(0.2)\), not \(0.2\).

A related error is to state without qualification that “the Poisson variance equals the mean” when discussing pooled observations with varying covariates. The exact model statement is \(\operatorname{Var}(Y\mid X=x)=E[Y\mid X=x]\) for each covariate value \(x\). Marginal variation also reflects variation in the conditional means.

Key takeaway: For the Poisson model, \(b(\theta)=\exp(\theta)\), so \(b'(\theta)=b''(\theta)=\exp(\theta)\). Thus the conditional mean and variance both equal \(\exp\{\theta(x)\}\). When \(\theta(x)=x^{\mathsf T}\beta\), the conditional mean is linked to the linear predictor by \(\log\{\mu(x)\}=x^{\mathsf T}\beta\).

Check Your Understanding

Use the log-partition function and conditional-mean interpretation to answer these questions.

For a Poisson response with natural parameter \(\theta=\log(5)\), what are the mean and variance?
Why does differentiating \(\log Z(\theta)\) once give the mean, while differentiating it twice gives the variance?
If a Poisson regression linear predictor at a covariate value is \(-0.3\), what are the conditional mean and variance?
State the conditional Poisson mean-variance relationship using \(\mu(x)\).
Why can the unconditional variance exceed the unconditional mean when the covariates vary?