Negative Binomial As A Poisson Overdispersion Family

Why Add a Count Likelihood Beyond Poisson?

A Poisson response is a natural starting point for nonnegative integer counts. Its defining restriction is that, conditional on the model predictors, the variance equals the mean. Real count outcomes can vary more than this restriction permits: observations with similar fitted means may include more unusually small and large counts than a Poisson model predicts. The negative binomial likelihood retains the count support and mean structure while adding a parameter that permits greater variance.

In a generalized linear mixed model (GLMM), a response distribution is specified conditional on the random effects. The choice between Poisson and negative binomial therefore concerns the conditional response likelihood. Variation induced by random effects and extra variation represented by the negative binomial dispersion parameter are conceptually distinct, even though either can contribute to greater variation across observations before accounting for the random effects.

Definition: Let \(Y\) be a count with conditional mean \(\mu>0\). A Poisson response has \(Y\sim\operatorname{Poisson}(\mu)\), with probability mass function \(P(Y=y)=e^{-\mu}\mu^y/y!\) for \(y=0,1,2,\ldots\). A negative binomial response with mean \(\mu\) and size \(r>0\) has \(P(Y=y)=\frac{\Gamma(y+r)}{\Gamma(r)y!}\left(\frac{r}{r+\mu}\right)^r\left(\frac{\mu}{r+\mu}\right)^y\). Its variance is \(\mu+\mu^2/r\). Equivalently, write \(\alpha=1/r\); then the variance is \(\mu+\alpha\mu^2\), where \(\alpha>0\) is a dispersion parameter.

Both likelihoods assign probabilities only to nonnegative integers, and both can use a model for the mean such as \(\log(\mu_{ij})=\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+\mathbf{z}_{ij}^{\mathsf T}\mathbf{b}_j\), conditional on cluster-level random effects \(\mathbf{b}_j\). Here \(i\) indexes observations within cluster \(j\), \(\mathbf{x}_{ij}\) and \(\mathbf{z}_{ij}\) are covariate vectors, and \(\boldsymbol{\beta}\) is a fixed-effect coefficient vector. Under a Poisson response, the conditional variance is \(\mu_{ij}\). Under a negative binomial response, it is \(\mu_{ij}+\alpha\mu_{ij}^2\). At the same conditional mean, the negative binomial variance is larger for every \(\alpha>0\), and it approaches the Poisson variance as \(\alpha\) approaches zero.

Where the Extra Variance Comes From

A useful way to understand the negative binomial is as a Poisson distribution whose rate varies across observations. Suppose a latent rate multiplier \(U\) follows a gamma distribution with shape \(r\) and rate \(r\), so that \(E(U)=1\) and \(\operatorname{Var}(U)=1/r=\alpha\). Conditional on \(U\), let \(Y\mid U\sim\operatorname{Poisson}(\mu U)\). Every observation has the same average rate \(\mu\), but its latent rate can differ. Integrating over that unobserved variation gives a negative binomial distribution.

Specify the latent rate.
Let \(U\) have a gamma density with shape \(r\) and rate \(r\): \(f(u)=r^r u^{r-1}e^{-ru}/\Gamma(r)\), for \(u>0\). Its mean is 1 and its variance is \(1/r\).

Condition on that rate.
The Poisson probability is \(P(Y=y\mid U=u)=e^{-\mu u}(\mu u)^y/y!\).

Average over the latent rate.
Integrating the conditional Poisson probability against the gamma density gives the negative binomial probability mass function.

The mean and variance also follow directly from conditional expectation and variance. The law of total expectation gives \(E(Y)=E\{E(Y\mid U)\}=E(\mu U)=\mu\). The law of total variance gives a Poisson contribution plus a contribution from rate variation:

\begin{aligned} \operatorname{Var}(Y) &=E\{\operatorname{Var}(Y\mid U)\}+\operatorname{Var}\{E(Y\mid U)\}\\ &=E(\mu U)+\operatorname{Var}(\mu U)\\ &=\mu+\mu^2\operatorname{Var}(U) =\mu+\frac{\mu^2}{r}. \end{aligned}

For completeness, the mixture probability itself is

\begin{aligned} P(Y=y) &=\int_0^\infty \frac{e^{-\mu u}(\mu u)^y}{y!} \frac{r^r u^{r-1}e^{-ru}}{\Gamma(r)}\,du\\ &=\frac{r^r\mu^y}{\Gamma(r)y!} \int_0^\infty u^{y+r-1}e^{-(r+\mu)u}\,du\\ &=\frac{\Gamma(y+r)}{\Gamma(r)y!} \left(\frac{r}{r+\mu}\right)^r \left(\frac{\mu}{r+\mu}\right)^y. \end{aligned}

The final equality uses the gamma integral \(\int_0^\infty u^{a-1}e^{-cu}\,du=\Gamma(a)/c^a\) for \(a,c>0\). This derivation explains one standard interpretation of the negative binomial dispersion: it represents residual multiplicative heterogeneity in the Poisson rate, conditional on the predictors and any random effects already in the model. It does not establish that this particular latent-rate mechanism is the only possible cause of overdispersion.

Worked Example: Comparing Variances at a Shared Mean

Suppose the conditional mean count for an observation is \(\mu=2\), and a negative binomial model has \(\alpha=0.5\), equivalently \(r=1/\alpha=2\). Compare the two response variances.

Under the Poisson likelihood, \(\operatorname{Var}(Y)=\mu=2\). Under the negative binomial likelihood, \[ \operatorname{Var}(Y)=\mu+\alpha\mu^2 =2+0.5(2^2)=2+2=4. \] Thus the negative binomial allows twice the conditional variance at this mean. It does not change the stipulated conditional mean of 2; it changes how spread out the count distribution may be around that mean.

This calculation alone does not show which model is appropriate for any particular data set. The distributional assumption must be assessed against the observed counts and the fitted mean structure.

Likelihoods and Count Probabilities

For conditionally independent observations with means \(\mu_i\), the Poisson likelihood is the product of their Poisson probabilities. The negative binomial likelihood is the corresponding product of negative binomial probabilities, with a common dispersion parameter or another explicitly specified dispersion structure. In a GLMM, these products are conditional on the random effects; likelihood-based fitting must also account for their distribution. The essential comparison is that the Poisson likelihood has no response-level dispersion parameter, whereas the negative binomial likelihood estimates one in addition to the mean-model parameters.

Worked Example: Comparing Probabilities for a High Count

At conditional mean \(\mu=2\), compare the probability of observing \(Y=5\) under a Poisson likelihood and a negative binomial likelihood with \(r=2\) (\(\alpha=0.5\)). These are illustrative model probabilities, not data from a real study.

For the Poisson model, \[ P(Y=5)=e^{-2}\frac{2^5}{5!} =e^{-2}\frac{32}{120} \approx 0.03609. \]

For the negative binomial model, \(r/(r+\mu)=2/4=0.5\) and \(\mu/(r+\mu)=2/4=0.5\). Therefore, \[ P(Y=5)=\frac{\Gamma(7)}{\Gamma(2)5!}(0.5)^2(0.5)^5 =6(0.25)(0.03125) =0.046875. \]

At this mean and dispersion, the negative binomial assigns a higher probability to the count 5 than the Poisson model does. The probabilities of other counts also differ; greater variance does not mean that every count away from the mean must receive a larger probability. The negative binomial redistributes probability across the count support while retaining the same mean.

Worked Example: A Small Set of Counts Under Two Likelihoods

Consider the hypothetical counts \(0,1,5\), each modeled with conditional mean 2. For illustration, first treat the observations as conditionally independent with no random effects. Compare the likelihood of these exact observations under the Poisson model and the negative binomial model with \(r=2\). This is a likelihood comparison at fixed parameters, not a general model-selection test.

The Poisson probabilities are \(P(0)=e^{-2}\approx0.1353\), \(P(1)=2e^{-2}\approx0.2707\), and \(P(5)\approx0.03609\). Their product is \[ L_{\mathrm{Pois}}=e^{-2}(2e^{-2})\left(e^{-2}\frac{32}{120}\right) \approx (0.1353)(0.2707)(0.03609) \approx 0.001322. \]

For the negative binomial with \(r=2\) and \(\mu=2\), the probabilities are \(P(0)=(0.5)^2=0.25\), \(P(1)=2(0.5)^2(0.5)=0.25\), and \(P(5)=0.046875\), using the mass function above. Hence \[ L_{\mathrm{NB}}=(0.25)(0.25)(0.046875)=0.002930. \]

For these fixed means and this fixed dispersion, the negative binomial assigns a larger joint probability to this particular set of counts. The difference is driven in part by the relatively high count 5, but it does not establish that the negative binomial is superior in general. In actual fitting, parameters are estimated, the mean structure matters, and GLMM random effects must be handled as part of the likelihood.

Assumptions, Diagnostics, and Model Choice

The response-family comparison is meaningful when the response is a count, the chosen mean model is plausible, and observations are independent conditional on the modeled covariates and random effects. The Poisson specification further assumes equality of conditional mean and variance. The negative binomial specification relaxes that equality through its extra parameter and, in the parameterization used here, assumes \(\operatorname{Var}(Y\mid\text{model terms})=\mu+\alpha\mu^2\) for a constant \(\alpha>0\). Neither likelihood automatically corrects dependence, incorrect predictors, exposure differences, or an unsuitable random-effects structure.

Diagnostics should examine whether the fitted model reproduces features of the observed counts, not merely whether its variance formula is more flexible. Useful checks include comparing observed and simulated count frequencies, examining residual patterns against fitted means and covariates, and looking for systematic excesses of zeros or unusually large counts. A negative binomial model can still fit poorly if its mean structure is wrong or if the data have features its particular variance pattern cannot represent.

Feature	Poisson response	Negative binomial response
Support	Nonnegative integer counts	Nonnegative integer counts
Conditional mean	\(\mu\)	\(\mu\)
Conditional variance	\(\mu\)	\(\mu+\alpha\mu^2\)
Response dispersion parameter	None	\(\alpha>0\), or equivalently \(r=1/\alpha\)
Typical role	Count variation consistent with Poisson conditional variance	Count variation exceeding Poisson variance under the chosen mean model

A GLMM may also produce greater marginal variation through random effects: units with different random effects have different conditional means. This should not be confused with a negative binomial dispersion parameter, which allows extra variation in the response conditional on the random effects. Either source, or both, may be relevant. Including both without sufficient information can make their contributions difficult to distinguish, so model structure should be scientifically motivated and checked for identifiability and fit.

The negative binomial is a full probability likelihood, so it specifies probabilities for individual counts and supports likelihood-based fitting. Its added flexibility is not a guarantee of adequate fit: its variance grows according to a particular mean-dependent form. Alternatives include changing the mean structure or random-effects structure, and other count response families. The introductory material on variance-mean mismatch gives the broad distribution-choice context; here the key comparison is specifically the Poisson restriction against the negative binomial's additional dispersion parameter.

Common Mistakes and Limitations

Assuming a larger variance changes the mean. In the parameterization here, both likelihoods have mean \(\mu\); the negative binomial adds \(\alpha\mu^2\) to the variance.
Calling all extra variation “negative binomial dispersion.” Random effects, omitted predictors, dependence, unequal exposures, and other misspecifications can also produce apparent excess variation. The response-family parameter addresses only the variation remaining conditional on the model terms.
Choosing a negative binomial solely because the sample variance exceeds the sample mean. That comparison may flag a possible mismatch, but it does not account for predictors, clustering, or the fitted conditional means. Assess the model conditional on its structure.
Assuming the negative binomial fixes every count-data problem. It allows a particular form of overdispersion; it does not automatically reproduce excess zeros, dependence, or arbitrary tail behavior.
Treating one likelihood value as decisive evidence. Likelihood values depend on the data and parameter values. A complete comparison must account for estimation, model structure, and appropriate inferential procedures.

Key takeaway: The Poisson response imposes conditional variance equal to the conditional mean. The negative binomial keeps the count support and mean while adding a dispersion parameter, giving variance \(\mu+\alpha\mu^2\). Interpret that parameter as response-level extra variation conditional on the rest of the GLMM, not as a substitute for a well-specified mean or random-effects structure.

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions.

For a negative binomial response with \(\mu=3\) and \(\alpha=0.2\), calculate the variance and compare it with the Poisson variance at the same mean.
In the gamma-mixture interpretation, what does \(E(U)=1\) ensure, and what does \(\operatorname{Var}(U)>0\) add?
At a fixed conditional mean, does a negative binomial model assign a larger probability than a Poisson model to every possible count? Explain.
Why can a GLMM with Poisson responses still show more variation across all observations than a single Poisson distribution with one common mean?

Solutions. (1) The negative binomial variance is \(3+0.2(3^2)=3+1.8=4.8\), compared with Poisson variance 3; both have mean 3. (2) Since \(E(Y\mid U)=\mu U\), \(E(U)=1\) preserves the marginal mean \(E(Y)=\mu\). Positive \(\operatorname{Var}(U)\) adds \(\mu^2\operatorname{Var}(U)\) to the variance through variation in latent rates. (3) No. The two distributions allocate probability differently across the support; a larger variance does not make each individual count more probable. (4) Random effects can give different clusters or observations different conditional means. Variation across those means adds variability across observations, even when each response is Poisson conditional on its random effect.