Why Counts May Vary More Than a Poisson Model Allows
A Poisson model is a useful starting point for counts, but it ties the variance to the mean: conditional on its modeled mean, a Poisson count has variance equal to that mean. In practice, counts with the same modeled predictors can be more variable. If that extra variability is ignored, fitted uncertainty can be too small and predictions may represent the data poorly.
A negative-binomial model relaxes the Poisson variance restriction. Its dispersion parameter controls how much the variance exceeds the mean. In a mixed-effects analysis, it is important to distinguish this within-observation dispersion from persistent differences among subjects. Both can increase the marginal variability of counts, but they imply different dependence across repeated observations.
Where the Mean–Variance Relationship Comes From
One way to derive NB2 is to let the Poisson rate vary from observation to observation. Let \(U\) be a positive latent multiplier with a gamma distribution having shape and rate both equal to \(\theta\). Then \(E(U)=1\) and \(\operatorname{Var}(U)=1/\theta\). Given \(U\), suppose \(Y\mid U\sim\operatorname{Poisson}(\mu U)\). The gamma multiplier allows the Poisson rate to fluctuate around \(\mu\).
By iterated expectation, \(E(Y)=E\{E(Y\mid U)\}=E(\mu U)=\mu\), since \(E(U)=1\).
The law of total variance gives \(\operatorname{Var}(Y)=E\{\operatorname{Var}(Y\mid U)\}+\operatorname{Var}\{E(Y\mid U)\}\).
The first term is \(E(\mu U)=\mu\); the second is \(\operatorname{Var}(\mu U)=\mu^2/\theta\). Thus \(\operatorname{Var}(Y)=\mu+\mu^2/\theta\).
Integrating the conditional Poisson probability over the gamma density gives the probability mass function in the definition. The variance derivation already shows the central point: the \(\mu\) term is ordinary Poisson variation, while \(\mu^2/\theta\) is the additional variation induced by the fluctuating rate.
Worked Example: Deriving the Extra Variance
Suppose a hypothetical count has conditional mean \(\mu=4\) and NB2 dispersion \(\theta=2\). The gamma multiplier has variance \(1/\theta=0.5\).
The mean is \(E(Y)=4\). The Poisson contribution to the variance is 4, and the extra contribution is \(\mu^2/\theta=4^2/2=8\). Therefore, $$ \operatorname{Var}(Y)=4+\frac{16}{2}=12. $$
The variance is three times the mean, not because the mean has changed, but because the latent rate multiplier adds variability. These are hypothetical model values, not results from a study.
Dispersion and Subject-Level Random Effects
The gamma–Poisson construction is a useful interpretation of NB2, but it should not be confused with a subject-level random effect. In a repeated-measures model, a subject random effect is shared across that subject’s observations. It represents persistent heterogeneity: a subject with a high rate tendency may tend to have high counts at several occasions. The gamma multiplier used to derive a standard NB2 observation distribution can instead be redrawn independently for each observation, producing extra variation without that persistent within-subject association.
For example, a conditional NB2 mixed-effects model can be written $$ Y_{ij}\mid b_i \sim NB2(\mu_{ij},\theta),\qquad \log(\mu_{ij})=\log(E_{ij})+\beta_0+\beta_1x_{ij}+b_i,\qquad b_i\sim N(0,\omega^2). $$ Here \(i\) indexes subjects, \(j\) indexes observations within subject, \(E_{ij}\) is exposure, \(x_{ij}\) is a predictor, and \(b_i\) is a subject-level random intercept. Conditional on \(b_i\), the NB2 variance is \(\mu_{ij}+\mu_{ij}^2/\theta\). The offset \(\log(E_{ij})\) accounts for differing exposure, while \(b_i\) allows persistent subject differences. The usual conditional-independence assumption is that observations are independent given the subject effects and model predictors.
Even if the conditional count variance is Poisson, a shared random effect can create marginal overdispersion and dependence. Conversely, NB2 dispersion can inflate conditional variances without representing a shared subject tendency. The two components can also be included together, but estimating both requires adequate repeated data and a model that is identifiable for the design. A single count per subject often provides little information to separate persistent heterogeneity from observation-level dispersion.
Worked Example: Estimating Dispersion From Hypothetical Counts
Suppose ten independent hypothetical observations at the same modeled condition have counts \(0,0,0,1,2,3,4,5,6,9\). Their sum is 30 and the sum of their squares is 172, so the sample mean is \(\bar{x}=30/10=3\). The sample variance is $$ s^2=\frac{\sum x_k^2-n\bar{x}^{\,2}}{n-1} =\frac{172-10(3^2)}{9} =\frac{82}{9}=9.111. $$
For NB2, equating the variance function to the sample variance gives the method-of-moments estimate \(\hat{\theta}=\bar{x}^{\,2}/(s^2-\bar{x})\), provided \(s^2>\bar{x}\). Here, $$ \hat{\theta}=\frac{3^2}{82/9-3} =\frac{9}{55/9} =\frac{81}{55}=1.473. $$ Using the unrounded estimate, the fitted moment relationship reproduces the sample variance exactly: $$ 3+\frac{3^2}{81/55}=3+\frac{55}{9}=\frac{82}{9}. $$
This calculation indicates greater variation than a Poisson model with mean 3 would allow: its variance would be 3 rather than \(82/9\). It is a moment estimate, not a full likelihood fit, and the sample is small; it should not be treated as precise evidence that the underlying process has exactly this dispersion.
Regression Interpretation and a Second Moment Calculation
In an NB2 regression, predictors usually determine \(\mu\) through a log link, while \(\theta\) controls the variance conditional on that mean. A coefficient is therefore interpreted through the mean, not as an additive change in the count. With a binary predictor and the same exposure, exponentiating its coefficient gives the ratio of conditional means. The dispersion parameter does not change that mean ratio, though it changes the variance and the likelihood weighting of observations.
Worked Example: A Mean Ratio and an NB2 Variance
Imagine a hypothetical model with conditional mean 3.00 for a reference condition and 1.80 for a treated condition, and use \(\theta=81/55\) from the preceding moment calculation for illustration. The treatment coefficient corresponding to these means is $$ \beta_1=\log\left(\frac{1.80}{3.00}\right)=\log(0.600)=-0.5108. $$
Thus the treated-to-reference conditional mean ratio is \(e^{\beta_1}=0.600\): the modeled mean count for the treated condition is 40.0% lower, at equal exposure and random effect. This is an association specified by the model; it does not by itself establish a causal treatment effect.
For the treated observation, the NB2 conditional variance is $$ 1.80+\frac{1.80^2}{81/55} =1.80+\frac{3.24(55)}{81} =1.80+2.20=4.000 $$ counts squared. The corresponding Poisson variance would be 1.80 counts squared. The larger NB2 variance reflects dispersion beyond the Poisson component.
As a separate arithmetic check, if hypothetical sample summaries are displayed as mean 2.266 and variance 3.725, the moment estimate based on those rounded inputs is $$ \hat{\theta}=\frac{2.266^2}{3.725-2.266} =\frac{5.134756}{1.459}=3.519. $$ Moment estimates calculated from unrounded summaries may differ slightly. Reported estimates should be consistent with the precision of the inputs used.
Fitting, Diagnostics, and Alternatives
A likelihood-based fit estimates regression coefficients and dispersion jointly (and, in a mixed-effects model, also estimates the random-effects distribution parameters). The fitted \(\mu_{ij}\) values and \(\theta\) specify the conditional distributions. A useful initial check is whether the Poisson variance restriction is implausible after accounting for predictors, exposure, and any random effects. A positive moment estimate requires variance greater than mean; if the sample variance is no larger than the mean, that simple moment formula does not provide a positive finite estimate of NB2 overdispersion.
Diagnostics should assess whether the fitted model reproduces the count distribution, not only whether its variance formula looks flexible. Compare observed and simulated counts across relevant predictor ranges, inspect residual patterns against fitted means and exposure, and assess whether large counts or repeated subject patterns remain systematically underpredicted. In repeated data, check whether the model reproduces within-subject dependence as well as marginal variability. A fitted NB2 model may broaden the distribution of counts, but it does not automatically explain every source of poor fit; in particular, unexplained structure in the zero counts or covariate effects calls for substantive investigation rather than simply assuming that a smaller \(\theta\) fixes the model.
| Approach | What it represents | Main distinction or limitation |
|---|---|---|
| Poisson mixed-effects model | Poisson variation conditional on predictors and subject effects; random effects represent persistent heterogeneity. | Conditional variance is restricted to equal the conditional mean. |
| NB2 model | Conditional variance \(\mu+\mu^2/\theta\), with dispersion beyond the Poisson mean. | Independent observation-level dispersion does not, by itself, create persistent within-subject dependence. |
| Quasi-Poisson model | Allows variance proportional to the mean, commonly \(\phi\mu\). | Specifies a mean–variance relation but not the NB2 probability mass function; likelihood-based comparisons and predictions are not directly interchangeable. |
| NB2 with subject random effects | Allows both conditional overdispersion and persistent subject heterogeneity. | More parameters can be hard to distinguish without enough repeated observations and information in the design. |
NB2 is not a universal remedy. Its variance grows quadratically with the mean, which may be inappropriate if the data instead support a different variance pattern. The assumed count distribution, link, predictor structure, exposure offset, and random-effects distribution still matter. Alternatives should be selected to reflect the data-generating structure and scientific question, rather than chosen only because one model gives a larger variance.
Common Mistakes
- Treating \(\theta\) as the extra variance itself. The additional variance is \(\mu^2/\theta\); \(\theta\) is a dispersion parameter whose smaller values imply more extra variation.
- Calling all overdispersion a random effect. A subject effect shared across occasions induces a repeated-subject association. Independent NB2 dispersion does not imply that same pattern.
- Using a rounded moment estimate to claim exact equality. Use unrounded values for an exact equality, or label the result approximate when rounded inputs are substituted.
- Assuming a better variance fit proves the model is adequate. Check the fitted count distribution, predictor patterns, and repeated-measures dependence as well.
Check Your Understanding
Try each question, then compare your reasoning with the short solutions below.
- If \(\mu=5\) and \(\theta=10\), what are the NB2 variance and extra-variance term?
- Why can two repeated counts from the same subject be associated under a shared random intercept even when their conditional residual counts are independent?
- A sample has mean 2 and variance 1.8. Does the NB2 moment formula give a positive finite estimate of \(\theta\)? Explain.
- For equal exposure, a fitted binary-predictor coefficient is \(\log(0.75)\). What is the conditional mean ratio, and how should it be interpreted?
Solutions. (1) The extra term is \(5^2/10=2.5\); the total variance is \(5+2.5=7.5\). (2) Both counts depend on the same latent subject effect. Integrating over that shared effect can induce marginal dependence, even if the counts are conditionally independent given it. (3) No. The moment formula requires \(s^2>\bar{x}\); here variance is below the mean, so it yields no positive finite overdispersion estimate. (4) The mean ratio is \(e^{\log(0.75)}=0.75\). At equal exposure and random effect, the modeled conditional mean is 25% lower for the group coded one than for the reference group; this association alone is not a causal conclusion.