Negative Binomial Type 2 Count Models

Why a Quadratic Variance Term Matters

A Poisson GLMM accounts for clustering through random effects, but conditional on those effects it still assumes that the response variance equals its mean. Some count data vary more than that even after conditioning on the modeled random effects. The negative binomial type 2 model, abbreviated NB2, represents this additional conditional variation with a quadratic term in the mean. Because the term grows as the mean grows, NB2 allows the amount of extra-Poisson variation to depend on the expected count.

The distinction between conditional and marginal variability is central. In a count GLMM, the conditional mean and variance describe outcomes for a fixed value of the random effects. The marginal variance also includes variation arising because the random effects themselves vary across groups. NB2 adds a conditional variance contribution; it does not replace the random-effects structure.

Model and Assumptions

Definition: Let $Y_{ij}$ be a count for observation $j$ in group $i$, and let $b_i$ be that group’s random-effects vector. An NB2 count GLMM specifies a negative binomial conditional response distribution with conditional mean $\mu_{ij}>0$ and conditional variance $$ E(Y_{ij}\mid b_i)=\mu_{ij}, \qquad \operatorname{Var}(Y_{ij}\mid b_i)=\mu_{ij}+\alpha\mu_{ij}^{2}, $$ where $\alpha\geq 0$ is the NB2 dispersion parameter. A typical log-link model is $$ \log(\mu_{ij})=\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta} +\mathbf{z}_{ij}^{\mathsf T}b_i+\log(e_{ij}), $$ where $e_{ij}>0$ is an exposure when an exposure adjustment is needed; omit the offset term when none is used.

The model assumes the stated negative binomial response distribution conditional on the random effects, not merely a variance formula in isolation. It also requires a suitable mean model and random-effects distribution; a common specification is $b_i\sim N(\mathbf{0},D)$, independently across groups. Responses are assumed conditionally independent given the random effects and covariates when that conditional independence structure is appropriate for the design. The same $\alpha$ is often used across observations, so the model assumes a common dispersion parameter even though the variance changes with $\mu_{ij}$.

When $\alpha=0$, the variance reduces to $\mu_{ij}$, the Poisson conditional variance. When $\alpha>0$, the excess conditional variance over Poisson is $\alpha\mu_{ij}^{2}$. Thus $\alpha$ is not itself the excess variance: it scales the square of the conditional mean. Under this convention, $\alpha$ is dimensionless, and larger values imply greater overdispersion at the same mean.

Reading the NB2 Variance Function

Dividing the variance by the mean gives the conditional variance-to-mean ratio:

\frac{\operatorname{Var}(Y_{ij}\mid b_i)}{\mu_{ij}} =1+\alpha\mu_{ij}.

For a Poisson response this ratio is 1. Under NB2 it increases linearly with the conditional mean if $\alpha>0$. The variance itself increases quadratically in $\mu_{ij}$: doubling the mean doubles the Poisson component but multiplies the NB2 excess component by four. Consequently, NB2 is particularly relevant when larger fitted means show disproportionately greater variability.

This relationship describes conditional dispersion. It does not mean every observed group must have a sample variance greater than its sample mean; finite samples fluctuate, and group-level random effects can also produce marginal overdispersion. The variance function is a model assumption about the response distribution, not a rule that every small subset of observed counts must obey exactly.

Worked Example: Comparing NB2 and Poisson Variances

Suppose hypothetical observations have conditional means $\mu=2$ and $\mu=8$, and the fitted NB2 dispersion is $\alpha=0.15$. Assume the conditional negative binomial response model is appropriate and compare its variance with the Poisson variance at each mean.

At $\mu=2$, the NB2 variance is $$ 2+0.15(2^2)=2+0.60=2.60. $$ The Poisson variance would be 2, so the NB2 model adds $0.60$ count-squared units of conditional variance.

At $\mu=8$, the NB2 variance is $$ 8+0.15(8^2)=8+9.60=17.60. $$ The Poisson variance would be 8, so the additional conditional variance is 9.60. The variance-to-mean ratios are $2.60/2=1.30$ and $17.60/8=2.20$, respectively. They also follow directly from $1+\alpha\mu$: $1+0.15(2)=1.30$ and $1+0.15(8)=2.20$.

The larger mean has a much larger absolute excess variance because the excess is proportional to $\mu^2$. These are conditional calculations: they hold at the specified means and fixed random effects.

Using the Variance Function in a Count GLMM

The linear predictor determines $\mu_{ij}$, and the NB2 variance function then determines conditional response variability at that mean. For example, with a log link, a coefficient changes the mean multiplicatively while $\alpha$ governs how the conditional variance departs from the Poisson value. Coefficients and dispersion therefore answer different questions: the coefficients describe the conditional mean, while $\alpha$ describes the additional mean-dependent variance.

Worked Example: Conditional Mean and Variance From a Predictor

Consider a hypothetical model with no offset and a fixed group effect $b_i=0$: $$ \log(\mu_{ij})=\log(3)+\log(2)x_{ij}, \qquad \alpha=0.20. $$ Assume the conditional NB2 response distribution and the specified linear predictor are adequate. Calculate the conditional means and variances for $x_{ij}=0$ and $x_{ij}=1$.

For $x_{ij}=0$, $\log(\mu_{ij})=\log(3)$, so $\mu_{ij}=3$. The conditional variance is $$ 3+0.20(3^2)=3+1.8=4.8. $$ For $x_{ij}=1$, $\log(\mu_{ij})=\log(3)+\log(2)=\log(6)$, so $\mu_{ij}=6$. Its conditional variance is $$ 6+0.20(6^2)=6+7.2=13.2. $$ The mean doubles between these predictor values, consistent with a conditional mean ratio of 2. The variance does not simply double: the Poisson component rises from 3 to 6, while the excess component rises from 1.8 to 7.2 because the squared mean increases fourfold. The conditional variances are in squared count units.

Separating NB2 Variation From Random-Effect Variation

The law of total variance gives the connection between the conditional NB2 variance and the marginal variance. For any random effect $b_i$,

\operatorname{Var}(Y_{ij}) =E\{\operatorname{Var}(Y_{ij}\mid b_i)\} +\operatorname{Var}\{E(Y_{ij}\mid b_i)\}.

Substituting the NB2 conditional moments yields

\operatorname{Var}(Y_{ij}) =E(\mu_{ij}+\alpha\mu_{ij}^2)+\operatorname{Var}(\mu_{ij}).

The first term averages conditional response variation over the random effects; the second is variation in the conditional means across random-effect values. This identity shows why a marginal variance larger than a marginal mean does not, by itself, establish that NB2 dispersion is needed. Random effects can also increase marginal variability. In a GLMM the two contributions should be distinguished rather than attributed to one another.

Worked Example: Combining NB2 and Random-Intercept Variation

Suppose a hypothetical group has random intercept $b_i\sim N(0,0.16)$, and for one observation the conditional mean is $\mu_i=2\exp(b_i)$. Let the conditional response be NB2 with $\alpha=0.10$. Assume this lognormal random-intercept specification and the NB2 conditional moments hold. Calculate the marginal mean and variance.

For a normal variable $b_i$ with mean 0 and variance 0.16, $E\{\exp(t b_i)\}=\exp(t^2(0.16)/2)$. Therefore $$ E(\mu_i)=2\exp(0.08)\approx 2.16657, \qquad E(\mu_i^2)=4\exp(0.32)\approx 5.50851. $$ The random-effect contribution to the variance of the conditional mean is $$ \operatorname{Var}(\mu_i) =E(\mu_i^2)-\{E(\mu_i)\}^2 =4\exp(0.32)-4\exp(0.16) \approx 0.814468. $$ The expected conditional NB2 variance is $$ E(\mu_i+0.10\mu_i^2) =2.16657+0.10(5.50851) \approx 2.71743. $$ Thus the marginal variance is $$ \operatorname{Var}(Y_i) \approx 2.71743+0.814468=3.53189. $$ The marginal mean is about 2.16657 counts and the marginal variance is about 3.53189 squared counts. The conditional NB2 component and the random-intercept component both contribute. If $\alpha$ were zero while the random-intercept distribution stayed the same, the marginal variance would be approximately $2.16657+0.814468=2.98104$, still greater than the marginal mean because of random-effect variation.

Diagnostics, Alternatives, and Limitations

A useful diagnostic question is whether variability remains greater than the Poisson model predicts after accounting for the mean structure and random effects. Examine conditional residual patterns and compare observed variability with variability predicted at similar fitted conditional means. A pattern in which the variance-to-mean ratio increases with the fitted mean is compatible with NB2’s $1+\alpha\mu$ relationship, but it does not prove that the negative binomial distribution is correct. Diagnostics should also check whether the mean model, link, exposure adjustment, and grouping structure are plausible.

A Poisson GLMM is the simpler alternative when conditional equidispersion is credible. NB2 allows extra-Poisson conditional variation that grows quadratically with the mean. Other model choices may represent different sources or patterns of variation, but should not be selected from a raw variance-to-mean ratio alone: clustering, omitted predictors, dependence, and a misspecified mean can all create apparent overdispersion. NB2 also has limitations. A single $\alpha$ imposes a particular mean-variance pattern, and the model does not automatically explain structural zeros or other distributional features that do not match its assumptions.

A common mistake is to compare a pooled sample variance with a pooled sample mean and conclude that NB2 is required whenever the former is larger. That comparison ignores differences in fitted means and random effects. The correction is to ask whether extra variability remains conditional on the modeled effects and mean, and whether the quadratic variance pattern is a reasonable description of it.

Key takeaway: In an NB2 count GLMM, conditional variance equals $\mu+\alpha\mu^2$. The quadratic term represents extra-Poisson response variation at a fixed random-effect value, while random effects can add a separate component to the marginal variance.

Check Your Understanding

Answer each question, then compare your work with the solutions below.

For $\mu=5$ and $\alpha=0.12$, calculate the NB2 conditional variance and the variance-to-mean ratio.
If the conditional mean doubles from 4 to 8 with $\alpha=0.25$, how does the NB2 excess variance change?
In a count GLMM, name the two contributions to marginal variance in the law of total variance.
Why does a marginal variance greater than the marginal mean not establish that conditional NB2 dispersion is needed?

Solutions. (1) The conditional variance is $5+0.12(25)=8$, and its ratio to the mean is $8/5=1.6$, also $1+0.12(5)=1.6$. (2) The excess variance changes from $0.25(4^2)=4$ to $0.25(8^2)=16$, a fourfold increase. (3) They are the expected conditional response variance and the variance of the conditional mean across random effects. (4) Random effects can increase marginal variance through variation in conditional means, even when the conditional response variance is Poisson.