Negative Binomial Type 1 Count Models

Why a Linear Variance Function Matters

A count model must describe both the expected count and how counts vary around that expectation. The Poisson model links these tightly: conditional on its mean, its variance equals that mean. When observed counts vary more than this permits, a negative binomial model can allow additional conditional variation. But “negative binomial” does not specify a single variance pattern. Negative binomial type 1 (NB1) makes variance increase linearly with the mean; negative binomial type 2 (NB2) makes it increase quadratically.

This distinction matters when fitted means differ substantially across observations. Two models can imply the same variability at one mean and quite different variability elsewhere. NB1 is useful when extra variation is approximately proportional to the mean, rather than growing rapidly as the mean becomes large.

Definition and Conditional Model

Definition: For a count response $Y_{ij}$, NB1 specifies a conditional mean $\mu_{ij}>0$ and conditional variance $$ \operatorname{Var}(Y_{ij}\mid b_j)=\mu_{ij}(1+\phi), $$ where $\phi\geq 0$ is a dispersion parameter. When $\phi=0$, the model reduces to the Poisson variance. For $\phi>0$, the conditional variance exceeds the mean.

In a GLMM, the conditional mean can be related to predictors and random effects by a log link, for example

\log(\mu_{ij})=\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta} +\mathbf{z}_{ij}^{\mathsf T}\mathbf{b}_j.

Here $\mathbf{x}_{ij}$ and $\mathbf{z}_{ij}$ are predictor vectors, $\boldsymbol{\beta}$ is the fixed-effect coefficient vector, and $\mathbf{b}_j$ is the random-effect vector for group $j$. The variance formula is conditional on those random effects: it describes response variation remaining after conditioning on the group effects and the predictors. It does not by itself describe the total variation after averaging over random effects.

A standard NB1 specification assumes nonnegative integer responses, a common $\phi$ unless the model explicitly allows it to vary, and conditional independence of responses given the random effects and model covariates. The chosen mean model must also be appropriate: a flexible variance function cannot repair a badly specified conditional mean. The conditional mean must be positive; the log link ensures this.

One probability mass function that makes the NB1 structure explicit is, for $\phi>0$,

P(Y_{ij}=y\mid b_j) = \frac{\Gamma(y+r_{ij})}{\Gamma(r_{ij})\,y!} p^{r_{ij}}(1-p)^y, \qquad r_{ij}=\frac{\mu_{ij}}{\phi}, \qquad p=\frac{1}{1+\phi}, \qquad y=0,1,2,\ldots .

The probability $p$ is constant across means when $\phi$ is fixed, while the size $r_{ij}$ changes with $\mu_{ij}$. The case $\phi=0$ is interpreted as the Poisson limit, rather than by substituting zero into this mass function.

Why the NB1 Variance Is Linear

The mean and variance follow directly from the probability generating function. For the mass function above, let $q=1-p$. Its probability generating function is

G(s)=E(s^{Y_{ij}}\mid b_j) =\left(\frac{p}{1-qs}\right)^{r_{ij}}.

For a probability generating function, $G'(1)$ is the mean and $G''(1)+G'(1)-[G'(1)]^2$ is the variance. Differentiating gives

G'(1)=\frac{r_{ij}q}{p}, \qquad G''(1)=\frac{r_{ij}(r_{ij}+1)q^2}{p^2}.

Since $q/p=\phi$ and $r_{ij}=\mu_{ij}/\phi$, the mean is $r_{ij}q/p=\mu_{ij}$. Substitution into the variance identity yields

\begin{aligned} \operatorname{Var}(Y_{ij}\mid b_j) &= \frac{r_{ij}q}{p^2}\\ &= \frac{\mu_{ij}}{\phi}(1+\phi)\phi\\ &= \mu_{ij}(1+\phi). \end{aligned}

Thus the extra variance over the Poisson value is $\phi\mu_{ij}$. For a fixed $\phi$, the variance-to-mean ratio is $1+\phi$ at every conditional mean. This constant ratio is the defining contrast with NB2, where the ratio increases as the mean increases.

NB1 and NB2: How Variability Changes With the Mean

NB2 uses a different variance function. With dispersion parameter $\alpha\geq0$, its conditional variance is $\mu_{ij}+\alpha\mu_{ij}^{2}$. The two models can be written side by side:

Model	Conditional variance	Variance-to-mean ratio	Growth of extra variance
Poisson	$\mu_{ij}$	1	None beyond Poisson variance
NB1	$\mu_{ij}(1+\phi)$	$1+\phi$	Linear in $\mu_{ij}$
NB2	$\mu_{ij}+\alpha\mu_{ij}^{2}$	$1+\alpha\mu_{ij}$	Quadratic in $\mu_{ij}$

The parameters $\phi$ and $\alpha$ belong to different variance functions and are not interchangeable. If the two models are calibrated to have equal variance at a positive reference mean $\mu_0$, then $\phi\mu_0=\alpha\mu_0^2$, so $\alpha=\phi/\mu_0$. At a smaller mean, the NB1 extra variance is greater; at a larger mean, the NB2 extra variance is greater. This comparison assumes positive dispersion and a shared reference mean.

Worked Example: Comparing Variances Across Three Means

Suppose two hypothetical models have conditional means $2$, $10$, and $30$ counts. Set $\phi=0.5$ for NB1 and $\alpha=0.05$ for NB2. The variances agree at mean $10$, because both models then have variance $15$.

For NB1, $\operatorname{Var}(Y\mid b)=\mu(1+0.5)=1.5\mu$. Thus the three variances are $1.5(2)=3$, $1.5(10)=15$, and $1.5(30)=45$ count-squared.

For NB2, $\operatorname{Var}(Y\mid b)=\mu+0.05\mu^2$. The corresponding calculations are $2+0.05(2^2)=2.2$, $10+0.05(10^2)=15$, and $30+0.05(30^2)=75$ count-squared.

At mean $2$, NB1 predicts more variance than NB2 (3 versus 2.2); at mean $30$, NB2 predicts more (75 versus 45). Both give variance $15$ at mean $10$, but their predictions diverge away from that reference point. These are conditional variances, not variances after accounting for variation in random effects.

Worked Predictions From a Log-Mean Model

Worked Example: From a Linear Predictor to Conditional Spread

Consider two hypothetical observations with log-mean linear predictors $0$ and $\log(4)$, and suppose their random effects are conditioned on at zero. Under the log link, their conditional means are $\exp(0)=1$ and $\exp(\log(4))=4$ counts. Let the fitted NB1 dispersion be $\phi=0.8$.

The conditional variances are $(1+0.8)(1)=1.8$ and $(1+0.8)(4)=7.2$ count-squared. Their conditional standard deviations are $\sqrt{1.8}=1.342$ counts and $\sqrt{7.2}=2.683$ counts, respectively.

The mean increases fourfold, and so does the variance: $7.2/1.8=4$. The standard deviation increases by a factor of $\sqrt{4}=2$, not four. This square-root relationship follows because the dispersion multiplier is constant while the variance is proportional to the mean.

Predicted Zero Probabilities Can Also Differ

A variance function affects the distribution of counts, not just a reported standard deviation. The following calculation compares the probability of a zero at a mean where NB1 and NB2 have been calibrated to agree, and then at a larger mean.

Worked Example: Zero Probabilities at a Larger Mean

Use NB1 with $\phi=0.5$, and NB2 with $\alpha=0.125$. At mean $\mu=4$, their variances agree: NB1 gives $4(1.5)=6$, while NB2 gives $4+0.125(4^2)=6$. For a negative binomial with size $r$ and probability $p$ as in the mass function above, $P(Y=0)=p^r$.

At mean $12$, NB1 has $r=12/0.5=24$ and $p=1/(1+0.5)=2/3$. Its zero probability is

P_{\text{NB1}}(Y=0)=\left(\frac{2}{3}\right)^{24}\approx 0.0000594.

For NB2, the size is $r=1/\alpha=8$, and $p=1/(1+\alpha\mu)=1/(1+0.125(12))=0.4$. Therefore

P_{\text{NB2}}(Y=0)=0.4^8=0.00065536.

At this mean, NB2 has variance $12+0.125(12^2)=30$, compared with NB1 variance $12(1.5)=18$. In this example, NB2 also assigns a larger probability to zero. A variance comparison alone does not determine every feature of a distribution in every setting, so probabilities of particular counts should be checked directly when they matter.

Assumptions, Diagnostics, and Alternatives

NB1 is a useful choice when a conditional negative-binomial distribution is appropriate and the data support approximately constant conditional overdispersion relative to the mean. Its assumption is not simply “variance exceeds the mean.” It makes the more specific claim that $\operatorname{Var}(Y\mid b)/\mu=1+\phi$ is constant over the fitted-mean range. An NB2 model makes a different claim: that the ratio increases linearly with the mean.

After fitting, examine whether residual variation is compatible with the fitted means across their range. In particular, compare observed and model-predicted variability in low-, middle-, and high-mean regions, and inspect whether discrepancies systematically increase with the mean. Simulation-based residual checks can assess whether the model reproduces the observed spread and the frequencies of zeros or unusually large counts. A stable-looking overall dispersion summary can conceal a poor fit at high means. Group structure also matters: if residuals remain dependent within groups after conditioning on the specified effects, a variance function alone is not a substitute for an adequate dependence model.

The main alternatives answer different questions. The Poisson model is more parsimonious but requires conditional equidispersion. NB2 allows extra variance to grow quadratically with the mean and may better represent increasing relative overdispersion at high means. Choosing between NB1 and NB2 should be guided by the mean-variance pattern, diagnostics, and the scientific scale of interest—not by assuming that one is universally more flexible or correct. Neither variance function automatically handles structural zeros, omitted predictors, or misspecified random-effects structure.

Common pitfall: “NB1 and NB2 have the same variance at the mean where I calibrated them, so their variability predictions are equivalent.” This is incorrect: equality at one positive mean does not imply equality elsewhere. For example, matching NB1 $\phi=0.5$ and NB2 $\alpha=0.05$ at $\mu=10$ gives variances $3$ and $2.2$ at $\mu=2$, and $45$ and $75$ at $\mu=30$. Compare predictions across the range of fitted means.

NB1 is also limited by its fixed dispersion assumption when extra variation changes systematically with predictors or groups. If the variance pattern is wrong, estimates and uncertainty may be unreliable even when the fitted mean appears reasonable. Treat the conditional variance as a modeling assumption to assess, not as a consequence guaranteed by choosing a negative binomial label.

Check Your Understanding

Work through the questions, then compare your reasoning with the solutions.

For NB1 with $\phi=0.3$ and conditional mean $8$ counts, find the conditional variance and standard deviation.
An NB1 model has $\phi=0.4$. By what factor does its conditional variance change when the mean rises from $3$ to $15$? By what factor does its standard deviation change?
NB1 with $\phi=0.6$ and NB2 with $\alpha=0.06$ are calibrated at mean $10$. Find both variances at means $2$ and $30$, and state which model predicts more variance at each mean.
Why is it not enough to check that a fitted NB1 model has variance greater than its mean?

Solutions. (1) The variance is $8(1+0.3)=10.4$ count-squared, so the standard deviation is $\sqrt{10.4}=3.225$ counts. (2) NB1 variance is proportional to the mean, so the variance changes by $15/3=5$; the standard deviation changes by $\sqrt{5}\approx2.236$. (3) At mean $2$, NB1 gives $2(1.6)=3.2$, whereas NB2 gives $2+0.06(2^2)=2.24$; NB1 predicts more variance. At mean $30$, NB1 gives $30(1.6)=48$, whereas NB2 gives $30+0.06(30^2)=84$; NB2 predicts more. Both variances equal $16$ at the calibration mean $10$. (4) Variance exceeding the mean establishes overdispersion relative to Poisson, but does not establish NB1’s specific constant variance-to-mean ratio. Check whether that ratio is approximately stable across the fitted-mean range.

Model	Conditional variance	Variance-to-mean ratio	Growth of extra variance
Poisson	\(\mu_{ij}\)	1	None beyond Poisson variance
NB1	\(\mu_{ij}(1+\phi)\)	\(1+\phi\)	Linear in \(\mu_{ij}\)
NB2	\(\mu_{ij}+\alpha\mu_{ij}^{2}\)	\(1+\alpha\mu_{ij}\)	Quadratic in \(\mu_{ij}\)