Negative Binomial Parameterizations and Dispersion

Why Parameterization Matters

“Negative binomial dispersion” does not refer to one universal parameter. A model may report a size or shape parameter, its reciprocal, or a coefficient in a mean–variance formula. The same underlying distribution can therefore appear to have different parameter values in different software or references. Comparing numbers without first checking their definitions can reverse an interpretation: a larger size means less overdispersion, while a larger reciprocal-size dispersion means more.

The useful anchor is the relationship between the conditional mean and variance. This tutorial makes the conversion rules explicit for the common size–probability distribution, NB1, and NB2. The calculations concern a count $Y$ with mean $\mu$ under the stated model; in a GLMM, these quantities may be conditional on the predictors and other specified conditioning information.

Definition: For $r>0$ and $0<p\leq 1$, the negative binomial distribution in size–probability form assigns probability $$ P(Y=y)=\frac{\Gamma(y+r)}{\Gamma(r)y!}p^r(1-p)^y,\qquad y=0,1,2,\ldots. $$ Here $r$ is the size (also called shape), $p$ is the probability parameter, and $1-p$ is the complementary probability. With this convention, $Y$ counts failures before $r$ successes, and $$ E(Y)=\frac{r(1-p)}{p},\qquad \operatorname{Var}(Y)=\frac{r(1-p)}{p^2}. $$

Some sources instead name $q=1-p$ as the probability parameter. That is a valid convention, but the probability mass function and mean formulas must then use $q$ consistently. In particular, if $q$ is the probability of a counted failure, the mean is $rq/(1-q)$. Always check which event the software’s probability parameter represents.

From Size and Probability to Mean and Variance

Writing the mean as $\mu$ makes the conversions especially direct. For finite positive size $r$ and positive mean $\mu$, solving the mean equation for $p$ gives $p=r/(r+\mu)$. Substituting this into the variance gives the size form of the mean–variance relationship.

\mu=\frac{r(1-p)}{p} \quad\Longrightarrow\quad p=\frac{r}{r+\mu}, \qquad \operatorname{Var}(Y)=\mu+\frac{\mu^2}{r}.

This identity is the central conversion rule. The term $\mu^2/r$ is the excess variance above the Poisson variance $\mu$. Thus, at fixed mean, smaller $r$ gives greater variance, while larger $r$ approaches the Poisson limit. A reciprocal-size parameter is often more convenient when the target is directly to measure excess dispersion.

Definition: In the NB2 mean–variance parameterization, $$ \operatorname{Var}(Y)=\mu+\alpha\mu^2,\qquad \alpha\geq 0. $$ For finite negative binomial size, $\alpha=1/r>0$, so $r=1/\alpha$. The case $\alpha=0$ is the Poisson boundary, corresponding to $r\to\infty$, not a finite-size negative binomial distribution.

For NB2, the probability parameter can also be written directly in terms of $\mu$ and $\alpha$. When $\alpha>0$, substitution of $r=1/\alpha$ gives $p=1/(1+\alpha\mu)$. This expression uses the failure-counting convention above. If a program reports a size $\theta$, verify whether it means $r$; if so, $\alpha=1/\theta$. Do not assume that a parameter called “dispersion” is $\alpha$: some software uses its reciprocal.

Worked Example: Converting an NB2 Mean–Variance Pair

Suppose a hypothetical count distribution has mean $\mu=4$ and variance $V=10$. Assume the variance is represented by an NB2 model. Since $V>\mu$, a finite overdispersed negative binomial distribution can match these moments.

Find the dispersion. Using $V=\mu+\alpha\mu^2$, $$ \alpha=\frac{V-\mu}{\mu^2} =\frac{10-4}{4^2} =\frac{6}{16} =0.375. $$

Find the size. The reciprocal relationship gives $$ r=\frac{1}{\alpha}=\frac{1}{0.375}=2.667. $$

Find the probability parameter. $$ p=\frac{r}{r+\mu} =\frac{2.667}{2.667+4} \approx 0.400. $$ Therefore $1-p\approx0.600$. Checking the moments from size and probability gives $E(Y)=2.667(0.600)/0.400=4.00$ and $\operatorname{Var}(Y)=2.667(0.600)/(0.400)^2=10.0$, up to rounding. The same model can thus be described by $(\mu,\alpha)=(4,0.375)$ or approximately $(r,p)=(2.667,0.400)$.

NB1 and Its Dispersion Conventions

NB1 uses a different mean–variance pattern: its excess variance grows linearly with the mean rather than quadratically. One common convention writes the variance as $\mu(1+\alpha)$, with $\alpha\geq0$. Another reports the variance-to-mean ratio itself as a parameter, which equals $1+\alpha$. These numbers are not interchangeable without the stated conversion.

Definition: In the NB1 convention used here, $$ \operatorname{Var}(Y)=\mu(1+\alpha)=\mu+\alpha\mu,\qquad \alpha\geq0. $$ For $\alpha>0$, an NB1 distribution at a particular mean $\mu>0$ can be represented in size–probability form using $r=\mu/\alpha$ and $p=1/(1+\alpha)$. At $\alpha=0$, the distribution is at the Poisson boundary.

The NB1 conversion follows by equating its variance to the size-form variance. For $\alpha>0$, require $\mu+\mu^2/r=\mu+\alpha\mu$. Subtracting $\mu$ and dividing by $\mu>0$ yields $\mu/r=\alpha$, hence $r=\mu/\alpha$. Substitution into $p=r/(r+\mu)$ gives $p=1/(1+\alpha)$. Unlike NB2, this NB1 size depends on the mean. In a model where $\mu$ varies across observations, a single fixed size generally does not express this NB1 variance relationship across all those means.

If a source instead calls $\phi=V/\mu$ the dispersion or scale, then for NB1 $\phi=1+\alpha$ and $\phi\geq1$. The Poisson boundary is $\phi=1$, which corresponds to $\alpha=0$. Stating the variance formula alongside the parameter name prevents ambiguity.

Worked Example: NB1 Dispersion, Size, and Probability

Consider a hypothetical count with mean $\mu=8$ and variance $V=12$, and suppose NB1 is the chosen variance pattern. The mean is positive and $V>\mu$, so the estimated NB1 overdispersion parameter is positive.

Calculate $\alpha$. From $V=\mu(1+\alpha)$, $$ \alpha=\frac{V}{\mu}-1 =\frac{12}{8}-1 =0.5. $$ The variance-to-mean ratio is $\phi=V/\mu=1.5$, so indeed $\phi=1+\alpha$.

Convert to size and probability at this mean. $$ r=\frac{\mu}{\alpha}=\frac{8}{0.5}=16, \qquad p=\frac{1}{1+\alpha}=\frac{1}{1.5}\approx0.667. $$ Checking the size form gives $E(Y)=16(1-2/3)/(2/3)=8$ and $\operatorname{Var}(Y)=16(1/3)/(2/3)^2=12$.

At this particular mean, the same mean–variance pair also has an NB2 representation: $\alpha_{\mathrm{NB2}}=(12-8)/8^2=0.0625$, so $r=16$ and $p=2/3$. The distribution at this one mean is identical under the two representations. The model distinction is how variance changes as the mean changes: NB1 adds $\alpha\mu$, while NB2 adds $\alpha\mu^2$.

Recovering Dispersion From Moments and Handling Boundaries

Suppose a positive mean $\mu$ and a proposed variance $V$ are given. Algebraic inversion identifies the dispersion that would match those moments under each variance pattern. These inversions are valid only in their stated domains; they do not make every mean–variance pair representable by an overdispersed negative binomial model.

\text{NB1: }\quad \alpha=\frac{V}{\mu}-1 \quad(\mu>0,\; V\geq\mu), $$ $$ \text{NB2: }\quad \alpha=\frac{V-\mu}{\mu^2},\qquad r=\frac{\mu^2}{V-\mu} \quad(\mu>0,\; V>\mu).

For either pattern, $V=\mu$ is the Poisson boundary: the dispersion in the stated convention is zero, and the size tends to infinity. For NB2 specifically, the displayed finite-size formula for $r$ applies only when $V>\mu$; at equality, its denominator is zero, and the correct description is $\alpha=0,\ r\to\infty$. If $\mu=0$, do not use these inversion formulas: a nonnegative count with mean zero is zero with probability one, so the distribution is degenerate at zero. If $V<\mu$, the pair cannot be represented by either of these overdispersed NB1 or NB2 forms with nonnegative dispersion.

Worked Example: Checking the Poisson Boundary

Suppose a hypothetical count has $\mu=5$ and $V=5$. For NB2, the dispersion is $$ \alpha=\frac{V-\mu}{\mu^2} =\frac{5-5}{5^2} =0. $$ The model is therefore at the Poisson boundary. It is not correct to substitute into $r=\mu^2/(V-\mu)$ and report a finite size: the denominator is zero. In the size parameterization the matching limit is $r\to\infty$, with $p=r/(r+5)\to1$; the negative binomial converges to a Poisson distribution with mean 5. NB1 gives $\alpha=V/\mu-1=0$ as well. Both parameterizations agree at this boundary.

Reading Software Output and Diagnosing Mismatches

The same numerical output can mean different things under different conventions. A disciplined check begins with the model’s variance formula, not its parameter label. For example, a reported size of 5 under NB2 means $\alpha=1/5=0.2$, whereas a reported NB2 dispersion of 5 means size $r=1/5=0.2$. These imply very different variances.

Convention	Variance formula	Overdispersion parameter	Poisson boundary
NB1, additive coefficient	$\mu+\alpha\mu$	$\alpha\geq0$	$\alpha=0$
NB1, variance-to-mean ratio	$\phi\mu$	$\phi=1+\alpha\geq1$	$\phi=1$
NB2, reciprocal size	$\mu+\alpha\mu^2$	$\alpha=1/r\geq0$	$\alpha=0,\ r\to\infty$
NB2, size	$\mu+\mu^2/r$	$r>0$ finite for overdispersion	$r\to\infty$

A basic diagnostic is to reconstruct the variance from the reported mean and parameter, then check whether the result agrees with the model description. For NB1, use $V=\mu(1+\alpha)$; for NB2, use $V=\mu+\alpha\mu^2$, or $V=\mu+\mu^2/r$. Also check whether a claimed dispersion is in its valid range and whether the reported probability parameter agrees with the chosen probability convention.

These parameterizations describe different ways of allowing variance to exceed the mean. The Poisson model fixes $V=\mu$; NB1 has a constant variance-to-mean ratio at a given dispersion, whereas NB2’s ratio $V/\mu=1+\alpha\mu$ increases with the mean. The formulas alone do not determine which variance pattern is appropriate for a particular data set, and matching moments does not establish that the full distributional assumptions hold. In practice, observed variability, fitted-model checks, and the scientific structure of the data also matter.

Common Mistakes and Limitations

Treating size and dispersion as the same direction. Under NB2, increasing size $r$ decreases excess variance because that variance is $\mu^2/r$. Increasing reciprocal-size dispersion $\alpha$ increases it.
Using one formula for both NB1 and NB2. NB1 has excess variance $\alpha\mu$; NB2 has excess variance $\alpha\mu^2$. Their dispersion values cannot be compared directly without specifying the mean and convention.
Using the finite-size inversion at the Poisson boundary. When $V=\mu$, NB2 has $\alpha=0$ and $r\to\infty$; the formula for finite $r$ requires $V>\mu$.
Ignoring zero means or impossible moment pairs. The inversion formulas require $\mu>0$. A zero mean for a nonnegative count is degenerate at zero, and $V<\mu$ is outside these overdispersed NB1/NB2 moment patterns.
Assuming matching moments identify the model. A single mean–variance pair can be represented by both NB1 and NB2, as in the worked example. Their difference concerns how variance behaves over a range of means, not necessarily the distribution at one mean.

The negative binomial family is a distributional assumption, not merely a formula for variance. The parameter conversions establish what each parameterization implies; they do not guarantee that a particular data set follows the negative binomial probability mass function or that one parameterization is adequate across all predictor values.

Check Your Understanding

Use the stated conventions: NB1 has $V=\mu(1+\alpha)$, and NB2 has $V=\mu+\alpha\mu^2$ with $r=1/\alpha$ when $\alpha>0$. The solutions follow each question.

A hypothetical NB2 count has $\mu=6$ and size $r=3$. Find $\alpha$, $p$, and the variance.
A hypothetical NB1 count has $\mu=10$ and variance $15$. Find $\alpha$, $\phi=V/\mu$, and the size–probability representation at this mean.
For NB2, what model corresponds to $\mu=3$ and $V=3$? Why is a finite size not obtained?
Can either overdispersed NB1 or NB2 match $\mu=4$ and $V=3$ with a nonnegative dispersion? Explain.
At $\mu=2$, an NB2 model has $\alpha=0.5$. Find its variance. What does the same $\alpha$ imply for NB1 variance at that mean?

Solutions. (1) $\alpha=1/r=1/3$, $p=r/(r+\mu)=3/9=1/3$, and $V=\mu+\mu^2/r=6+36/3=18$. (2) $\alpha=15/10-1=0.5$, and $\phi=15/10=1.5$. Then $r=\mu/\alpha=20$, $p=1/(1+\alpha)=2/3$; these give mean 10 and variance $20(1/3)/(2/3)^2=15$. (3) This is the Poisson boundary, with $\alpha=0$ and $r\to\infty$. The finite-size expression divides by $V-\mu=0$, so it does not give a finite size. (4) No. Both NB1 and NB2 with nonnegative dispersion have $V\geq\mu$, but here $3<4$. (5) Under NB2, $V=2+0.5(2^2)=4$. Under NB1, $V=2(1+0.5)=3$. The equal dispersion numbers produce different variances because the parameterizations use different mean–variance formulas.

Convention	Variance formula	Overdispersion parameter	Poisson boundary
NB1, additive coefficient	\(\mu+\alpha\mu\)	\(\alpha\geq0\)	\(\alpha=0\)
NB1, variance-to-mean ratio	\(\phi\mu\)	\(\phi=1+\alpha\geq1\)	\(\phi=1\)
NB2, reciprocal size	\(\mu+\alpha\mu^2\)	\(\alpha=1/r\geq0\)	\(\alpha=0,\ r\to\infty\)
NB2, size	\(\mu+\mu^2/r\)	\(r>0\) finite for overdispersion	\(r\to\infty\)