NB1 and NB2 Variance Functions

Why Two Negative-Binomial Variance Functions?

Count data are often more variable than a Poisson model allows. A negative-binomial model accommodates extra variation, but “negative-binomial variance” does not identify a single pattern: the variance may grow linearly or quadratically with the mean. The labels NB1 and NB2 distinguish these two variance functions.

The difference matters when the mean varies across observations. Under NB1, the excess variance relative to a Poisson count grows in proportion to the mean. Under NB2, it grows in proportion to the square of the mean. Thus the models make different predictions about how dispersion changes as expected counts increase.

A Negative-Binomial Mean–Size Form

Use the negative-binomial parameterization in which \(Y\) takes nonnegative integer values, \(r>0\) is its size parameter, and \(0<p<1\). Its probability mass function is

P(Y=y)=\frac{\Gamma(y+r)}{\Gamma(r)y!}p^r(1-p)^y,\qquad y=0,1,2,\ldots.

In this parameterization, \(E[Y]=r(1-p)/p\). The following derivation establishes the variance formula we will use for both NB1 and NB2.

Result: If \(Y\) has the negative-binomial mass function above, then \(E[Y]=r(1-p)/p\) and \(\operatorname{Var}(Y)=r(1-p)/p^2=E[Y]+E[Y]^2/r\).

To prove the result, the probability-generating function is \(G(s)=E[s^Y]\). For \(s\) in a neighborhood of 1, the negative-binomial series gives

G(s)=\left(\frac{p}{1-(1-p)s}\right)^r.

The power series converges in a neighborhood of \(s=1\), since its radius of convergence is \(1/(1-p)>1\). Therefore its derivatives at 1 give the first two factorial moments: \(G'(1)=E[Y]\) and \(G''(1)=E[Y(Y-1)]\). Differentiating,

G'(s)=rp^r(1-p)\{1-(1-p)s\}^{-r-1},

G''(s)=r(r+1)p^r(1-p)^2\{1-(1-p)s\}^{-r-2}.

At \(s=1\), these become \(G'(1)=r(1-p)/p\) and \(G''(1)=r(r+1)(1-p)^2/p^2\). Since \(E[Y^2]=E[Y(Y-1)]+E[Y]\), subtracting the squared mean yields

\begin{aligned} \operatorname{Var}(Y) &= \frac{r(r+1)(1-p)^2}{p^2}+\frac{r(1-p)}{p} -\left\{\frac{r(1-p)}{p}\right\}^2\\ &= \frac{r(1-p)}{p^2} = E[Y]+\frac{E[Y]^2}{r}. \end{aligned}

This identity makes the role of the size parameter explicit: at a fixed mean, a smaller \(r\) means a larger variance above the Poisson value.

NB2: Fixed Size and a Quadratic Term

Write the mean as \(\mu>0\), and hold the size parameter \(r=\kappa>0\) fixed as \(\mu\) changes. The mean equation \( \mu=\kappa(1-p)/p\) gives \(p=\kappa/(\kappa+\mu)\). Substituting into the variance identity gives the NB2 variance function.

Definition: The NB2 parameterization has mean \(\mu>0\), fixed size \(\kappa>0\), and \(p=\kappa/(\kappa+\mu)\). Its variance is \(\operatorname{Var}(Y)=\mu+\mu^2/\kappa=\mu+\alpha\mu^2\), where \(\alpha=1/\kappa>0\) is the NB2 dispersion parameter.

The extra variance beyond the Poisson value \(\mu\) is \(\alpha\mu^2\). It is quadratic in the mean, while the variance-to-mean ratio is \(1+\alpha\mu\), which increases with \(\mu\). The parameter \(\alpha\) determines the rate of that increase; equivalently, a larger \(\kappa\) means less extra variation.

Worked Example: Calculating NB2 Variance

Suppose a negative-binomial count has mean \(\mu=6\) and fixed size \(\kappa=4\). Find its probability parameter, variance, and variance-to-mean ratio.

From \(p=\kappa/(\kappa+\mu)\),

p=\frac{4}{4+6}=0.4.

The mean check using the negative-binomial parameterization is

r\frac{1-p}{p} =4\frac{0.6}{0.4}=6.

The variance is

\operatorname{Var}(Y) =6+\frac{6^2}{4} =6+9=15.

Thus the variance-to-mean ratio is \(15/6=2.5\), also obtained as \(1+\mu/\kappa=1+6/4=2.5\). The excess over the Poisson variance is 9.

NB1: Size Changes with the Mean

To obtain a variance proportional to the mean, the size parameter cannot remain fixed. Let \(\phi>0\) and set the negative-binomial size to \(r=\mu/\phi\). Choosing \(p=1/(1+\phi)\) then gives mean \(\mu\), because \(r(1-p)/p=(\mu/\phi)\phi=\mu\). The variance identity gives

\operatorname{Var}(Y) =\mu+\frac{\mu^2}{\mu/\phi} =\mu(1+\phi).

Definition: The NB1 parameterization has mean \(\mu>0\), dispersion parameter \(\phi>0\), size \(r=\mu/\phi\), and \(p=1/(1+\phi)\). Its variance is \(\operatorname{Var}(Y)=\mu(1+\phi)=\mu+\phi\mu\).

Here the variance-to-mean ratio is \(1+\phi\), constant across mean levels. The extra variance is \(\phi\mu\), so it grows linearly with the mean. Unlike NB2, NB1 achieves this pattern by letting the size vary with the mean: higher means have larger size \(r=\mu/\phi\), with the probability parameter \(p\) held fixed.

Worked Example: Checking an NB1 Parameterization

Let the mean be \(\mu=6\) and the NB1 dispersion be \(\phi=0.5\). Calculate the negative-binomial size and probability parameter, then verify the mean and variance.

The NB1 definitions give

r=\frac{\mu}{\phi}=\frac{6}{0.5}=12, \qquad p=\frac{1}{1+\phi}=\frac{1}{1.5}=\frac{2}{3}.

The mean is

r\frac{1-p}{p} =12\frac{1/3}{2/3}=6.

The variance calculated from the negative-binomial mean–size identity is

\operatorname{Var}(Y) =6+\frac{6^2}{12} =6+3=9.

The NB1 variance function gives the same result: \(6(1+0.5)=9\). Its variance-to-mean ratio is \(9/6=1.5=1+\phi\).

Comparing the Implied Dispersion

For either model, define the excess variance as the variance minus the mean, and the variance-to-mean ratio as the variance divided by the mean. These summaries make the difference between the models especially direct.

Feature	NB1	NB2
Variance	\(\mu+\phi\mu\)	\(\mu+\alpha\mu^2\)
Excess above the mean	\(\phi\mu\)	\(\alpha\mu^2\)
Variance-to-mean ratio	\(1+\phi\)	\(1+\alpha\mu\)
Size as mean changes	\(r=\mu/\phi\)	\(r=\kappa=1/\alpha\), fixed

For fixed positive \(\phi\) and \(\alpha\), the excess-variance terms are equal when \(\phi\mu=\alpha\mu^2\). Since \(\mu>0\), this occurs at \(\mu=\phi/\alpha\). Below that mean, NB1 has the greater excess variance; above it, NB2 does. At the crossing mean, the two variances agree. This comparison holds for the specified parameter values; it does not assert that \(\phi\) and \(\alpha\) have the same numerical meaning.

Worked Example: Comparing Dispersion Across Means

Compare NB1 with \(\phi=0.4\) and NB2 with \(\alpha=0.2\) at means \(0.5\), \(2\), and \(8\). For each mean, calculate both variances.

Under NB1, \(V_1(\mu)=\mu+0.4\mu=1.4\mu\). Under NB2, \(V_2(\mu)=\mu+0.2\mu^2\). At \(\mu=0.5\),

V_1(0.5)=1.4(0.5)=0.7,\qquad V_2(0.5)=0.5+0.2(0.5)^2=0.55.

At \(\mu=2\),

V_1(2)=1.4(2)=2.8,\qquad V_2(2)=2+0.2(2)^2=2.8.

At \(\mu=8\),

V_1(8)=1.4(8)=11.2,\qquad V_2(8)=8+0.2(8)^2=20.8.

The equality occurs at \(\mu=\phi/\alpha=0.4/0.2=2\), matching the calculations. Below 2, NB1 has greater variance; above 2, NB2 has greater variance. The Poisson contribution \(\mu\) is present in both, so the changing comparison comes from the excess-variance terms.

Why the Distinction Matters

The variance function describes how the model’s variability changes with the expected count. If observations have very different means, assuming a constant variance-to-mean ratio (NB1) is not equivalent to allowing that ratio to rise with the mean (NB2). The choice therefore affects the amount of variability the model assigns to observations at different mean levels.

The variance function alone does not specify every feature of a probability distribution. Here, NB1 and NB2 are particular negative-binomial parameterizations, not just labels for any count distribution that happens to have the stated variance. The size and probability parameters above show how each variance pattern arises within the negative-binomial family.

Common Mistake

Wrong reasoning: “Both models are negative binomial, so their variance is \(\mu+\mu^2/r\) with the same fixed \(r\); NB1 and NB2 are just two names for the same formula.”
Correction: The general identity is \(\mu+\mu^2/r\), but whether \(r\) is fixed matters. NB2 keeps \(r=\kappa\) fixed, giving \(\mu+\mu^2/\kappa\). NB1 sets \(r=\mu/\phi\), giving \(\mu+\phi\mu\). Substituting the size rule before interpreting the variance is essential.
Another caution: Do not interpret \(\phi\) and \(\alpha\) as interchangeable dispersion values. They multiply different functions of the mean: \(\phi\mu\) for NB1 and \(\alpha\mu^2\) for NB2.

Key takeaway: NB1 has variance \(\mu(1+\phi)\) and a constant variance-to-mean ratio. NB2 has variance \(\mu+\alpha\mu^2\) and a variance-to-mean ratio that increases with the mean. Their contrast is the mean-dependence of excess variation.

Check Your Understanding

Use the variance functions and parameterizations above to answer these questions.

For NB2 with mean \(\mu\) and size \(\kappa\), derive \(p\) from the negative-binomial mean equation.
An NB1 model has \(\mu=10\) and \(\phi=0.3\). What are its size, probability parameter, variance, and variance-to-mean ratio?
For NB2 with \(\mu=5\) and \(\alpha=0.4\), calculate the variance and excess variance.
Given NB1 dispersion \(\phi=0.6\) and NB2 dispersion \(\alpha=0.2\), at what positive mean do their excess-variance terms agree?
Explain why a fixed negative-binomial size gives the NB2, rather than the NB1, variance function.

What You'll Learn