Moments Of The Negative Binomial Count Model

Why the Moments Matter

A count model specifies more than the expected number of observed molecules. It also specifies how much counts vary around that expectation. For single-cell RNA sequencing, two genes or cell groups can have the same expected count but different variability. The negative binomial model represents this extra variability through a dispersion parameter. Understanding its moments makes the role of that parameter precise.

This tutorial uses the mean-dispersion parameterization. The mean is $\mu$, and the nonnegative dispersion parameter is $\phi$. The central result is that the variance is $\mu+\phi\mu^2$: the Poisson contribution is $\mu$, while the added term increases with both the mean and dispersion.

Definition: In the mean-dispersion parameterization, a negative binomial random variable $X$ has mean $\mu>0$, dispersion $\phi>0$, and probability mass function $$ P(X=x)=\frac{\Gamma(x+r)}{\Gamma(r)x!}\,\pi^r(1-\pi)^x,\qquad x=0,1,2,\ldots, $$ where $r=1/\phi$ is the size parameter and $\pi=r/(r+\mu)=1/(1+\phi\mu)$. Under this convention, $X$ counts failures before $r$ successes. Its mean is $\mu$ and its variance is $\mu+\phi\mu^2$.

Here $\Gamma(\cdot)$ is the gamma function, which extends the factorial: for a positive integer $k$, $\Gamma(k)=(k-1)!$. The probability parameter $\pi$ is included to make the link with another common negative binomial form explicit. The count $X$ is not the number of successes; it is the number of failures before the specified number of successes. This convention matters because negative binomial texts sometimes assign the probability parameter to the other outcome.

The variance is measured in squared count units, while the mean is measured in count units. Dispersion $\phi$ is dimensionless in this model. A larger $\phi$ means a larger variance when the mean is held fixed; it does not mean that the expected count itself has increased.

Deriving the Mean and Variance

A useful derivation uses gamma-Poisson mixing. Conditional on a latent rate $\Lambda$, let $X$ be Poisson with rate $\Lambda$. Let that rate vary according to a gamma distribution with shape $1/\phi$ and scale $\mu\phi$. The scale convention means that a gamma variable with shape $a$ and scale $b$ has mean $ab$ and variance $ab^2$.

Model assumptions: For one gene in a specified population of comparable cells, assume $\Lambda\sim\operatorname{Gamma}(1/\phi,\text{ scale } \mu\phi)$ and $X\mid\Lambda\sim\operatorname{Poisson}(\Lambda)$. Thus, conditional on the latent rate, the count has Poisson mean and variance $\Lambda$. The negative binomial moments follow from this mixture; they do not require counts from different cells to be independent.

Under these assumptions, the gamma distribution gives $E[\Lambda]=\mu$ and $\operatorname{Var}(\Lambda)=\phi\mu^2$. Conditional on $\Lambda$, $E[X\mid\Lambda]=\Lambda$ and $\operatorname{Var}(X\mid\Lambda)=\Lambda$. Apply the law of total expectation to obtain the marginal mean:

E[X]=E\{E[X\mid\Lambda]\}=E[\Lambda]=\mu.

For the marginal variance, the law of total variance separates the average conditional variance from the variance of the conditional mean:

\begin{aligned} \operatorname{Var}(X) &=E\{\operatorname{Var}(X\mid\Lambda)\} +\operatorname{Var}\{E[X\mid\Lambda]\}\\ &=E[\Lambda]+\operatorname{Var}(\Lambda)\\ &=\mu+\phi\mu^2. \end{aligned}

The first term, $\mu$, is the variation expected from Poisson sampling at a fixed rate. The second, $\phi\mu^2$, arises because the rate itself varies among observations in the mixture. It is nonnegative, so the negative binomial variance is at least as large as the Poisson variance with the same mean.

Interpreting Dispersion at a Fixed Mean

The variance-to-mean ratio, also called the Fano factor, gives a compact comparison with a Poisson model:

\frac{\operatorname{Var}(X)}{E[X]} =\frac{\mu+\phi\mu^2}{\mu} =1+\phi\mu.

A Poisson variable has variance-to-mean ratio 1. In the negative binomial model, the ratio is greater than 1 when $\phi>0$. Importantly, the ratio depends on both $\phi$ and $\mu$: the same dispersion value does not imply the same variance-to-mean ratio at every mean.

Worked Example: Converting Dispersion to Size and Computing Moments

Suppose a hypothetical gene has mean count $\mu=4$ molecules per cell and dispersion $\phi=0.25$. Find its negative binomial size, probability parameter, variance, and variance-to-mean ratio.

The size parameter is the reciprocal of dispersion:

r=\frac{1}{\phi}=\frac{1}{0.25}=4.

The success probability under the stated convention is:

\pi=\frac{1}{1+\phi\mu} =\frac{1}{1+(0.25)(4)} =\frac{1}{2}=0.5.

The mean is $\mu=4$ molecules per cell by parameterization. The variance is:

\operatorname{Var}(X)=\mu+\phi\mu^2 =4+(0.25)(4^2) =4+4=8\ \text{molecules}^2.

Consequently, the variance-to-mean ratio is $8/4=2$. The count variability exceeds that of a Poisson model with mean 4, whose variance would be 4. These are model-implied moments, not claims about a particular observed data set.

Worked Example: Changing Dispersion While Holding the Mean Fixed

Compare two hypothetical negative binomial models for the same gene, both with mean $\mu=3$ molecules per cell. Let the first have $\phi=0.1$ and the second have $\phi=0.5$. Calculate each variance and explain the difference.

For the first model:

\operatorname{Var}(X)=3+(0.1)(3^2)=3+0.9=3.9\ \text{molecules}^2.

For the second model:

\operatorname{Var}(X)=3+(0.5)(3^2)=3+4.5=7.5\ \text{molecules}^2.

The difference is $7.5-3.9=3.6\ \text{molecules}^2$. Both models have the same expected count of 3 molecules per cell; the model with larger dispersion assigns more variation around that mean. Its variance-to-mean ratio is $7.5/3=2.5$, compared with $3.9/3=1.3$ for the first model.

A Second Moment Identity

The factorial second moment provides another way to check the variance formula. For any count $X$,

\operatorname{Var}(X)=E[X(X-1)]+E[X]-\{E[X]\}^2.

Under the gamma-Poisson model, a Poisson variable has conditional factorial second moment $E[X(X-1)\mid\Lambda]=\Lambda^2$. Therefore,

E[X(X-1)] =E[\Lambda^2] =\operatorname{Var}(\Lambda)+\{E[\Lambda]\}^2 =\phi\mu^2+\mu^2 =(1+\phi)\mu^2.

Substitution gives $\operatorname{Var}(X)=(1+\phi)\mu^2+\mu-\mu^2=\mu+\phi\mu^2$, agreeing with the total-variance derivation. This identity is also useful when reasoning about count variability: the negative binomial model has a factorial second moment larger than $\mu^2$ when $\phi>0$.

Worked Example: Checking the Variance With a Factorial Moment

For a hypothetical count with $\mu=2$ and $\phi=0.5$, calculate the factorial second moment and recover the variance.

First, the factorial second moment is:

E[X(X-1)]=(1+\phi)\mu^2 =(1+0.5)(2^2)=1.5(4)=6.

Using the identity, the variance is:

\operatorname{Var}(X) =E[X(X-1)]+E[X]-\{E[X]\}^2 =6+2-2^2=4.

Direct substitution confirms the result: $\mu+\phi\mu^2=2+(0.5)(4)=4$ squared count units. The two calculations agree.

Parameterizations and Comparisons

The size-probability form and the mean-dispersion form describe the same distribution when their parameters are converted consistently. With the failure-count convention used here, the conversion is:

r=\frac{1}{\phi},\qquad \pi=\frac{r}{r+\mu}=\frac{1}{1+\phi\mu},\qquad \mu=\frac{r(1-\pi)}{\pi}.

The corresponding variance in size notation is $\mu+\mu^2/r$, because $1/r=\phi$. A reported size parameter and a reported dispersion parameter therefore move in opposite directions: larger $r$ means smaller $\phi$, and hence less extra variance at a fixed mean. Always check the probability convention as well as the names of the parameters before comparing formulas.

Model or parameter	Mean	Variance	Interpretation at fixed mean
Poisson	$\mu$	$\mu$	Variance equals the mean.
Negative binomial, dispersion form	$\mu$	$\mu+\phi\mu^2$	Positive $\phi$ adds a quadratic variance term.
Negative binomial, size form	$\mu$	$\mu+\mu^2/r$	Larger $r$ reduces the extra variance.

For a fixed finite $\mu$, letting $\phi$ approach zero makes the extra variance term vanish, and the negative binomial approaches the Poisson model. This limiting relationship does not mean that every count data set should be assigned a negative binomial model. The distributional assumptions and the observed data still need to be considered.

Assumptions, Diagnostics, and Pitfalls

The moment derivation depends on the stated gamma-Poisson mixture: the conditional count is Poisson, and the latent rate follows the specified gamma distribution. In particular, a negative binomial variance larger than the mean is a consequence of this model for a single count distribution. It does not identify the biological or technical source of the rate variation, and it does not by itself establish that the model adequately describes all features of observed counts.

As a basic diagnostic, compare empirical means and variances for a clearly defined set of comparable observations with the pattern predicted by the model. A variance far below the mean conflicts with the ordinary negative binomial moment relation for positive $\phi$. A variance above the mean is compatible with it, but does not prove the negative binomial distribution is appropriate: many count distributions can share similar first two moments. The grouping used for such a comparison matters; combining observations with substantially different expected counts can create variability that is not captured by interpreting one common mean as if it applied to every observation.

The most common parameterization error is to mistake $\phi$ for the added variance itself. The added term is $\phi\mu^2$, so changing the mean changes the contribution even if dispersion is unchanged. Another error is to say that larger dispersion raises the mean. In this parameterization the mean is $\mu$; at fixed $\mu$, increasing $\phi$ changes the variance, not the expected count.

The Poisson model is a useful comparison when variance equals the mean, while the negative binomial allows extra variance through rate mixing. Neither variance formula alone is a complete model check: a match in mean and variance does not ensure that the distribution’s probabilities, including its probability of zero, match the data. Moment calculations are a foundation for understanding variability, not a substitute for assessing the full count distribution.

Key takeaway: Under the mean-dispersion negative binomial model, $E[X]=\mu$ and $\operatorname{Var}(X)=\mu+\phi\mu^2$. At a fixed mean, dispersion controls the additional quadratic component of variance; equivalently, the size parameter is its reciprocal.

Check Your Understanding

Answer each question, then compare with the solutions below.

A negative binomial count has $\mu=5$ and $\phi=0.2$. What are its variance and variance-to-mean ratio?
At a fixed mean, what happens to the variance when $\phi$ increases? Does the mean change?
A software output reports size $r=10$. What is the corresponding dispersion, and what is the extra variance term in terms of $\mu$?
Why does a variance greater than the mean not, by itself, confirm that a negative binomial distribution fits observed counts?

Solutions. (1) $\operatorname{Var}(X)=5+(0.2)(25)=10$ squared count units, and the ratio is $10/5=2$. (2) The variance increases by $\mu^2$ times the increase in $\phi$; the mean remains $\mu$ because it is held fixed. (3) $\phi=1/r=0.1$, so the extra variance term is $0.1\mu^2$. (4) Different distributions can have a variance above the mean or share similar first two moments; that comparison does not establish that the negative binomial’s full probability distribution matches the observations.

Model or parameter	Mean	Variance	Interpretation at fixed mean
Poisson	\(\mu\)	\(\mu\)	Variance equals the mean.
Negative binomial, dispersion form	\(\mu\)	\(\mu+\phi\mu^2\)	Positive \(\phi\) adds a quadratic variance term.
Negative binomial, size form	\(\mu\)	\(\mu+\mu^2/r\)	Larger \(r\) reduces the extra variance.