Deriving The Negative Binomial Variance

Why Derive the Variance From the Probability-Generating Function?

RNA-seq counts are nonnegative integers, and a negative binomial model allows their variance to exceed their mean. Deriving that variance from the probability-generating function (PGF) makes the role of the dispersion parameter explicit. It also shows why the Poisson model is the limiting case when dispersion tends to zero. These relationships are useful checks on model parameterizations and on interpretations of count variability.

Use the mean-dispersion parameterization: $X$ is a negative binomial count with mean $\mu$ and dispersion $\phi$, where $\mu>0$ and $\phi>0$. Its PGF is $G(s)=E(s^X)$. In the equivalent size-probability parameterization, the size is $r=1/\phi$ and the success probability is $p=r/(r+\mu)$. The PGF form below is the one used for the derivation.

Definition: For a nonnegative integer-valued random variable $X$, its probability-generating function is $G(s)=E(s^X)=\sum_{k=0}^{\infty}P(X=k)s^k$, for values of $s$ where the sum exists. For the negative binomial model with mean $\mu$ and dispersion $\phi>0$, $$ G(s)=\left[1+\phi\mu(1-s)\right]^{-1/\phi}. $$

The mean-dispersion parameterization uses $\operatorname{Var}(X)=\mu+\phi\mu^2$. The purpose here is to derive that result rather than assume it. The derivation uses the standard PGF identities $G'(1)=E(X)$ and $G''(1)=E[X(X-1)]$, which follow by differentiating each term $P(X=k)s^k$. For a negative binomial distribution with finite $\mu$ and $\phi>0$, these moments are finite, so the derivatives at 1 give the corresponding moments.

Differentiate the PGF

Write $A(s)=1+\phi\mu(1-s)$, so that $G(s)=A(s)^{-1/\phi}$ and $A'(s)=-\phi\mu$. The chain rule gives the first derivative:

G'(s)=\left(-\frac{1}{\phi}\right)A(s)^{-1/\phi-1}(-\phi\mu) =\mu A(s)^{-1/\phi-1}.

Differentiating once more,

G''(s)=\mu\left(-\frac{1}{\phi}-1\right)A(s)^{-1/\phi-2}(-\phi\mu) =\mu^2(1+\phi)A(s)^{-1/\phi-2}.

Since $A(1)=1$, evaluating the derivatives at $s=1$ gives $G'(1)=\mu$ and $G''(1)=\mu^2(1+\phi)$. The second derivative is a factorial moment, not the raw second moment: $G''(1)=E[X(X-1)]$. Because $X^2=X(X-1)+X$, the raw second moment is $E(X^2)=G''(1)+G'(1)$. Therefore,

\begin{aligned} \operatorname{Var}(X) &=E(X^2)-[E(X)]^2\\ &=G''(1)+G'(1)-[G'(1)]^2\\ &=\mu^2(1+\phi)+\mu-\mu^2\\ &=\mu+\phi\mu^2. \end{aligned}

The term $\mu$ is the Poisson-level variance, while $\phi\mu^2$ is the additional variance associated with negative binomial dispersion. For positive $\mu$ and $\phi$, the variance exceeds the mean. This is the overdispersion relative to a Poisson count with the same mean.

Worked Example: Calculating the Variance From the Derivatives

Suppose a hypothetical gene count is modeled with mean $\mu=10$ reads per sample and dispersion $\phi=0.2$. Use the PGF derivatives to calculate its variance.

At $s=1$, the first derivative is $G'(1)=\mu=10$, so the expected count is 10 reads. The second derivative is $G''(1)=\mu^2(1+\phi)=10^2(1.2)=120$. Thus $E(X^2)=G''(1)+G'(1)=120+10=130$ reads squared. Subtracting the squared mean gives $130-10^2=30$ reads squared.

Equivalently, $\operatorname{Var}(X)=10+0.2(10^2)=10+20=30$ reads squared. The standard deviation is $\sqrt{30}\approx5.477$ reads. The variance is larger than the mean because the dispersion is positive.

What the Dispersion Parameter Changes

At a fixed mean, the negative binomial variance increases with $\phi$, since its derivative with respect to dispersion is $\mu^2>0$. The increase is especially pronounced at larger means because the extra term is quadratic in $\mu$. This statement concerns the model variance, not a guarantee that every finite data set will show an empirical variance larger than its empirical mean.

Worked Example: Comparing Two Dispersion Values

Consider two hypothetical negative binomial models for counts with the same mean, $\mu=8$ reads, but with dispersions $\phi=0.05$ and $\phi=0.20$. Calculate and compare their variances and standard deviations.

For $\phi=0.05$, $\operatorname{Var}(X)=8+0.05(8^2)=8+3.2=11.2$ reads squared, and the standard deviation is $\sqrt{11.2}\approx3.347$ reads.

For $\phi=0.20$, $\operatorname{Var}(X)=8+0.20(8^2)=8+12.8=20.8$ reads squared, and the standard deviation is $\sqrt{20.8}\approx4.561$ reads. Both models have the same expected count of 8 reads, but the model with greater dispersion allows more variability around that mean.

The Poisson Limit as Dispersion Approaches Zero

The PGF itself shows the limiting relationship. For fixed $\mu$ and $s$, take the logarithm of the negative binomial PGF:

\log G(s)=-\frac{1}{\phi}\log\!\left(1+\phi\mu(1-s)\right).

Using the standard limit $\lim_{u\to0}\log(1+u)/u=1$, set $u=\phi\mu(1-s)$. For $s$ fixed and $\phi\to0^+$, this gives $\log G(s)\to-\mu(1-s)=\mu(s-1)$. Exponentiating, the PGF tends to $\exp(\mu(s-1))$, which is the PGF of a Poisson random variable with mean $\mu$. The variance formula has the matching limit:

\lim_{\phi\to0^+}\left(\mu+\phi\mu^2\right)=\mu.

Thus, as dispersion tends to zero while the mean stays fixed, both the negative binomial PGF and its variance approach their Poisson counterparts. The limiting argument does not say that a negative binomial model with any positive dispersion is exactly Poisson; it identifies the boundary case.

Worked Example: Checking the PGF Limit Numerically

For a hypothetical count model, let $\mu=4$, evaluate the PGF at $s=0.5$, and compare a small-dispersion negative binomial model with the limiting Poisson value.

With $\phi=0.001$, the negative binomial PGF is $G(0.5)=[1+0.001(4)(1-0.5)]^{-1000}=1.002^{-1000}$. Taking logs gives $-1000\log(1.002)\approx-1.99800$, so $G(0.5)\approx e^{-1.99800}\approx0.135606$.

The limiting Poisson PGF at the same argument is $\exp(4(0.5-1))=e^{-2}\approx0.135335$. These values are close, as expected for small dispersion. Their small difference is consistent with a positive, rather than zero, dispersion. The variances also approach one another: the negative binomial variance is $4+0.001(4^2)=4.016$ counts squared, while the limiting Poisson variance is 4 counts squared.

Assumptions, Checks, and Common Pitfalls

The derivation assumes the stated negative binomial PGF, a finite nonnegative count, finite mean $\mu$, and positive dispersion $\phi$. The Poisson limit holds with $\mu$ fixed as $\phi$ tends to zero. The formula is a model implication: it does not establish that observed counts follow a negative binomial distribution, nor does it account by itself for covariates or differences in sample exposure.

A useful check is whether the variance formula agrees with the parameterization being used. In size notation, $r=1/\phi$, so the same result is $\operatorname{Var}(X)=\mu+\mu^2/r$. If a source instead uses a different parameter called “dispersion,” translate its definition before comparing formulas. The symbol alone does not determine whether a parameter is $\phi$, $r$, or another quantity.

Confusing the factorial moment with the raw second moment. $G''(1)$ equals $E[X(X-1)]$, not $E(X^2)$. Add $G'(1)=E(X)$ to obtain the raw second moment before subtracting the squared mean.
Dropping the extra-variance term. The negative binomial variance is $\mu+\phi\mu^2$, not just $\phi\mu^2$. The Poisson-level term remains present.
Expecting every sample variance to exceed its sample mean. The model specifies a population variance. Finite samples fluctuate, so an empirical variance can fall below the empirical mean even when the negative binomial model is appropriate.
Treating a small dispersion as exactly zero. Small positive $\phi$ gives a variance near, but not equal to, $\mu$; exact equality occurs at the Poisson boundary.

A variance-versus-mean comparison can reveal patterns inconsistent with a proposed count model, but it is not a definitive diagnostic on its own. Sampling variability, a small number of replicates, and variation among samples can affect the observed variance. The Poisson model is simpler and fixes variance equal to mean; the negative binomial model permits extra variance through $\phi$. Neither model should be selected from one variance comparison without considering the data and study design. The derivation establishes the mathematical consequence of the model, not the adequacy of that model for every gene.

Key takeaway: Differentiating the negative binomial PGF gives $E(X)=\mu$ and $E[X(X-1)]=\mu^2(1+\phi)$. Converting the factorial moment to a raw second moment yields $\operatorname{Var}(X)=\mu+\phi\mu^2$, which tends to the Poisson variance $\mu$ as $\phi\to0^+$.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For a negative binomial count with $\mu=5$ and $\phi=0.4$, what are $G'(1)$, $G''(1)$, and the variance?
Why is $G''(1)$ not itself the variance?
At fixed $\mu=12$, does increasing $\phi$ increase or decrease the model variance? By how much does the variance change when $\phi$ rises by 0.1?
State the limiting PGF and variance as $\phi\to0^+$ with $\mu$ fixed.

Solutions. (1) $G'(1)=5$, $G''(1)=5^2(1+0.4)=35$, and $\operatorname{Var}(X)=35+5-5^2=15$ counts squared. (2) $G''(1)=E[X(X-1)]$; the raw second moment is $G''(1)+G'(1)$, and variance then subtracts $[G'(1)]^2$. (3) It increases, because $\operatorname{Var}(X)=\mu+\phi\mu^2$. A rise of 0.1 in $\phi$ increases variance by $0.1(12^2)=14.4$ counts squared. (4) The PGF tends to $\exp(\mu(s-1))$, the Poisson PGF, and the variance tends to $\mu$ counts squared.