Poisson And Negative Binomial Parameterizations

Why Parameterization Matters

A count distribution can be written using different parameters that describe the same probability model. In RNA-seq, one common form makes the expected count and dispersion explicit; another uses a size parameter and a probability. The formulas can look quite different, so translating carefully matters: confusing a size with its reciprocal, or using the wrong probability convention, changes the implied variance even when the mean looks right.

The earlier material on the negative binomial count model introduced the idea that a count can have variance greater than its mean. Here the goal is narrower: define two common parameterizations, show precisely how they correspond, and use each to read off the variance. The counts discussed below are nonnegative integers, and all numerical data are hypothetical.

Two Negative Binomial Parameterizations

Definition: In the mean-dispersion parameterization, write $Y\sim NB(\mu,\alpha)$, where $\mu>0$ is the mean and $\alpha>0$ is the dispersion. Its variance is $\mu+\alpha\mu^2$. In the size-probability parameterization, write $Y\sim NB(r,p)$, where $r>0$ is the size and $0<p<1$ is the probability of success. Here $Y$ counts failures before $r$ successes, so $$ P(Y=y)=\binom{y+r-1}{y}p^r(1-p)^y,\qquad y=0,1,2,\ldots, $$ with mean $r(1-p)/p$ and variance $r(1-p)/p^2$. The probability $p$ in this parameterization is a distribution parameter, not the population proportion notation used elsewhere.

The “failures before successes” convention fixes which probability appears in the mass function. In particular, the exponent on $p$ is $r$, while the exponent on $1-p$ is the observed count $y$. Other references may define a negative binomial count using a different convention; check that convention before comparing formulas.

In the mean-dispersion form, the variance is the mean plus a quadratic term in the mean. Thus, for positive dispersion, the variance exceeds the mean when $\mu>0$. In the size-probability form, $r$ controls the shape and $p$ sets the probability of success under the stated convention. Neither parameter by itself is the expected count; the mean depends on both.

Deriving the Parameter Map

To translate between forms, match both the mean and the variance. Matching only the means determines the probability $p$ once $r$ and $\mu$ are specified, but it does not determine the relationship between $r$ and $\alpha$. That second relationship comes from matching the variances.

Match the means.
Set $\mu=r(1-p)/p$. Solving for $p$ gives $p=r/(r+\mu)$, and therefore $1-p=\mu/(r+\mu)$.

Express the size-probability variance using $\mu$ and $r$.
Substitute the matched probabilities into $r(1-p)/p^2$, yielding $\mu+\mu^2/r$.

Match the variances.
Equate $\mu+\mu^2/r$ to $\mu+\alpha\mu^2$. For $\mu>0$, subtract $\mu$ and divide by $\mu^2$, so $1/r=\alpha$.

The complete correspondence is therefore

r=\frac{1}{\alpha},\qquad p=\frac{r}{r+\mu}=\frac{1}{1+\alpha\mu},\qquad 1-p=\frac{\alpha\mu}{1+\alpha\mu}.

In the reverse direction, given $r$ and $p$, the mean-dispersion parameters are

\mu=\frac{r(1-p)}{p},\qquad \alpha=\frac{1}{r}.

The variance calculation in the derivation is a useful check:

\operatorname{Var}(Y) =\frac{r(1-p)}{p^2} =\frac{\mu}{p} =\mu+\frac{\mu^2}{r} =\mu+\alpha\mu^2.

The middle equality follows because $\mu=r(1-p)/p$; dividing that expression by $p$ gives the size-probability variance. The final equality uses $1/r=\alpha$. This is a parameter conversion, not a claim that equating means alone identifies the dispersion.

Worked Example: Converting Mean and Dispersion to Size and Probability

Suppose a hypothetical gene count is modeled with mean $\mu=12$ and dispersion $\alpha=0.25$. Convert to $r,p$, then check the mean and variance in both forms.

First, the size is the reciprocal of the dispersion:

r=\frac{1}{0.25}=4.

Then the success probability is

p=\frac{r}{r+\mu}=\frac{4}{4+12}=\frac{4}{16}=0.25.

The size-probability mean is $4(1-0.25)/0.25=4(0.75)/0.25=12$, as required. The mean-dispersion variance is $12+0.25(12^2)=12+36=48$. Independently, the size-probability variance is $4(0.75)/(0.25^2)=3/0.0625=48$. Both parameterizations therefore describe the same mean and variance.

Reading Variance From Each Form

Once the parameterization is known, its variance formula gives a direct interpretation. In mean-dispersion form, $\alpha$ multiplies $\mu^2$, so it describes how quickly variance grows beyond the mean as the mean increases. In size-probability form, a smaller $r$ means a larger $1/r$ contribution to the quadratic variance term. With a fixed mean, lower size corresponds to greater dispersion.

The Poisson distribution is the limiting case with variance equal to the mean. In the mean-dispersion notation, this is $\alpha=0$. In the size-probability notation, it is a limit $r\to\infty$ while the mean remains fixed, with $p=r/(r+\mu)\to1$. It is not an ordinary negative binomial with a finite positive size; it is the limiting distribution. This distinction is useful because some software or texts use “Poisson” as a separate model rather than as a negative-binomial parameter value.

Worked Example: Converting Size and Probability to Mean and Dispersion

A hypothetical count model uses $r=5$ and $p=0.40$, with $Y$ defined as failures before $r$ successes. Find its mean, dispersion, and variance, and express the same model in mean-dispersion form.

Using the stated convention,

\mu=\frac{r(1-p)}{p} =\frac{5(1-0.40)}{0.40} =\frac{5(0.60)}{0.40} =\frac{3}{0.40}=7.5.

The dispersion is $\alpha=1/r=1/5=0.20$. Thus the variance in mean-dispersion form is $7.5+0.20(7.5^2)=7.5+0.20(56.25)=7.5+11.25=18.75$. Directly, the size-probability variance is $5(0.60)/(0.40^2)=3/0.16=18.75$. The two calculations agree.

This count model has expected count 7.5 and variance 18.75 counts squared. The variance is greater than the mean, consistent with its positive dispersion. The decimals describe model moments; they do not imply that an observed count can be fractional.

Worked Example: Comparing Two Dispersions at the Same Mean

Consider two hypothetical count models, both with mean $\mu=10$. Model A has $\alpha=0.05$, and Model B has $\alpha=0.20$. Compare their size parameters, probabilities, and variances.

For Model A, $r=1/0.05=20$, and $p=20/(20+10)=2/3$. Its variance is $10+0.05(10^2)=10+5=15$.

For Model B, $r=1/0.20=5$, and $p=5/(5+10)=1/3$. Its variance is $10+0.20(10^2)=10+20=30$.

Both models have the same expected count, but Model B has twice the variance of Model A. The difference is not a change in mean; it is the larger dispersion. In the size-probability form, the corresponding smaller size for Model B ($5$ rather than $20$) also signals greater variance at this matched mean.

Checks, Alternatives, and Limitations

A practical conversion should pass several simple checks. The calculated $r$ must be positive; $p$ must lie strictly between zero and one when $\mu>0$, $\alpha>0$; and substitution should recover both the original mean and variance. If a result gives $p\leq0$, $p\geq1$, or a variance inconsistent across the two forms, revisit the probability convention and algebra.

The parameterizations are alternatives for expressing the same negative binomial model, not competing biological explanations. Mean-dispersion form is especially convenient when the mean and extra-Poisson variation are the quantities of interest. Size-probability form is natural when the probability mass function is written as failures before successes or when a software interface asks for $r,p$. A Poisson model is a separate useful comparison when the variance is constrained to equal the mean; it cannot represent the same positive overdispersion at a fixed mean.

These moment relationships do not establish that a negative binomial distribution is suitable for every gene or experiment. They only state what variance follows if that model and parameterization are used. For example, strong departures in the observed count distribution or an incorrect probability convention can make an apparently sensible parameter conversion misleading. Also, $\alpha$ is not the variance itself: the variance is $\mu+\alpha\mu^2$, and it depends on both the mean and dispersion.

Common Mistakes

Taking size to equal dispersion. The correct relation is $r=1/\alpha$, not $r=\alpha$. Check that larger dispersion implies smaller size.
Using the mean equation to infer the dispersion map. Matching means gives $p=r/(r+\mu)$; it does not by itself imply $r=1/\alpha$. Match the variances as well.
Using a different success-probability convention without changing formulas. Under the convention here, $Y$ counts failures before $r$ successes, and the mean is $r(1-p)/p$.
Calling $\alpha$ the variance. It is the coefficient of $\mu^2$ in the variance expression, not the complete variance.

Key takeaway: Matching the mean gives $p=r/(r+\mu)$. Matching the variances then gives $r=1/\alpha$. Consequently, both forms imply $\operatorname{Var}(Y)=\mu+\alpha\mu^2=\mu+\mu^2/r$.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A model has $\mu=6$ and $\alpha=0.5$. Find $r$, $p$, and the variance.
A model uses $r=8$ and $p=0.5$, with the failures-before-successes convention. Find its mean, dispersion, and variance.
Explain why equating the means does not, by itself, establish $r=1/\alpha$.
At mean $\mu=4$, compare the variance for $\alpha=0.1$ with the Poisson variance.
For positive $\mu$ and $\alpha$, what range must the converted probability $p$ occupy, and why?

Solutions. (1) $r=1/0.5=2$; $p=2/(2+6)=0.25$; the variance is $6+0.5(36)=24$. (2) $\mu=8(1-0.5)/0.5=8$; $\alpha=1/8=0.125$; the variance is $8+0.125(64)=16$. (3) Matching means gives $p=r/(r+\mu)$ for any positive $r$; using that expression in the size-probability variance gives $\mu+\mu^2/r$, which must be matched to $\mu+\alpha\mu^2$ to conclude $1/r=\alpha$. (4) The negative binomial variance is $4+0.1(16)=5.6$, while the Poisson variance is $4$; the negative binomial variance is larger by $1.6$. (5) $p=r/(r+\mu)$, and with $r>0,\mu>0$, the denominator exceeds the numerator, so $0<p<1$.