Negative Binomial Parameterizations

Why Parameterization Matters

A negative binomial model can describe the same count distribution using different parameter pairs. One common form uses a mean and a dispersion; another uses a size and a probability. The formulas may look different, and software packages do not always use the word “probability” for the same event. A correct translation is therefore more than a change of notation: it requires identifying what each parameter means and checking that the constraints still hold.

In single-cell RNA sequencing, a count may be modeled for a particular gene and cell, with the mean and dispersion depending on the analysis. This tutorial concerns only how to express a given negative binomial distribution in two parameterizations. It does not address how to estimate those parameters or how to assess a fitted model.

Definition: In the mean-dispersion parameterization, $\mu$ is the mean parameter and $\phi$ is the dispersion parameter. For $\mu>0$ and $\phi>0$, the probability mass function is $$ P(Y=y)= \frac{\Gamma(y+1/\phi)}{\Gamma(1/\phi)y!} \left(\frac{1}{1+\phi\mu}\right)^{1/\phi} \left(\frac{\phi\mu}{1+\phi\mu}\right)^y, \qquad y=0,1,2,\ldots. $$ Here $\Gamma(\cdot)$ is the gamma function, which extends the factorial through $\Gamma(k)=(k-1)!$ for positive integers $k$. The dispersion is conventionally nonnegative; $\phi=0$ denotes the Poisson limit, rather than a finite-size negative binomial distribution.

The other form describes $Y$ as the number of failures before $r$ successes, with success probability $p$ on each trial. Its size parameter $r$ may be any positive real number, not only an integer; the gamma function in the probability formula makes that extension well-defined. For $r>0$ and $0<p<1$, its mass function is

P(Y=y)= \frac{\Gamma(y+r)}{\Gamma(r)y!}p^r(1-p)^y, \qquad y=0,1,2,\ldots.

Some sources instead call $q=1-p$ the probability parameter, where $q$ is the probability of a failure on one trial. These conventions are equivalent, but the displayed formula makes clear which probability appears with exponent $r$: in this tutorial, $p$ is the success probability and $q=1-p$ is the failure probability.

The Conversion Formulas

For a positive mean and positive dispersion, the two parameterizations are related by

r=\frac{1}{\phi}, \qquad p=\frac{r}{r+\mu}=\frac{1}{1+\phi\mu}, \qquad 1-p=\frac{\mu}{r+\mu}=\frac{\phi\mu}{1+\phi\mu}.

Conversely, if $r>0$ and $0<p<1$, the mean-dispersion parameters are

\mu=\frac{r(1-p)}{p}, \qquad \phi=\frac{1}{r}.

These equations require care at the boundaries. If $\mu>0$ and $\phi>0$, then $r>0$ and $0<p<1$. If $\phi=0$ with a finite positive $\mu$, the distribution is Poisson with mean $\mu$; it is approached as $r$ tends to infinity and $p$ tends to one while $r(1-p)/p$ remains $\mu$. There is no finite $r$ corresponding to $\phi=0$. If $\mu=0$, the count is identically zero; it can be represented by $p=1$ in the size-probability form, but the conversion back to $\mu=r(1-p)/p$ is a boundary case.

Key takeaway: For the convention used here, size is the reciprocal of dispersion, $r=1/\phi$, and the success probability is $p=1/(1+\phi\mu)$. The probability multiplying the count-dependent term is $1-p$, not $p$.

Why the Formulas Agree

The conversion follows by matching the probability mass functions. Begin with the mean-dispersion form and substitute $r=1/\phi$. Then $1+\phi\mu=(r+\mu)/r$, so

\frac{1}{1+\phi\mu}=\frac{r}{r+\mu}=p, \qquad \frac{\phi\mu}{1+\phi\mu}=\frac{\mu}{r+\mu}=1-p.

The gamma-function coefficient also matches because $1/\phi=r$. Thus, for every nonnegative integer $y$, the mean-dispersion mass is

\frac{\Gamma(y+r)}{\Gamma(r)y!}p^r(1-p)^y,

which is precisely the size-probability mass function. For the reverse direction, solve $p=r/(r+\mu)$ for $\mu$: $p(r+\mu)=r$, so $p\mu=r(1-p)$, giving $\mu=r(1-p)/p$. The dispersion relation follows directly from $r=1/\phi$. This establishes the conversion algebraically, without changing the distribution of the count.

Worked Example: Convert Mean and Dispersion to Size and Probability

Suppose a hypothetical gene-cell count model has mean parameter $\mu=4$ counts and dispersion parameter $\phi=0.25$. Convert to size and success probability, then verify the probability of observing $Y=2$.

First, the size is $r=1/\phi=1/0.25=4$. The success probability is

p=\frac{1}{1+\phi\mu} =\frac{1}{1+(0.25)(4)} =\frac{1}{2} =0.5.

Thus $1-p=0.5$. In the size-probability form, for $y=2$,

P(Y=2)= \frac{\Gamma(6)}{\Gamma(4)2!}(0.5)^4(0.5)^2 =\frac{120}{6\cdot2}(0.015625) =10(0.015625) =0.15625.

Check in the mean-dispersion form: the coefficient is $\Gamma(2+4)/[\Gamma(4)2!]=10$, the first powered factor is $(1+0.25\cdot4)^{-4}=2^{-4}$, and the count-dependent factor is $(0.25\cdot4/2)^2=0.5^2$. Their product is $10(1/16)(1/4)=0.15625$, the same result. The conversion preserves the probability mass at $Y=2$.

Worked Example: Convert Size and Probability Back to Mean and Dispersion

A hypothetical analysis reports size $r=4$ and success probability $p=0.4$. Find the mean-dispersion parameters and check the zero-count probability in both forms.

The dispersion is $\phi=1/r=1/4=0.25$. The mean is

\mu=\frac{r(1-p)}{p} =\frac{4(0.6)}{0.4} =6 \text{ counts}.

For $Y=0$, the size-probability form gives $P(Y=0)=p^r=(0.4)^4=0.0256$. In the mean-dispersion form,

P(Y=0)= \left(\frac{1}{1+\phi\mu}\right)^{1/\phi} = \left(\frac{1}{1+(0.25)(6)}\right)^4 =(0.4)^4 =0.0256.

The two parameter pairs therefore agree on this probability as well. The units belong to the mean: it is an expected count, while $r$, $p$, and $\phi$ are dimensionless parameters.

Worked Example: Fractional Size and a Boundary Check

Consider a hypothetical count model with $\mu=0.8$ counts and $\phi=1.5$. Convert its parameters and check whether a non-integer size is allowed. Also calculate the zero-count probability using each form.

The size and success probability are

r=\frac{1}{1.5}=\frac{2}{3}, \qquad p=\frac{1}{1+(1.5)(0.8)} =\frac{1}{2.2} =\frac{5}{11}\approx0.4545.

Here $r=2/3$ is positive, and $p$ lies strictly between zero and one, so both are valid. Size need not be an integer. For a zero count, the size-probability form gives

P(Y=0)=p^r =\left(\frac{5}{11}\right)^{2/3} \approx0.5912.

The mean-dispersion form gives

P(Y=0)= \left(\frac{1}{1+\phi\mu}\right)^{1/\phi} = \left(\frac{1}{2.2}\right)^{2/3} \approx0.5912.

The two expressions are equal because $p=1/(1+\phi\mu)=5/11$ and $1/\phi=2/3$. A proposed size of $2/3$ is not an error; a negative size or a success probability outside $(0,1)$ would be invalid for a nondegenerate distribution.

Comparing Conventions and Checking a Translation

The following table summarizes the usual interior case, where the mean is positive and the dispersion is strictly positive.

Quantity	Mean-dispersion form	Size-probability form
Mean parameter	$\mu$	$r(1-p)/p$
Dispersion or size	$\phi>0$	$r=1/\phi>0$
Success probability	$1/(1+\phi\mu)$	$p$, with $0<p<1$
Failure probability	$\phi\mu/(1+\phi\mu)$	$1-p$
Zero-count probability	$(1+\phi\mu)^{-1/\phi}$	$p^r$

A useful check is to compare the zero-count probabilities after conversion. Agreement there is a quick diagnostic, though a full algebraic match of the mass functions establishes equality for every count value. When reading software documentation, verify the exact formula or parameter definition rather than relying on a function argument named “prob,” “size,” “theta,” or “dispersion.” Some interfaces use a success probability, some use its complement, and some use a reciprocal-dispersion parameter with a different name.

The mean-dispersion form is often convenient when the scientific model directly specifies a mean and a dispersion. The size-probability form is common in probability texts and software interfaces built around the number of failures before a specified number of successes. Neither form is a different statistical model when the conversion equations are applied correctly. The Poisson model is a limiting alternative at $\phi=0$, not a negative binomial distribution with a finite size $r$.

Common Mistakes and Limitations

Swapping the probability and its complement. If $p$ is the success probability, the formula contains $p^r(1-p)^y$. Using $p^y(1-p)^r$ describes a different parameter convention and gives the wrong conversion unless the meanings are also swapped.
Using size equal to dispersion. Under the convention here, $r=1/\phi$, not $r=\phi$. For example, $\phi=0.25$ means $r=4$.
Requiring integer size. The generalized negative binomial formula permits any $r>0$. A fractional value is valid, as the third example demonstrates.
Treating the Poisson boundary as a finite-size case. When $\phi=0$, $1/\phi$ is undefined. Use the Poisson limit instead of inserting zero into the finite-size conversion.
Ignoring model context. These equations translate parameters; they do not establish that a negative binomial model is appropriate for a particular gene, cell, or data set. Suitability and estimation require separate analysis.

These checks can catch a parameterization error, but they cannot diagnose whether a fitted count model represents the data well. Nor does matching a parameter pair across formulas validate the assumptions that led to the model. The conversion is exact only when both expressions use the same negative binomial convention and describe the same count variable.

Check Your Understanding

For each question, identify the parameter convention before calculating. Solutions follow the questions.

For $\mu=3$ counts and $\phi=0.5$, find $r$, $p$, and $1-p$.
For $r=2.5$ and success probability $p=0.5$, find $\mu$ and $\phi$. Is the size valid even though it is not an integer?
Explain why $\phi=0$ cannot be converted to a finite size using $r=1/\phi$, and state the corresponding distributional boundary.
A formula uses $p^y(1-p)^r$ and describes $p$ as the failure probability. What is the success probability in the convention used in this tutorial?

Solutions. (1) $r=1/0.5=2$, $p=1/[1+(0.5)(3)]=0.4$, and $1-p=0.6$. (2) $\mu=2.5(0.5)/0.5=2.5$ counts and $\phi=1/2.5=0.4$. The size is valid because any positive real $r$ is allowed. (3) The reciprocal is undefined at $\phi=0$; that case is the Poisson limit with mean $\mu$, approached as size tends to infinity. (4) If the displayed formula calls $p$ the failure probability, the success probability in this tutorial’s convention is $1-p$.

Quantity	Mean-dispersion form	Size-probability form
Mean parameter	\(\mu\)	\(r(1-p)/p\)
Dispersion or size	\(\phi>0\)	\(r=1/\phi>0\)
Success probability	\(1/(1+\phi\mu)\)	\(p\), with \(0<p<1\)
Failure probability	\(\phi\mu/(1+\phi\mu)\)	\(1-p\)
Zero-count probability	\((1+\phi\mu)^{-1/\phi}\)	\(p^r\)