Deriving the Negative-Binomial Probability Mass Function

Introduction

The negative-binomial mass function comes directly from a repeated-trial experiment. Each trial either succeeds or fails, and observation continues until a specified number of successes has occurred. To find the probability of a particular failure count, we must identify exactly which trial is the stopping trial, count the sequences that meet the condition, and calculate the probability of each such sequence.

This tutorial uses one convention throughout: the random variable counts failures before the rth success. The distinction between this count and the trial number at which observation stops is essential to getting both the support and the formula right.

Experiment, Definition, and Support

Suppose an experiment consists of an indefinitely available sequence of independent Bernoulli trials. Each trial has success probability $p$, where $0<p<1$, and failure probability $q=1-p$. Fix a positive integer $r$. Stop observing at the trial on which the rth success occurs.

Definition: Let $X$ be the number of failures before the rth success in independent Bernoulli trials with common success probability $p$, where $0<p<1$ and $r$ is a fixed positive integer. If the rth success never occurs, set $X=\infty$. The possible finite values of $X$ are $0,1,2,\ldots$.

The smallest possible value is zero: all of the first $r$ trials may be successes. There is no finite upper bound on the number of failures, because any finite number of failures could occur before the rth success. If $X=x$, then there have been $x+r$ trials by the stopping time: exactly $x$ failures and $r$ successes. In particular, the final trial is necessarily a success.

The formula below will also show that $P(X<\infty)=1$. Including the value $\infty$ in the definition lets us establish this fact from the probability calculation, rather than assume it without justification.

Counting the Trial Sequences

Fix a nonnegative integer $x$. For $X=x$, the first $x+r-1$ trials must contain exactly $r-1$ successes and $x$ failures. The $(x+r)$th trial must then be a success, which is the rth success. The first $x+r-1$ trials can be arranged in different orders, but every arrangement must have those same counts.

Theorem: For $x=0,1,2,\ldots$, $$ P(X=x)=\binom{x+r-1}{r-1}p^r(1-p)^x. $$

To count the arrangements, choose which $r-1$ of the first $x+r-1$ positions contain successes. The remaining $x$ positions contain failures. Therefore, the number of admissible arrangements is $\binom{x+r-1}{r-1}$. For any one such arrangement, the first $x+r-1$ trials have probability $p^{r-1}q^x$, by independence. The final trial contributes a further factor of $p$. Each arrangement thus has probability $p^rq^x$.

The arrangements are mutually exclusive: a particular sequence of outcomes for the first $x+r-1$ trials cannot place successes in two different sets of positions. Adding the probabilities over all the arrangements gives the stated mass function. This also explains why the coefficient counts placements in the trials before the final success, rather than placements across all $x+r$ trials.

Worked Example: Two Failures Before the Third Success

Let the success probability be $p=0.4$, and find the probability of exactly $x=2$ failures before the third success. Here $r=3$ and $q=1-p=0.6$.

The stopping trial is $x+r=2+3=5$, and it must be a success. Among the first four trials, exactly $r-1=2$ must be successes. The number of ways to choose their positions is $$ \binom{x+r-1}{r-1}=\binom{4}{2}=6. $$

Each of those six arrangements has two successes and two failures in the first four trials, followed by the final success. Its probability is $$ p^rq^x=(0.4)^3(0.6)^2=0.064(0.36)=0.02304. $$

The six arrangements are disjoint, so $$ P(X=2)=6(0.02304)=0.13824. $$

For instance, the sequence success, failure, success, failure, success is one admissible arrangement. The other five arise by choosing a different pair of success positions among the first four trials. All six have the same probability.

Worked Example: No Failures Before the Second Success

Suppose $p=0.7$, $r=2$, and we want the probability of no failures before the second success. In this case $x=0$, so the stopping trial is $x+r=2$.

The first $x+r-1=1$ trial must contain $r-1=1$ success. There is only one placement: $$ \binom{x+r-1}{r-1}=\binom{1}{1}=1. $$

The only possible sequence is success, success. Its probability, and therefore the requested probability, is $$ P(X=0)=1(0.7)^2(0.3)^0=0.49. $$

The factor $(0.3)^0=1$ reflects that no failures occur. This case checks the lower endpoint of the support and confirms that the final success is still included in the factor $p^r$.

Worked Example: Three Failures Before the Fourth Success

Let $p=0.25$, $r=4$, and $x=3$. Then $q=0.75$, and the stopping trial is $x+r=7$. Among the first six trials, three must be successes and three must be failures.

The number of ways to place those first three successes is $$ \binom{x+r-1}{r-1}=\binom{6}{3} =\frac{6\cdot5\cdot4}{3\cdot2\cdot1}=20. $$

Each arrangement followed by the final success has probability $$ (0.25)^4(0.75)^3 =\frac{1}{256}\frac{27}{64} =\frac{27}{16384}. $$

Multiplying by the 20 disjoint arrangements gives $$ P(X=3)=20\left(\frac{27}{16384}\right) =\frac{540}{16384} =\frac{135}{4096} \approx 0.03296. $$

The exponent of $0.25$ is four, not three: there are three successes in the first six trials and one more success on trial seven.

Why the Formula Is a Probability Mass Function

The derivation gives the probability of each finite count. To verify that these probabilities account for the entire sample space, sum them over all nonnegative integers $x$. We first establish the series identity needed for that calculation.

Theorem: For $0\leq q<1$ and a positive integer $r$, $$ \sum_{x=0}^{\infty}\binom{x+r-1}{r-1}q^x=\frac{1}{(1-q)^r}. $$

The ordinary geometric series gives $\sum_{k=0}^{\infty}q^k=1/(1-q)$. Taking the product of $r$ copies of this convergent series yields $$ \left(\sum_{k=0}^{\infty}q^k\right)^r =\sum_{k_1=0}^{\infty}\cdots\sum_{k_r=0}^{\infty}q^{k_1+\cdots+k_r}. $$ All terms are nonnegative, and the geometric series converges, so the product and its grouping by the total exponent are valid. For a given total $x$, the coefficient of $q^x$ is the number of $r$-tuples of nonnegative integers $(k_1,\ldots,k_r)$ with $k_1+\cdots+k_r=x$. The stars-and-bars count of these tuples is $\binom{x+r-1}{r-1}$. Grouping terms with the same total therefore gives the left side of the identity. The product of the $r$ geometric-series sums is $1/(1-q)^r$, proving the identity. $\square$

Apply the identity with $q=1-p$. Since $1-q=p$, the total probability assigned by the derived formula is $$ \sum_{x=0}^{\infty}\binom{x+r-1}{r-1}p^r q^x =p^r\frac{1}{(1-q)^r} =p^r\frac{1}{p^r} =1. $$

The events $\{X=x\}$, for finite nonnegative $x$, are disjoint. Their probabilities sum to one, so $P(X<\infty)=1$. Thus the rth success occurs after a finite number of trials with probability one, and the formula is a probability mass function on the stated support.

Common Mistakes and a Reliable Check

The most common counting error is to use $\binom{x+r}{r}$, as if all $r$ successes could be placed freely among all $x+r$ trials. That count ignores the stopping rule. The last trial is already determined to be the rth success; only the first $r-1$ successes can be placed among the first $x+r-1$ trials. The correct number of arrangements is $\binom{x+r-1}{r-1}$.

A related error is to multiply by $p^{r-1}q^x$ and forget the probability of the final success. That expression gives only the probability of a specified arrangement in the first $x+r-1$ trials. The stopping trial contributes one more factor of $p$, so the full sequence probability is $p^rq^x$.

A useful check is to write the stopping trial number before counting. For a proposed failure count $x$, it is $x+r$; the preceding $x+r-1$ trials must contain $r-1$ successes and $x$ failures; and the last trial is a success. These three facts determine the support, the binomial coefficient, and the powers in the mass function.

Key takeaway: For the count of failures before the rth success, the final success is fixed at the stopping trial. Choose the $r-1$ earlier success positions from the first $x+r-1$ trials, then multiply the number of arrangements by the probability $p^r(1-p)^x$ of each complete sequence.

Check Your Understanding

Use the stopping-trial and placement-count reasoning to answer these questions.

For $r=3$ and $x=4$, what is the stopping trial, and how many placements of the first $r-1$ successes are possible?
Why is the exponent of $p$ equal to $r$, even though only $r-1$ successes are placed among the first $x+r-1$ trials?
With $p=0.2$, find $P(X=1)$ when $r=2$, showing the placement count and the probability of one complete sequence.
What is the probability of $X=0$ for general $r$, and which trial sequence does this event describe?
Explain why $\binom{x+r}{r}$ does not count the admissible arrangements for this stopping rule.