Interpreting Negative Binomial Overdispersion

Why Compare the Mean and Variance?

A claim count’s mean describes its long-run average, but it does not describe how widely individual counts vary around that average. For a negative binomial count, the variance is greater than the mean under the usual model assumptions. This property is called overdispersion relative to the Poisson model: a Poisson count with the same mean has variance equal to that mean, whereas the negative binomial count has greater variance.

The amount of overdispersion has a direct interpretation in the failure-count version of the negative binomial model. The model counts failures before a specified number of successes. Its success-probability parameter $p$ determines both the expected number of failures and how variable that number is. Interpret $p$ only after confirming what the model counts and which outcome is called a success.

Definition: Let $N$ be the number of failures observed before the $r$th success in a sequence of independent Bernoulli trials. Each trial has the same success probability $p$, where $0<p\leq 1$, and $r$ is a positive integer. This is the negative binomial failure-count model. For $0<p<1$, $$ E[N]=\frac{r(1-p)}{p}, \qquad \operatorname{Var}(N)=\frac{r(1-p)}{p^2}. $$

The assumptions matter: trials must be independent, their success probability must remain constant, and observation stops when the $r$th success occurs. Here a failure is the event being counted. The symbol $p$ is the probability of success on one trial, not the probability of a failure or a claim. If $p=1$, every trial is a success and $N=0$ with probability one, so its mean and variance are both zero.

Mean, Variance, and the Dispersion Ratio

For $0<p<1$, divide the variance by the mean:

\frac{\operatorname{Var}(N)}{E[N]} = \frac{r(1-p)/p^2}{r(1-p)/p} = \frac{1}{p}.

Since $p<1$, the ratio $1/p$ is greater than one. Thus the variance exceeds the mean. The difference is also useful:

\operatorname{Var}(N)-E[N] = \frac{r(1-p)}{p^2}-\frac{r(1-p)}{p} = \frac{r(1-p)^2}{p^2}.

This difference is positive when $0<p<1$. It is not a separate model parameter: it is determined by $r$ and $p$. The ratio $1/p$ is a convenient measure of relative dispersion. For example, if $p=0.5$, the variance is twice the mean; if $p=0.25$, it is four times the mean. These ratios do not mean that every observed count will be a particular multiple of the mean. They compare overall variability with the average.

Quantity	Formula	Interpretation
Expected failure count	$r(1-p)/p$	Average number of failures before $r$ successes
Variance	$r(1-p)/p^2$	Spread of the failure count around its mean
Variance-to-mean ratio	$1/p$	Relative dispersion; greater than one for $0<p<1$
Variance minus mean	$r(1-p)^2/p^2$	Amount by which the variance exceeds the mean

Why These Formulas Hold

The failure count can be split into $r$ stages: count failures before the first success, then failures between the first and second successes, and continue through the $r$th success. Let $G_j$ be the number of failures in stage $j$. Under the independent, constant-$p$ trial assumptions, the stages have the same distribution and are independent, and $N=G_1+\cdots+G_r$.

For one stage, $G_j$ has probability $p(1-p)^k$ of equaling $k$, for $k=0,1,2,\ldots$. Its probability generating function, obtained by summing this geometric series, is:

G(z)=\sum_{k=0}^{\infty}p(1-p)^kz^k =\frac{p}{1-(1-p)z}.

For a probability generating function, $G'(1)=E[G_j]$ and $G''(1)=E[G_j(G_j-1)]$. Differentiating and evaluating at one gives:

G'(1)=\frac{1-p}{p}, \qquad G''(1)=\frac{2(1-p)^2}{p^2}.

Using $\operatorname{Var}(G_j)=G''(1)+G'(1)-[G'(1)]^2$, the variance of one stage is:

\operatorname{Var}(G_j) = \frac{2(1-p)^2}{p^2} +\frac{1-p}{p} -\left(\frac{1-p}{p}\right)^2 = \frac{1-p}{p^2}.

Finally, expectations add for sums, and variances add for independent random variables. Therefore:

E[N]=\sum_{j=1}^{r}E[G_j]=\frac{r(1-p)}{p}, \qquad \operatorname{Var}(N)=\sum_{j=1}^{r}\operatorname{Var}(G_j) =\frac{r(1-p)}{p^2}.

Interpreting the Success Probability

With $r$ fixed, a smaller $p$ means success is less likely on any one trial. More failures are then expected before reaching the target of $r$ successes. In the formulas, decreasing $p$ increases the mean and increases the variance. It also increases the variance-to-mean ratio $1/p$, so the variance grows more than proportionally to the mean.

A fair comparison of relative dispersion can instead hold the mean fixed. If the mean is $m>0$, the formulas imply:

m=\frac{r(1-p)}{p}, \qquad \operatorname{Var}(N)=\frac{m}{p}.

At the same mean, a lower $p$ gives a higher variance-to-mean ratio. To keep the mean fixed while changing $p$, $r$ must also change: rearranging the mean formula gives $r=mp/(1-p)$. Thus, do not claim that changing $p$ alone keeps the mean fixed.

Worked Example: Quantifying Overdispersion

A hypothetical count model records failures before 5 successes. Assume independent trials with a constant success probability $p=0.5$. Find the mean, variance, variance-to-mean ratio, and variance excess over the mean. Compare the variance with that of a Poisson count having the same mean.

Method. Use the failure-count formulas with $r=5$ and $p=0.5$. The common mistake to avoid is treating $p$ as the failure probability.

The failure probability per trial is $1-p=0.5$. The mean is:

E[N]=\frac{5(1-0.5)}{0.5} =\frac{5(0.5)}{0.5} =5.

The variance is:

\operatorname{Var}(N)=\frac{5(1-0.5)}{(0.5)^2} =\frac{2.5}{0.25} =10.

The variance-to-mean ratio is $10/5=2$, which also equals $1/p=1/0.5=2$. The variance excess is $10-5=5$. A Poisson count with mean 5 has variance 5, so this negative binomial model has twice the Poisson variance at the same mean.

Check. The formula for the excess gives $5(0.5)^2/(0.5)^2=5$, agreeing with $10-5$.

Worked Example: Comparing Equal Means

Two hypothetical portfolios are described using negative binomial failure counts. Portfolio A counts failures before 4 successes with $p=0.5$. Portfolio B counts failures before 8 successes with $p=2/3$. Calculate the mean and variance for each, then identify which has greater relative dispersion. Assume the stated trial assumptions hold for each portfolio.

Method. Calculate both moments for each parameter pair before comparing. The common mistake to avoid is comparing variances without first checking whether the means are equal.

For Portfolio A, $r=4$, $p=0.5$, and $1-p=0.5$:

E[N_A]=\frac{4(0.5)}{0.5}=4, \qquad \operatorname{Var}(N_A)=\frac{4(0.5)}{(0.5)^2} =\frac{2}{0.25}=8.

For Portfolio B, $r=8$, $p=2/3$, and $1-p=1/3$:

E[N_B]=\frac{8(1/3)}{2/3}=4, \qquad \operatorname{Var}(N_B)=\frac{8(1/3)}{(2/3)^2} =\frac{8/3}{4/9}=6.

Both means equal 4. Portfolio A has variance-to-mean ratio $8/4=2$; Portfolio B has ratio $6/4=1.5$. Portfolio A is more dispersed relative to its mean. This agrees with the parameter comparison: its $p=0.5$ is lower than Portfolio B’s $p=2/3$, and $1/0.5=2$ exceeds $1/(2/3)=1.5$.

Check. The variances exceed their respective means: $8>4$ and $6>4$, as required for both models with $0<p<1$.

Worked Example: Recovering Parameters From Moments

A hypothetical negative binomial failure count is known to have mean 6 and variance 18. Assuming the failure-count parameterization with a positive integer $r$, find $p$ and $r$. Then verify both moments.

Method. First use the variance-to-mean ratio to identify $p$, then use the mean equation to solve for $r$. The common mistake to avoid is solving for $r$ before distinguishing the success probability from the failure probability.

The ratio is:

\frac{\operatorname{Var}(N)}{E[N]}=\frac{18}{6}=3=\frac{1}{p}.

Therefore $p=1/3$, so the failure probability is $1-p=2/3$. Substitute into the mean formula:

6=\frac{r(2/3)}{1/3}=2r, \qquad r=3.

The value of $r$ is a positive integer, as required. Verification gives:

E[N]=\frac{3(2/3)}{1/3}=6, \qquad \operatorname{Var}(N)=\frac{3(2/3)}{(1/3)^2} =\frac{2}{1/9}=18.

Both specified moments are recovered, so the parameters are consistent with the model.

Common Mistakes and Exam Interpretation

Reversing success and failure. In this parameterization, $p$ is the success probability, while the counted events have probability $1-p$. The mean is $r(1-p)/p$, not $rp/(1-p)$.
Using the wrong variance formula. The variance has $p^2$ in the denominator: $r(1-p)/p^2$. Omitting one factor of $p$ incorrectly makes variance equal to the mean.
Comparing counts with different means as though only dispersion changed. If the means differ, compare the variance-to-mean ratios or explicitly account for both moments.
Changing $p$ while saying the mean is fixed. At fixed $r$, changing $p$ changes the mean. To compare dispersion at a fixed mean, $r$ must change as well.
Assuming every negative binomial parameterization uses the same meaning for its parameters. State what is counted and what $p$ represents before applying formulas.

The mean and variance relationship is a useful interpretation, not a complete description of the count distribution. Matching these two moments does not by itself establish that the independent, constant-probability trial assumptions are appropriate.

Key takeaway: For the negative binomial model that counts failures before $r$ successes, $E[N]=r(1-p)/p$ and $\operatorname{Var}(N)=r(1-p)/p^2$. For $0<p<1$, the variance-to-mean ratio is $1/p>1$. A smaller success probability means greater relative dispersion, provided the parameterization is interpreted correctly.

Check Your Understanding

Use the failure-count parameterization throughout. For each problem, assume independent trials with a constant success probability and state what the calculated ratio means.

A hypothetical count is the number of failures before 5 successes, with $p=0.8$. Find its mean, variance, and variance excess over the mean. (Answer: mean $1.25$, variance $1.5625$, excess $0.3125$.)
A hypothetical model counts failures before 4 successes with $p=0.4$. Find the mean, variance, and variance-to-mean ratio. (Answer: mean $6$, variance $15$, ratio $2.5$.)
A negative binomial failure count has mean 3 and variance 6. Find $p$, $r$, and verify that $r$ is a permitted positive integer. (Answer: $p=0.5$, $r=3$, a positive integer.)
Model A has $r=3$, $p=0.5$; Model B has $r=9$, $p=0.75$. Calculate each mean and variance, and say which has greater relative dispersion. (Answer: both means are $3$; Model A variance $6$, Model B variance $4$; Model A has greater relative dispersion.)
For a fixed $r=4$, compare $p=0.5$ with $p=0.25$. Find each mean, variance, and variance-to-mean ratio, then describe the effect of lowering $p$. (Answer: at $p=0.5$, mean $4$, variance $8$, ratio $2$; at $p=0.25$, mean $12$, variance $48$, ratio $4$. Lowering $p$ increases the mean, variance, and relative dispersion.)

Quantity	Formula	Interpretation
Expected failure count	\(r(1-p)/p\)	Average number of failures before \(r\) successes
Variance	\(r(1-p)/p^2\)	Spread of the failure count around its mean
Variance-to-mean ratio	\(1/p\)	Relative dispersion; greater than one for \(0<p<1\)
Variance minus mean	\(r(1-p)^2/p^2\)	Amount by which the variance exceeds the mean

What You'll Learn