Worked Overdispersion Problems With Negative Binomial Models

Why Overdispersion Changes Frequency Predictions

A Poisson model describes claim counts with variance equal to the mean. In a portfolio whose risks differ in underlying claim propensity, that equality can be too restrictive: some risks tend to produce more claims, while others tend to produce fewer. A negative binomial model allows the variance to exceed the mean. This difference matters not only for the spread of counts, but also for predictions such as the probability of no claims or of several claims.

The earlier material on variance-to-mean ratios and risk heterogeneity motivates this model choice. Here the emphasis is on solving problems: infer the negative binomial parameters from moments, calculate probabilities, and identify what assumptions about heterogeneity are needed. These methods are aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Definition: Use the negative binomial convention in which $N$ is the number of failures before $r$ successes, with success probability $p$, where $r>0$ and $0<p<1$. Then $$ P(N=n)=\frac{\Gamma(n+r)}{\Gamma(r)n!}p^r(1-p)^n,\qquad n=0,1,2,\ldots $$ The mean and variance are $$ E[N]=\mu=\frac{r(1-p)}{p}, \qquad \operatorname{Var}(N)=\frac{r(1-p)}{p^2} =\mu+\frac{\mu^2}{r}. $$

The parameter $r$ controls dispersion: for a fixed mean, a smaller $r$ gives a larger variance. It is often convenient to use $\alpha=1/r$, so that $\operatorname{Var}(N)=\mu+\alpha\mu^2$. The parameter $p$ is not the probability of a claim; in this convention it is the success probability in the failures-before-successes count. Keeping that convention clear prevents a common parameterization error.

Inferring Parameters From the Mean and Variance

Suppose a negative binomial count has mean $\mu$ and variance $\sigma_N^2$. Rearranging the variance formula gives a direct method-of-moments solution:

r=\frac{\mu^2}{\sigma_N^2-\mu}, \qquad p=\frac{r}{r+\mu}=\frac{\mu}{\sigma_N^2}.

These expressions describe a finite overdispersed negative binomial model only when $\sigma_N^2>\mu>0$. If variance equals the mean, the Poisson model is the limiting case as $r$ tends to infinity. If variance is below the mean, these moment equations do not produce a finite positive $r$; a standard overdispersed negative binomial model is not supported by those moments.

Worked Example: Inferring Parameters and Comparing a Low-Count Probability

A hypothetical insurer models the number $N$ of claims per policy-year with a negative binomial distribution. The assumed mean is 6 claims and the assumed variance is 18 claims squared. Infer $r$ and $p$, then calculate $P(N\leq 1)$. Compare it with the Poisson prediction having the same mean.

Method and assumptions. Use the failures-before-successes parameterization above, and treat the stated mean and variance as model moments. Since $18>6$, the moments allow a finite overdispersed negative binomial model.

Step 1: Find $r$. The excess of variance over mean is $18-6=12$, so $$ r=\frac{6^2}{18-6}=\frac{36}{12}=3. $$

Step 2: Find $p$. $$ p=\frac{r}{r+\mu}=\frac{3}{3+6}=\frac{1}{3}. $$ Thus $1-p=2/3$. As a check, the mean is $3(2/3)/(1/3)=6$, and the variance is $3(2/3)/(1/3)^2=18$.

Step 3: Calculate the negative binomial probability. The probabilities at zero and one are $$ P(N=0)=\left(\frac{1}{3}\right)^3=\frac{1}{27}, \qquad P(N=1)=\frac{\Gamma(4)}{\Gamma(3)1!}\left(\frac{1}{3}\right)^3\left(\frac{2}{3}\right) =3\cdot\frac{1}{27}\cdot\frac{2}{3}=\frac{2}{27}. $$ Therefore $P(N\leq1)=3/27=1/9=0.1111$, to four decimal places.

Step 4: Compare with Poisson. For a Poisson model with mean 6, $$ P(N\leq1)=e^{-6}\left(1+6\right)=7e^{-6}=0.0174 $$ to four decimal places. The negative binomial model assigns substantially more probability to this low-count event, even though both models have mean 6. The negative binomial distribution is more dispersed, so it puts more probability in the tails overall.

Common mistake avoided: Setting $p=1/3$ and treating it as the probability of a claim would reverse the interpretation. Here $p$ is the success probability in the specified count convention; the claim count probabilities must use the full negative binomial formula.

Why Heterogeneity Produces the Extra Variance

A useful justification comes from the Gamma-Poisson mixture. Suppose the claim count, conditional on a risk-specific claim intensity $\Lambda$, is Poisson with mean $\Lambda$. Let $\Lambda$ have a gamma distribution with shape $r$ and rate $r/\mu$. Its mean is $\mu$, and its variance is $\mu^2/r$. Conditional Poisson variation and variation in the underlying intensity both contribute to the unconditional variance.

Conditional mean.
Given $\Lambda$, $E[N\mid\Lambda]=\Lambda$. Taking expectations gives $E[N]=E[\Lambda]=\mu$.

Conditional variance.
Given $\Lambda$, $\operatorname{Var}(N\mid\Lambda)=\Lambda$, so $E[\operatorname{Var}(N\mid\Lambda)]=\mu$.

Variation between risks.
The variance of the conditional mean is $\operatorname{Var}(E[N\mid\Lambda])=\operatorname{Var}(\Lambda)=\mu^2/r$.

Combine the terms.
The law of total variance gives $\operatorname{Var}(N)=E[\operatorname{Var}(N\mid\Lambda)]+\operatorname{Var}(E[N\mid\Lambda])=\mu+\mu^2/r$.

The first term is ordinary Poisson variation within a risk at a given intensity. The second term is added variation because intensities differ across risks. Integrating the conditional Poisson probabilities over this gamma distribution gives the negative binomial probabilities. Thus, the model’s extra variance has a direct interpretation as unobserved variation in claim propensity.

This explanation also clarifies why the duration assumption matters. If a risk’s latent intensity persists over several periods, counts across those periods are conditionally Poisson given the same $\Lambda$, but they are not independent after $\Lambda$ is averaged out. If each period instead has an independent draw of the latent intensity, the total count has a different variance. The problem must specify, or permit you to infer, which structure is intended.

Worked Example: Scaling a Persistent Risk Mix Across Periods

For a hypothetical book of policies, the one-year claim count has mean 2 and variance 4 under a Gamma-Poisson model. Assume each policy’s latent claim intensity remains fixed over a three-year period, and, conditional on that intensity, claims follow a Poisson process. Find the three-year mean, variance, and probability of no claims.

Method and assumptions. First infer the gamma shape $r$ from one-year moments. A shared latent intensity means the three-year conditional Poisson mean is $3\Lambda$; it is not three independent draws of $\Lambda$.

Step 1: Infer the heterogeneity parameter. $$ r=\frac{\mu^2}{\sigma_N^2-\mu} =\frac{2^2}{4-2}=2. $$ The one-year success probability is $p=r/(r+\mu)=2/(2+2)=1/2$.

Step 2: Scale the mean and variance. The three-year conditional mean is $3\Lambda$, whose unconditional mean is $3E[\Lambda]=6$. Its variance is $3E[\Lambda]+\operatorname{Var}(3\Lambda)$, so $$ E[N_3]=6,\qquad \operatorname{Var}(N_3)=6+9\operatorname{Var}(\Lambda) =6+9\left(\frac{2^2}{2}\right)=24. $$ Equivalently, the three-year negative binomial has $r=2$, mean 6, and variance $6+6^2/2=24$.

Step 3: Find the probability of no claims. The three-year success probability is $p_3=r/(r+6)=2/8=1/4$. Therefore $$ P(N_3=0)=p_3^r=\left(\frac14\right)^2=0.0625. $$

Step 4: Check the independent-period alternative. If each year instead had an independent latent intensity with the same one-year distribution, the three-year count would be the sum of three independent negative binomial counts, each with $r=2,p=1/2$. Its mean would still be 6, but its variance would be $3(4)=12$, and its zero probability would be $(1/4)^3=0.015625$. The stated persistent-intensity assumption therefore matters.

Common mistake avoided: Multiplying the one-year variance by three without checking dependence gives 12. That calculation is appropriate for independent annual counts, not for a persistent latent intensity shared across all three years.

Using a Fitted Model to Compare Count Predictions

Once $r$ and $p$ are known, individual probabilities can be calculated directly. For integer $r$, the coefficient $\Gamma(n+r)/(\Gamma(r)n!)$ is $\binom{n+r-1}{n}$. If $r$ is not an integer, the gamma-function form remains valid. A tail probability such as $P(N\geq k)$ can be found by summing the probabilities below $k$ and subtracting from 1.

Worked Example: Comparing a Multiple-Claim Probability

A hypothetical roadside-assistance portfolio uses a negative binomial model with $r=4$ and $p=1/2$ for the number of service claims in a year. Calculate $P(N\geq3)$, verify the model moments, and compare the result with a Poisson model having the same mean.

Method and assumptions. The count is failures before four successes, with $1-p=1/2$. Use the complement $P(N\geq3)=1-P(N\leq2)$, which requires only three negative binomial probabilities.

Step 1: Verify the mean and variance. $$ E[N]=\frac{4(1/2)}{1/2}=4, \qquad \operatorname{Var}(N)=\frac{4(1/2)}{(1/2)^2}=8. $$

Step 2: Calculate the probabilities from zero through two. $$ P(N=0)=\left(\frac12\right)^4=\frac{1}{16}, $$ $$ P(N=1)=\binom{4}{1}\left(\frac12\right)^4\left(\frac12\right)=\frac18, $$ $$ P(N=2)=\binom{5}{2}\left(\frac12\right)^4\left(\frac12\right)^2 =\frac{10}{64}=\frac{5}{32}. $$ Thus $P(N\leq2)=2/32+4/32+5/32=11/32$, and $$ P(N\geq3)=1-\frac{11}{32}=\frac{21}{32}=0.6563 $$ to four decimal places.

Step 3: Compare with Poisson. A Poisson model with mean 4 gives $$ P(N\geq3)=1-e^{-4}\left(1+4+\frac{4^2}{2}\right) =1-13e^{-4}=0.7619 $$ to four decimal places. The negative binomial probability is lower for this particular threshold, while its zero probability is higher: $1/16=0.0625$ versus the Poisson value $e^{-4}=0.0183$. Overdispersion shifts probability toward both tails; it does not mean that every high-count event must be more likely.

Common mistake avoided: “More dispersed” does not justify claiming that $P(N\geq3)$ must exceed its Poisson counterpart. Compare the actual tail sums for the threshold asked.

Moment Checks and Exam Traps

For data, a common moment-matching procedure replaces the population mean and variance by corresponding sample summaries. If the resulting sample variance exceeds the sample mean, the formulas give moment estimates of $r$ and $p$. These estimates are not guarantees that the negative binomial is the best model; they simply match two moments. Model choice and formal goodness-of-fit procedures are separate questions.

If a sample variance is at or below its mean, do not force the moment formula to produce a finite positive dispersion parameter. At equality, the moment fit approaches the Poisson limit. Below equality, the negative binomial overdispersion formula cannot match both moments. Also distinguish the variance of a single period from the variance of an aggregate over periods: the latter depends on whether the underlying risks or latent intensities are shared.

Check the convention. In this tutorial $p$ is the success probability and $1-p$ is the failure probability counted by $N$.
Check the moment condition. A finite overdispersed fit requires variance greater than mean.
Check the exposure assumption. A persistent latent intensity and independent period-specific intensities produce different aggregate variances.
Compare probabilities, not labels. Overdispersion increases variance, but whether a specified event becomes more or less likely depends on the event.

Key takeaway: For a negative binomial count with mean $\mu$ and variance $\sigma_N^2>\mu$, use $r=\mu^2/(\sigma_N^2-\mu)$ and $p=\mu/\sigma_N^2$. Then verify the moments, apply the matching count convention, and make the heterogeneity and exposure assumptions explicit before comparing probabilities.

Check Your Understanding

For each problem, use the failures-before-successes convention and round probabilities to four decimal places.

A hypothetical policy group has claim-count mean 5 and variance 15. Find the moment-matched $r$ and $p$, then calculate the probability of zero claims. (Answer: $r=2.5,\ p=1/3,\ P(N=0)=0.0642$)
A count model has $r=3$ and $p=0.4$. Find its mean and variance, then calculate $P(N=0)$. (Answer: Mean $=4.5$, variance $=11.25$, $P(N=0)=0.0640$)
A hypothetical portfolio has count mean 3 and variance 6. Find its negative binomial parameters and $P(N\leq1)$. Compare with the Poisson probability at the same mean. (Answer: $r=3,\ p=0.5$; negative binomial $P(N\leq1)=0.3125$, Poisson $P(N\leq1)=0.1991$)
One-year count moments are mean 1.5 and variance 2.25. A risk’s latent intensity persists for two years, with conditional Poisson counts. Find the two-year mean, variance, and probability of zero claims. (Answer: Mean $=3$, variance $=6$, $P(N_2=0)=0.1250$)
A set of observed counts has sample mean 4 and sample variance 3.5. Can a finite overdispersed negative binomial be moment-matched to both? State the relevant alternative indicated by equality of variance and mean. (Answer: No; variance is below the mean, so these moments do not support a finite overdispersed negative binomial fit. Equality would give the Poisson limiting case.)

What You'll Learn