Deriving Negative Binomial Variance

Why the Negative Binomial Has Extra Variance

A Poisson count model assumes that, conditional on its expected count, the variance equals the mean. Insurance counts can vary more than that assumption allows. The negative binomial model represents this extra variation by allowing the underlying claim rate to vary across observations. Its variance then has a linear term, the mean, and a quadratic term that reflects rate heterogeneity.

The focus here is the derivation and interpretation of that variance. Use \(\mu\) for the mean count and \(r\) for the positive shape parameter. Under the parameterization used throughout this tutorial, the variance is \(\mu+\mu^2/r\). A larger \(r\) means less additional variation when the mean is held fixed.

Definition: A negative binomial count \(N\) with mean \(\mu>0\) and shape \(r>0\) can be constructed by first drawing a Poisson rate \(\Lambda\) from a Gamma distribution with shape \(r\) and rate \(r/\mu\), then drawing \(N\) conditionally as Poisson with rate \(\Lambda\). Thus, \(\Lambda\sim\operatorname{Gamma}(r,\text{rate}=r/\mu)\) and \(N\mid\Lambda\sim\operatorname{Poisson}(\Lambda)\). The resulting count has mean \(\mu\) and variance \(\mu+\mu^2/r\).

This is a mixture model: individual counts are Poisson once their rates are known, but the rates vary. The Gamma distribution is parameterized here by shape and rate, not shape and scale. For shape \(r\) and rate \(\beta\), its mean is \(r/\beta\) and variance is \(r/\beta^2\). Those parameter definitions matter when deriving the count moments.

Deriving the Mean and Variance

For the Gamma rate in the definition, \(\beta=r/\mu\). Its moments are therefore

E[\Lambda]=\frac{r}{r/\mu}=\mu,\qquad \operatorname{Var}(\Lambda)=\frac{r}{(r/\mu)^2}=\frac{\mu^2}{r}.

Conditional on \(\Lambda\), the count is Poisson, so both its conditional mean and its conditional variance equal \(\Lambda\). Apply the laws of total expectation and total variance:

E[N]=E\big[E[N\mid\Lambda]\big] =E[\Lambda]=\mu,

\begin{aligned} \operatorname{Var}(N) &=E\big[\operatorname{Var}(N\mid\Lambda)\big] +\operatorname{Var}\big(E[N\mid\Lambda]\big)\\ &=E[\Lambda]+\operatorname{Var}(\Lambda)\\ &=\mu+\frac{\mu^2}{r}. \end{aligned}

The first variance component, \(E[\Lambda]=\mu\), is the average of the conditional Poisson variances. The second, \(\operatorname{Var}(\Lambda)=\mu^2/r\), is the variance of the conditional means across observations. It is this second component that creates variation beyond the Poisson variance.

Theorem: Under the Poisson-Gamma mixture above, \(E[N]=\mu\) and \(\operatorname{Var}(N)=\mu+\mu^2/r\). Equivalently, if \(\alpha=1/r\), then \(\operatorname{Var}(N)=\mu+\alpha\mu^2\). This form assumes a positive mean and a positive finite shape parameter; it is not the variance formula for a Poisson model with a fixed rate.

The size-probability form is another way to describe the same negative binomial distribution. If \(N\) counts failures before \(r\) successes, its probability mass function is

P(N=n)=\frac{\Gamma(n+r)}{\Gamma(r)n!}p^r(1-p)^n,\qquad n=0,1,2,\ldots,

where \(0<p<1\). In the mixture parameterization, \(p\) is the Gamma rate divided by that rate plus one:

p=\frac{r/\mu}{r/\mu+1}=\frac{r}{r+\mu}, \qquad 1-p=\frac{\mu}{r+\mu}.

Thus the size-probability form has mean \(r(1-p)/p\) and variance \(r(1-p)/p^2\). Substituting \(p=r/(r+\mu)\) recovers the mean \(\mu\) and variance \(\mu+\mu^2/r\). This conversion is useful because software and references do not all use the same parameterization.

Identify the parameterization.
Check whether the given values are the mean and shape \((\mu,r)\), or the size and probability \((r,p)\).

Find the mean if needed.
For size \(r\) and probability \(p\), use \(\mu=r(1-p)/p\).

Calculate the variance.
Use \(\mu+\mu^2/r\), or equivalently \(r(1-p)/p^2\), with parameters from the same convention.

Worked Applications

Worked Example: Separating the Two Sources of Count Variance

A hypothetical insurer models the number \(N\) of claims for a specified policy-year segment using a Poisson-Gamma mixture. The latent claim rate is \(\Lambda\sim\operatorname{Gamma}(r=2,\text{rate}=0.5)\), and \(N\mid\Lambda\sim\operatorname{Poisson}(\Lambda)\). Find the mean and variance of \(N\), and separate the variance into its conditional Poisson and rate-heterogeneity components. This is a count model; no interest or mortality assumptions enter the calculation.

Method. First calculate the moments of the Gamma rate, then apply total expectation and total variance. The Gamma mean is

E[\Lambda]=\frac{2}{0.5}=4.

Its variance is

\operatorname{Var}(\Lambda)=\frac{2}{(0.5)^2} =\frac{2}{0.25}=8.

Conditional on the rate, the Poisson count has \(E[N\mid\Lambda]=\Lambda\) and \(\operatorname{Var}(N\mid\Lambda)=\Lambda\). Therefore,

E[N]=E[\Lambda]=4

and

\begin{aligned} \operatorname{Var}(N) &=E[\operatorname{Var}(N\mid\Lambda)] +\operatorname{Var}(E[N\mid\Lambda])\\ &=E[\Lambda]+\operatorname{Var}(\Lambda)\\ &=4+8=12. \end{aligned}

The conditional Poisson contribution is \(4\), and the variation in rates contributes \(8\). As a check, the Gamma rate can be written as \(r/\mu=2/4=0.5\), so the mean-shape formula gives \(4+4^2/2=4+8=12\). Common mistake avoided: using only the conditional Poisson variance and reporting \(4\) ignores the variation in \(\Lambda\).

Worked Example: Converting Size and Probability to Mean and Variance

For a hypothetical policy group, the claim count follows the negative binomial probability mass function with size \(r=5\) and success probability \(p=0.75\), counting failures before the fifth success. Find the expected count and variance, then verify the variance using the mean-shape form. No discounting or mortality assumptions are involved; the specified distribution is a negative binomial count model.

Method. Use the size-probability moments first. Since \(1-p=1-0.75=0.25\), the mean is

\mu=\frac{r(1-p)}{p} =\frac{5(0.25)}{0.75} =\frac{1.25}{0.75} =\frac{5}{3}.

The variance in this parameterization is

\operatorname{Var}(N) =\frac{r(1-p)}{p^2} =\frac{5(0.25)}{(0.75)^2} =\frac{1.25}{0.5625} =\frac{20}{9} \approx 2.2222.

Verify with the mean-shape formula. Here \(\mu=5/3\), so

\mu+\frac{\mu^2}{r} =\frac{5}{3}+\frac{(5/3)^2}{5} =\frac{5}{3}+\frac{25/9}{5} =\frac{15}{9}+\frac{5}{9} =\frac{20}{9}.

The two forms agree. Common mistake avoided: inserting \(p=0.75\) as though it were the mean, or using \(r(1-p)/p\) as the variance, mixes up the size-probability formulas. The variance is \(r(1-p)/p^2\).

Worked Example: Recovering Shape From a Mean and Variance

A hypothetical book of small commercial policies has a modeled claim count with mean \(8\) and variance \(24\). Assume a negative binomial model with the same finite shape parameter across the policies being modeled. Find the shape parameter and the additional variance beyond the mean. Then, if a revised segment has mean count \(16\) with that same shape, find its variance. These are count moments, so neither interest nor mortality assumptions are applicable.

Method. Start with \(24=8+8^2/r\). The extra variance at mean \(8\) is

$$ 24-8=16. $$

Consequently,

\frac{8^2}{r}=16,\qquad r=\frac{64}{16}=4.

The shape is \(4\), and the additional variance at mean \(8\) is \(16\). For the revised segment, retain \(r=4\) and substitute the new mean \(\mu=16\):

\operatorname{Var}(N)=16+\frac{16^2}{4} =16+\frac{256}{4} =16+64=80.

The result is consistent with the model: doubling the mean from \(8\) to \(16\) quadruples the quadratic component, from \(8^2/4=16\) to \(16^2/4=64\). Common mistake avoided: treating the extra variance as a fixed amount of \(16\) after changing the mean. It is quadratic in the mean when \(r\) is held fixed.

Interpreting the Shape Parameter

At a fixed mean, the extra variance is \(\mu^2/r\). Increasing \(r\) reduces this component; decreasing \(r\) increases it. In the limit as \(r\) grows without bound, the extra term approaches zero, and the variance approaches the Poisson value \(\mu\). For any finite positive \(r\) and positive \(\mu\), the variance exceeds the mean.

The relative size of the extra component can also be seen from the variance-to-mean ratio:

\frac{\operatorname{Var}(N)}{E[N]} =\frac{\mu+\mu^2/r}{\mu} =1+\frac{\mu}{r}.

This ratio exceeds \(1\) for a finite shape parameter. It is not constant as the mean changes if \(r\) is held fixed. In applications, the alternative parameter \(\alpha=1/r\) expresses the variance as \(\mu+\alpha\mu^2\), where \(\alpha\) is the coefficient on the quadratic term.

The model’s construction explains why the extra term is quadratic: the Gamma rate variance is \(\mu^2/r\), not a constant independent of \(\mu\). This matters when interpreting model parameters and calculating uncertainty for counts. The mean-shape formula assumes the model is parameterized with the same stated \(r\); if the problem specifies a different parameterization or a changing shape, establish those details before applying the formula.

Common Exam Trap

A frequent error is to state that the negative binomial variance is \(\mu+\mu/r\), perhaps because the shape parameter is mistakenly applied to the mean rather than its square. For example, at \(\mu=6\) and \(r=3\), that incorrect expression gives \(6+6/3=8\). The correct calculation is

\operatorname{Var}(N)=6+\frac{6^2}{3} =6+\frac{36}{3} =18.

The incorrect answer is not merely a rounding difference: it omits the quadratic dependence on the mean. A useful check is to re-derive the extra term as \(\operatorname{Var}(\Lambda)=\mu^2/r\) from the Gamma mixing distribution.

Key takeaway: For a negative binomial count with mean \(\mu\) and shape \(r\), the variance is \(\mu+\mu^2/r\). The mean term comes from conditional Poisson variation; the quadratic term comes from variation in the latent Poisson rate.

Check Your Understanding

For each problem, use the parameterization stated and keep the shape or size convention explicit.

A hypothetical policy segment has negative binomial count mean \(\mu=3\) and shape \(r=3\). Find its variance. (Answer: \(6\))
A count follows the size-probability form with size \(r=2\) and probability \(p=0.5\), counting failures before the required successes. Find its mean and variance. (Answer: mean \(2\), variance \(4\))
A negative binomial count has mean \(10\) and variance \(35\). Find its shape \(r\) and its additional variance beyond the mean. (Answer: \(r=4\), additional variance \(25\))
A modeled segment has mean count \(6\) and shape \(r=3\). A revised segment has mean count \(12\), with the same shape. Find the revised variance and state the quadratic component. (Answer: variance \(60\), quadratic component \(48\))
For a negative binomial count with mean \(\mu=4\) and shape \(r=2\), calculate the variance-to-mean ratio. (Answer: \(3\))

What You'll Learn