Beta-Negative-Binomial Updating for Overdispersed Counts

Why Update a Negative Binomial Count Model?

Claim and event counts can vary more across risks than a simple fixed-rate count model allows. One way to represent that heterogeneity is to give each risk its own probability of a modeled event and treat that probability as uncertain. When observations are collected until a fixed number of the complementary events occurs, the resulting count has a negative binomial model. A beta prior for the probability then gives a tractable posterior and predictive distribution.

The stopping rule matters: this tutorial models the number of failures observed before a prespecified number of successes. It does not model the number of events in a fixed time interval. The words “success” and “failure” are labels for the two possible outcomes in the model; their interpretation depends on the application.

Definition: Let $p$ be the probability of success on each independent trial. For a fixed positive integer $r$, let $K$ be the number of failures before the $r$th success. Then $$ P(K=k\mid p)=\binom{k+r-1}{k}p^r(1-p)^k,\qquad k=0,1,2,\ldots. $$ This convention counts failures, not total trials. Its conditional mean and variance are $r(1-p)/p$ and $r(1-p)/p^2$, respectively.

Suppose $p$ is unknown and has a beta prior, $p\sim\operatorname{Beta}(\alpha,\beta)$, with density proportional to $p^{\alpha-1}(1-p)^{\beta-1}$ for $0<p<1$, where $\alpha,\beta>0$. The beta distribution is supported on probabilities and is conjugate to this likelihood: the posterior remains beta.

Deriving the Posterior Update

For one observation, the fixed-success count $r$ is part of the sampling design, while the observed number of failures is $k$. As a function of $p$, the likelihood is proportional to $p^r(1-p)^k$; the binomial coefficient does not depend on $p$. Multiplying by the prior density gives

p^{\alpha-1}(1-p)^{\beta-1}\,p^r(1-p)^k \ \propto\ p^{\alpha+r-1}(1-p)^{\beta+k-1}.

The expression on the right is the kernel of a beta density. Thus the posterior parameters increase by the observed success and failure counts, respectively.

Theorem: If $p\sim\operatorname{Beta}(\alpha,\beta)$ and $K=k$ failures are observed before a fixed $r$ successes, then $$ p\mid K=k\sim\operatorname{Beta}(\alpha+r,\beta+k). $$ For conditionally independent observations $K_j=k_j$, each collected up to a prespecified $r_j$ successes using the same $p$, the posterior is $\operatorname{Beta}(\alpha+\sum_j r_j,\beta+\sum_j k_j)$.

The second result follows because multiplying the individual likelihoods gives a factor proportional to $p^{\sum_j r_j}(1-p)^{\sum_j k_j}$. It assumes the observations are independent conditional on the same $p$, and that each success target was fixed by design. If each observation has a different latent probability, or the data were collected under a different stopping rule, this pooled update need not apply.

Worked Example: Updating and Finding an Exact Predictive Probability

A hypothetical portfolio has a shared unknown success probability $p\sim\operatorname{Beta}(3,2)$. For one modeled risk, observation continues until four successes occur; six failures are recorded first. Find the posterior distribution, the expected number of failures before two future successes, and the predictive probability of exactly two such failures. Use the failures-before-successes convention.

Method and posterior. The observed count has $r=4$ successes and $k=6$ failures. Add these to the corresponding prior parameters: $$ \alpha'=3+4=7,\qquad \beta'=2+6=8. $$ Therefore $p\mid\text{data}\sim\operatorname{Beta}(7,8)$.

Predictive mean. For a future count $K_{\mathrm{new}}$ taken until $r_{\mathrm{new}}=2$ successes, the conditional mean given $p$ is $2(1-p)/p$. For a beta posterior with parameters $a,b$, $E[(1-p)/p]=b/(a-1)$, provided $a>1$. Here: $$ E[K_{\mathrm{new}}\mid\text{data}] =2\frac{8}{7-1} =\frac{16}{6} =\frac{8}{3} \approx 2.6667. $$

Predictive probability. Exactly two failures before two successes has conditional probability $\binom{3}{2}p^2(1-p)^2$. Averaging over the posterior gives $$ P(K_{\mathrm{new}}=2\mid\text{data}) =3\frac{B(7+2,8+2)}{B(7,8)} =3\frac{(7)(8)(8)(9)}{(15)(16)(17)(18)} =\frac{12096}{73440} \approx 0.1647. $$ The beta-function ratio follows by applying the beta-function recurrence for the two added powers of $p$ and the two added powers of $1-p$. The probability is between zero and one, as required. The common mistake this method avoids is swapping the update counts: the four successes update $\alpha$, while the six failures update $\beta$.

Posterior Predictive Distribution

After observing data, write the posterior as $p\mid\text{data}\sim\operatorname{Beta}(a,b)$, where $a=\alpha+\sum_j r_j$ and $b=\beta+\sum_j k_j$. Let $K_{\mathrm{new}}$ count failures before a future fixed target $s$ successes. Conditional on $p$, this is negative binomial. To obtain its predictive distribution, average the conditional probability over the posterior density:

\begin{aligned} P(K_{\mathrm{new}}=k\mid\text{data}) &=\binom{k+s-1}{k} \frac{1}{B(a,b)} \int_0^1 p^{a+s-1}(1-p)^{b+k-1}\,dp\\ &=\binom{k+s-1}{k}\frac{B(a+s,b+k)}{B(a,b)},\qquad k=0,1,2,\ldots. \end{aligned}

This is the beta-negative-binomial predictive distribution under the stated parameterization. The integral is a beta function because its integrand has powers corresponding to beta parameters $a+s$ and $b+k$. The predictive distribution accounts for uncertainty in $p$, rather than substituting one estimate of $p$ into the conditional negative binomial.

Predictive Mean and Variance

For $p\mid\text{data}\sim\operatorname{Beta}(a,b)$, the future count to $s$ successes has conditional mean $s(1-p)/p$ and conditional variance $s(1-p)/p^2$. The required beta expectations exist for $a>1$ for the mean and $a>2$ for the variance. In particular,

E\!\left[\frac{1-p}{p}\right]=\frac{b}{a-1},\qquad E\!\left[\frac{1-p}{p^2}\right]=\frac{b(a+b-1)}{(a-1)(a-2)}.

Consequently, the predictive mean is $s b/(a-1)$. For the variance, apply the law of total variance: the predictive variance is the expected conditional variance plus the variance of the conditional mean. Since $\operatorname{Var}((1-p)/p)=b(a+b-1)/((a-1)^2(a-2))$, the result is

\operatorname{Var}(K_{\mathrm{new}}\mid\text{data}) = \frac{s\,b(a+b-1)}{(a-1)(a-2)} + \frac{s^2b(a+b-1)}{(a-1)^2(a-2)},\qquad a>2.

The first term is the average negative binomial variance conditional on $p$; the second is the variation in the conditional mean as the uncertain $p$ varies. Omitting the second term understates predictive variance. A practical way to retain it is to write both terms before simplifying.

Worked Example: Predictive Variance After Several Observations

A hypothetical monitoring process uses the same unknown success probability in two independent runs. The prior is $p\sim\operatorname{Beta}(2,3)$. Each run stops after two successes; the observed failure counts are one and four. Find the posterior and the mean and variance of the failures in a future run that stops after three successes.

Method. The two runs contribute $2+2=4$ successes and $1+4=5$ failures. Therefore the posterior is $$ p\mid\text{data}\sim\operatorname{Beta}(2+4,3+5)=\operatorname{Beta}(6,8). $$

For a future target of $s=3$ successes, the predictive mean is $$ E[K_{\mathrm{new}}\mid\text{data}] =\frac{3(8)}{6-1} =\frac{24}{5} =4.8. $$ Since $a=6>2$, the predictive variance formula applies. Its expected conditional variance term is $$ \frac{3(8)(6+8-1)}{(6-1)(6-2)} =\frac{312}{20} =15.6. $$ The parameter-uncertainty term is $$ \frac{3^2(8)(6+8-1)}{(6-1)^2(6-2)} =\frac{936}{100} =9.36. $$ Adding the terms gives $\operatorname{Var}(K_{\mathrm{new}}\mid\text{data})=15.6+9.36=24.96$ failures squared. The positive second term is the extra uncertainty from not knowing $p$; treating the posterior mean of $p$ as known would omit it.

Interpretation, Assumptions, and Exam Traps

A mixture over uncertain probabilities is more variable than a model that conditions on a single fixed probability in the same way: the total-variance decomposition adds variation in the conditional mean. This is one mechanism by which a beta-negative-binomial model represents overdispersed counts. Its use is justified by the sampling design and modeling assumptions, not merely by observing that a count sample has a large variance.

The central assumption is that, conditional on one $p$, trials are independent with constant success probability, and the count is observed until the fixed success target is reached. A count observed over a fixed duration has a different exposure-based likelihood. Do not apply the beta-negative-binomial update to such data simply because the counts are overdispersed; first identify the count-generating model and its observation rule.

Common exam trap: In the failures-before-$r$-successes convention, the likelihood contains $p^r(1-p)^k$. If a prior is $\operatorname{Beta}(3,2)$ and the data are four successes and six failures, reversing the roles would incorrectly give $\operatorname{Beta}(9,6)$. The correct posterior is $\operatorname{Beta}(7,8)$, with mean $7/15\approx0.4667$, not $9/15=0.6000$. Check which outcome the model calls a success before updating either parameter.

A useful exam routine is to state the convention, identify the fixed success target and observed failures, update the two beta parameters, and only then calculate predictive quantities. Predictive mean and variance formulas also have existence conditions: a posterior with $a\leq1$ does not give a finite predictive mean, and one with $a\leq2$ does not give a finite predictive variance. The model and its assumptions are aligned with the SOA Exam ASTAM (Advanced Short-Term Actuarial Mathematics) syllabus.

Worked Example: Checking a Zero-Failure Predictive Probability

For a hypothetical risk, a beta prior is $\operatorname{Beta}(4,2)$. A run ends at three successes, with two failures observed before the third success. Find the posterior mean of $p$, and the probability that a future run ending at two successes has no failures.

Method. Update by the observed three successes and two failures: $$ p\mid\text{data}\sim\operatorname{Beta}(4+3,2+2)=\operatorname{Beta}(7,4). $$ The posterior mean is $7/(7+4)=7/11\approx0.6364$.

For no failures before two successes, the conditional probability is $p^2$. The predictive probability is therefore the posterior expectation of $p^2$: $$ P(K_{\mathrm{new}}=0\mid\text{data}) =E[p^2\mid\text{data}] =\frac{7(8)}{(7+4)(7+4+1)} =\frac{56}{132} =\frac{14}{33} \approx0.4242. $$ This is not the square of the posterior mean: $(7/11)^2=49/121\approx0.4050$, which differs because $p$ is uncertain. The beta moment, rather than a plug-in probability, gives the correct predictive result.

Check Your Understanding

For each problem, use the failures-before-a-fixed-number-of-successes convention. Give probabilities to four decimals where needed.

A prior is $\operatorname{Beta}(2,3)$. A run stops at two successes and records one failure. Find the posterior and the expected failures in a future run ending at one success. (Answer: $\operatorname{Beta}(4,4)$; $4/3$ failures.)
A posterior is $\operatorname{Beta}(3,2)$. Find the predictive probability of zero failures before one future success. (Answer: $3/5=0.6000$.)
A prior is $\operatorname{Beta}(2,2)$. Two independent runs using the same $p$ each stop at two successes; their observed failure counts are zero and three. Find the posterior and the expected failures in a future run ending at two successes. (Answer: $\operatorname{Beta}(6,5)$; $2$ failures.)
After data, $p\mid\text{data}\sim\operatorname{Beta}(5,4)$. Find the predictive variance of failures before two future successes. (Answer: $8$ failures squared.)
After data, $p\mid\text{data}\sim\operatorname{Beta}(4,3)$. Find the predictive probability of exactly one failure before two future successes. (Answer: $5/21=0.2381$.)

What You'll Learn