Beta-Binomial Laws from Random Success Probabilities

Introduction

A binomial model assumes that every trial uses the same success probability. In some settings, however, that probability varies from one group, patient, or batch to another. If the trials in a group share an unknown probability, treating them as independent after averaging over that uncertainty misses an important feature: observing successes gives information about the probability governing the other trials.

A convenient model assigns the shared probability a beta distribution and then uses a binomial model conditional on its value. Integrating over the random probability gives the beta-binomial distribution. The resulting count has a simple probability mass function, but its variance is larger than the variance of a binomial count with the same mean. The same shared probability also induces positive dependence between distinct trial outcomes.

Model and Probability Mass Function

Definition: Let $\alpha>0$ and $\beta>0$. Let $\Theta$ have the beta distribution with density $$ f_{\Theta}(\theta)=\frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)},\qquad 0<\theta<1, $$ where $$ B(a,b)=\int_0^1 t^{a-1}(1-t)^{b-1}\,dt. $$ Conditional on $\Theta=\theta$, let $X_1,\ldots,X_n$ be independent Bernoulli random variables with success probability $\theta$. The count $S_n=\sum_{i=1}^n X_i$ has the beta-binomial distribution with parameters $n,\alpha,\beta$.

The conditional independence assumption means that once the shared success probability is fixed, the trial outcomes are independent. The unconditional model is different: all outcomes depend on the same random $\Theta$. To obtain the count distribution, average the conditional binomial probability over the beta density.

Theorem: For $k=0,1,\ldots,n$, the beta-binomial probability mass function is $$ P(S_n=k)=\binom{n}{k}\frac{B(\alpha+k,\beta+n-k)}{B(\alpha,\beta)}. $$ It is zero for integer $k$ outside $0,\ldots,n$.

Derivation. Conditional on $\Theta=\theta$, the count has the binomial probability $\binom{n}{k}\theta^k(1-\theta)^{n-k}$. The law of total probability and the beta density therefore give

\begin{aligned} P(S_n=k) &=\int_0^1 P(S_n=k\mid \Theta=\theta)f_{\Theta}(\theta)\,d\theta\\ &=\binom{n}{k}\int_0^1 \theta^k(1-\theta)^{n-k} \frac{\theta^{\alpha-1}(1-\theta)^{\beta-1}}{B(\alpha,\beta)}\,d\theta\\ &=\binom{n}{k}\frac{1}{B(\alpha,\beta)} \int_0^1\theta^{\alpha+k-1}(1-\theta)^{\beta+n-k-1}\,d\theta\\ &=\binom{n}{k}\frac{B(\alpha+k,\beta+n-k)}{B(\alpha,\beta)}. \end{aligned}

The integral is finite because $\alpha+k>0$ and $\beta+n-k>0$, so it is exactly the beta function in the numerator. This proves the stated mass function. $\square$

Worked Example: A Particular Beta-Binomial Probability

Suppose $\Theta\sim\operatorname{Beta}(2,3)$, and conditional on $\Theta$, four trials are independent Bernoulli trials with success probability $\Theta$. Find the probability of exactly two successes.

Using the mass function, $$ P(S_4=2)=\binom{4}{2}\frac{B(4,5)}{B(2,3)}. $$ For positive integers $r,s$, $B(r,s)=(r-1)!(s-1)!/(r+s-1)!$. Thus $$ B(4,5)=\frac{3!4!}{8!}=\frac{1}{280}, \qquad B(2,3)=\frac{1!2!}{4!}=\frac{1}{12}. $$ Therefore $$ P(S_4=2)=6\left(\frac{1/280}{1/12}\right) =6\left(\frac{12}{280}\right) =\frac{9}{35}. $$ The probability is approximately $0.2571$.

Mean, Variance, and Dependence

Write $p=\alpha/(\alpha+\beta)$, the mean of $\Theta$, and $s=\alpha+\beta$. The conditional mean and variance of a binomial count are $n\Theta$ and $n\Theta(1-\Theta)$, respectively. Averaging these conditional quantities alone is not enough for the unconditional variance: the conditional mean itself varies with $\Theta$.

Theorem: Under the beta-binomial model, $$ E[S_n]=n\frac{\alpha}{\alpha+\beta}, \qquad \operatorname{Var}(S_n) =n\frac{\alpha\beta(\alpha+\beta+n)} {(\alpha+\beta)^2(\alpha+\beta+1)}. $$ For distinct $i,j$, the trial indicators satisfy $$ \operatorname{Cov}(X_i,X_j) =\frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}, \qquad \operatorname{Corr}(X_i,X_j)=\frac{1}{\alpha+\beta+1}. $$

Proof. The beta distribution has $$ E[\Theta]=\frac{\alpha}{s}, \qquad \operatorname{Var}(\Theta)=\frac{\alpha\beta}{s^2(s+1)}. $$ These identities follow, for example, by evaluating $E[\Theta]=B(\alpha+1,\beta)/B(\alpha,\beta)$ and $E[\Theta^2]=B(\alpha+2,\beta)/B(\alpha,\beta)$, and using $B(a+1,b)=aB(a,b)/(a+b)$. The conditional mean gives $$ E[S_n]=E\{E[S_n\mid\Theta]\}=E[n\Theta]=n\frac{\alpha}{s}. $$ For the variance, the law of total variance gives $$ \operatorname{Var}(S_n)=E\{\operatorname{Var}(S_n\mid\Theta)\} +\operatorname{Var}\{E[S_n\mid\Theta]\}. $$ The first term is $nE[\Theta(1-\Theta)]$. Since $E[\Theta(1-\Theta)]=E[\Theta]-E[\Theta^2]$, or equivalently $E[\Theta(1-\Theta)]=\alpha\beta/[s(s+1)]$, it equals $n\alpha\beta/[s(s+1)]$. The second term is $n^2\operatorname{Var}(\Theta)=n^2\alpha\beta/[s^2(s+1)]$. Adding and simplifying, $$ \operatorname{Var}(S_n) =\frac{n\alpha\beta}{s(s+1)} +\frac{n^2\alpha\beta}{s^2(s+1)} =\frac{n\alpha\beta(s+n)}{s^2(s+1)}. $$ This is the claimed variance.

For distinct trial indicators, $E[X_i]=E\{E[X_i\mid\Theta]\}=E[\Theta]=p$. Conditional independence gives $E[X_iX_j\mid\Theta]=\Theta^2$, and hence $$ \operatorname{Cov}(X_i,X_j) =E[\Theta^2]-\{E[\Theta]\}^2 =\operatorname{Var}(\Theta) =\frac{\alpha\beta}{s^2(s+1)}. $$ Each $X_i$ is marginally Bernoulli with success probability $p$, so $\operatorname{Var}(X_i)=p(1-p)=\alpha\beta/s^2$. Dividing the covariance by this variance gives $\operatorname{Corr}(X_i,X_j)=1/(s+1)$. The covariance is positive for $\alpha,\beta>0$, establishing the induced dependence. $\square$

Worked Example: Comparing Beta-Binomial and Binomial Variability

Let $\Theta\sim\operatorname{Beta}(3,7)$, and conditionally observe $n=10$ trials. The marginal success probability is $3/(3+7)=0.3$, so the mean count is $$ E[S_{10}]=10(0.3)=3. $$ The beta-binomial variance is $$ \operatorname{Var}(S_{10}) =\frac{10(3)(7)(10+10)}{(10)^2(11)} =\frac{4200}{1100} =\frac{42}{11}\approx 3.818. $$ If the success probability were fixed at $0.3$, a binomial count would instead have variance $10(0.3)(0.7)=2.1$.

The excess variability comes from the shared random probability. Here $$ \operatorname{Cov}(X_i,X_j)=\operatorname{Var}(\Theta) =\frac{21}{100(11)}=\frac{21}{1100} \quad (i\ne j). $$ Indeed, summing the individual variances and all pairwise covariances gives $$ \operatorname{Var}(S_{10}) =10(0.3)(0.7)+10(9)\frac{21}{1100} =2.1+1.71818\ldots =\frac{42}{11}. $$

Updating the Success Probability After Observations

The shared-probability model also makes the dependence intuitive. Observing successes shifts what is plausible about $\Theta$, which changes the probability of success on another trial. If among $m$ observed trials there are $r$ successes and $m-r$ failures, the likelihood for $\theta$ is proportional to $\theta^r(1-\theta)^{m-r}$. Multiplying by the beta density shows that the conditional distribution of $\Theta$ given those outcomes is $\operatorname{Beta}(\alpha+r,\beta+m-r)$. Consequently, the predictive probability that the next trial succeeds is the mean of this updated beta distribution: $$ P(X_{m+1}=1\mid\text{observed outcomes}) =\frac{\alpha+r}{\alpha+\beta+m}. $$

Worked Example: Predicting a Trial After One Success

Take a uniform prior, $\Theta\sim\operatorname{Beta}(1,1)$. The first trial succeeds. The updated distribution is $\operatorname{Beta}(2,1)$, so the probability that the second trial succeeds, conditional on the first success, is $$ P(X_2=1\mid X_1=1)=\frac{1+1}{1+1+1}=\frac{2}{3}. $$ Before observing the first trial, the success probability for either trial was $1/2$. The increase to $2/3$ after a success is a direct consequence of learning about their common $\Theta$; it would not occur if the trials were marginally independent with success probability $1/2$.

Common Mistake and Practical Interpretation

A common mistake is to say, “Each trial has average success probability $p$, so the count is binomial with parameters $n,p$.” Having the same marginal probability for every trial is not enough for a binomial distribution; the trials must also be independent. Under the beta-binomial model, they are independent only conditional on $\Theta$. After $\Theta$ is averaged out, two successes tend to occur together because both are more likely when the shared probability happens to be high.

The beta-binomial model is useful when outcomes are clustered and the success rate varies across clusters. It accommodates greater count variability than a binomial model with the same mean. The concentration $s=\alpha+\beta$ controls the strength of the induced association: the pairwise correlation is $1/(s+1)$, so larger $s$ means weaker dependence. The mean success probability $p=\alpha/s$ and the concentration $s$ together provide an alternative way to describe the beta parameters.

Key takeaway: Mixing a binomial count over a shared beta-distributed probability gives a beta-binomial count. The random probability preserves the mean success rate on average but increases count variance and induces positive dependence among trials.

Check Your Understanding

Use the conditional model and the beta-binomial formulas to answer these questions.

State the conditions on $\alpha$ and $\beta$, and describe the conditional distribution of the trial indicators.
For a beta-binomial count with $n=5$, write the formula for the probability of exactly one success.
Explain why the unconditional variance is not obtained merely by averaging the conditional variance.
For $\alpha=2,\beta=2$, calculate the marginal success probability and the correlation between two distinct trials.
With a $\operatorname{Beta}(2,3)$ model, two successes and one failure have been observed. What is the predictive probability of success on the next trial?