Extra-Binomial Variation in Grouped Data

Why Grouped Counts Can Vary More Than a Binomial Predicts

A grouped-binomial observation records the number of successes among $m$ trials, often people or items within a group. The ordinary binomial model assumes that these trials are independent and have a common success probability $p$. When trials within a group are positively associated, or when groups differ in their underlying success probabilities, observed counts can vary more than that model predicts. This excess is called extra-binomial variation or overdispersion relative to the binomial model.

The variance alone does not tell us which mechanism produced the excess. Dependence among outcomes and unobserved differences in success probabilities can lead to the same mean and variance, even when their full count distributions differ. Keeping that distinction clear is important when describing grouped data.

Definition: For group $i$, let $Y_i$ be the number of successes in $m_i$ trials, with marginal success probability $p_i$. Under the binomial model, $Y_i\sim\operatorname{Binomial}(m_i,p_i)$, so $E[Y_i]=m_i p_i$ and $\operatorname{Var}(Y_i)=m_i p_i(1-p_i)$. A variance-inflation description replaces the latter relationship with $$ \operatorname{Var}(Y_i)=\phi\,m_i p_i(1-p_i), $$ where $\phi>1$ indicates variance larger than the binomial reference. This specifies a mean-variance relationship, not by itself a probability distribution for $Y_i$.

The factor $\phi$ is often called a dispersion or overdispersion factor. A common-$\phi$ description across groups is a working assumption; it need not follow from every mechanism that causes extra variation. In particular, the amount of inflation can depend on group size or on group-specific features.

Within-Group Dependence and the Intraclass Correlation

Write $X_{ij}=1$ if trial $j$ in group $i$ is a success and $X_{ij}=0$ otherwise. Then $Y_i=\sum_{j=1}^{m_i}X_{ij}$. Suppose the trials in a group have the same marginal success probability $p_i$, and every distinct pair has the same correlation $\rho_i$. This equal-correlation assumption describes exchangeable pairwise dependence; it does not, on its own, specify the joint distribution of all trials.

Result: Under these assumptions, $$ \operatorname{Var}(Y_i) =m_i p_i(1-p_i)\{1+(m_i-1)\rho_i\}. $$ Thus the variance-inflation factor relative to the binomial variance is $1+(m_i-1)\rho_i$.

To see the source of the formula, each indicator has variance $p_i(1-p_i)$. Each distinct pair has covariance $\rho_i p_i(1-p_i)$. Adding all individual variances and all ordered pair covariances gives

\begin{aligned} \operatorname{Var}(Y_i) &=\sum_{j=1}^{m_i}\operatorname{Var}(X_{ij}) +\sum_{j\ne k}\operatorname{Cov}(X_{ij},X_{ik})\\ &=m_i p_i(1-p_i)+m_i(m_i-1)\rho_i p_i(1-p_i)\\ &=m_i p_i(1-p_i)\{1+(m_i-1)\rho_i\}. \end{aligned}

There are $m_i(m_i-1)$ ordered pairs in the covariance sum. Positive $\rho_i$ therefore increases the count variance, and the increase is more pronounced for larger groups. When $\rho_i=0$, this variance agrees with the binomial variance, although zero pairwise correlation alone does not generally establish full independence.

Worked Example: Pairwise Dependence Within a Group

Suppose a group contains $m=8$ individuals. Each has success probability $p=0.4$, and every pair of outcomes has correlation $\rho=0.1$. Find the variance of the group’s success count and compare it with the independent-binomial variance.

The marginal variance of each indicator is $p(1-p)=0.4(0.6)=0.24$. The independent-binomial reference variance is therefore $8(0.24)=1.92$.

The within-group variance formula gives the inflation factor $1+(8-1)(0.1)=1.7$. Hence $\operatorname{Var}(Y)=8(0.24)(1.7)=3.264$.

The covariance calculation checks the same result: the eight individual variances sum to $8(0.24)=1.92$; each pair covariance is $0.1(0.24)=0.024$; and there are $8(7)=56$ ordered pairs. Thus $1.92+56(0.024)=1.92+1.344=3.264$.

Latent Heterogeneity and the Beta-Binomial Model

A different mechanism arises when trials are independent conditional on a group-specific probability, but that probability varies across groups. Let $P_i$ be a random success probability for group $i$. Given $P_i$, suppose the $m_i$ outcomes are independent Bernoulli variables with success probability $P_i$. This is a specific probability model once the distribution of $P_i$ is supplied.

For any distribution of $P_i$ supported on $[0,1]$, with $E[P_i]=p_i$, the law of total expectation and the law of total variance give

\begin{aligned} E[Y_i]&=m_i p_i,\\ \operatorname{Var}(Y_i) &=E\{\operatorname{Var}(Y_i\mid P_i)\} +\operatorname{Var}\{E(Y_i\mid P_i)\}\\ &=m_i E[P_i(1-P_i)]+m_i^2\operatorname{Var}(P_i)\\ &=m_i p_i(1-p_i)+m_i(m_i-1)\operatorname{Var}(P_i). \end{aligned}

The last equality uses $E[P_i(1-P_i)]=p_i-E[P_i^2]=p_i(1-p_i)-\operatorname{Var}(P_i)$. Conditional independence therefore does not imply marginal independence: two outcomes in the same group share $P_i$, and their covariance is $\operatorname{Var}(P_i)$. Their correlation is $\operatorname{Var}(P_i)/\{p_i(1-p_i)\}$, provided $0<p_i<1$.

Definition: In the beta-binomial model, $P_i\sim\operatorname{Beta}(\alpha,\beta)$, with $\alpha,\beta>0$, and $Y_i\mid P_i\sim\operatorname{Binomial}(m_i,P_i)$. The beta distribution has density proportional to $p^{\alpha-1}(1-p)^{\beta-1}$ on $0<p<1$. Integrating out $P_i$ gives the beta-binomial distribution.

Set $s=\alpha+\beta$. The beta mean and variance are $p=\alpha/s$ and $\operatorname{Var}(P)=\alpha\beta/\{s^2(s+1)\}=p(1-p)/(s+1)$. Substitution into the total-variance expression shows that

E[Y_i]=m_i p,\qquad \operatorname{Var}(Y_i)=m_i p(1-p)\left(1+\frac{m_i-1}{s+1}\right).

The factor $1/(s+1)$ is the within-group correlation induced by the shared latent probability. The beta distribution is one choice of mixing distribution, not a necessary consequence of observing extra-binomial variance.

Result: For $y=0,1,\ldots,m_i$, the beta-binomial probability mass function is $$ P(Y_i=y) =\binom{m_i}{y}\frac{B(\alpha+y,\beta+m_i-y)}{B(\alpha,\beta)}, $$ where $B(a,b)=\int_0^1 p^{a-1}(1-p)^{b-1}\,dp$ is the beta function.

To derive this probability, condition on $P_i=p$, multiply the binomial mass by the beta density, and integrate:

\begin{aligned} P(Y_i=y) &=\int_0^1 \binom{m_i}{y}p^y(1-p)^{m_i-y} \frac{p^{\alpha-1}(1-p)^{\beta-1}}{B(\alpha,\beta)}\,dp\\ &=\binom{m_i}{y} \frac{\int_0^1p^{\alpha+y-1}(1-p)^{\beta+m_i-y-1}\,dp} {B(\alpha,\beta)}\\ &=\binom{m_i}{y}\frac{B(\alpha+y,\beta+m_i-y)}{B(\alpha,\beta)}. \end{aligned}

The integral is finite because $\alpha,\beta>0$, and the integrand is nonnegative. This derivation specifies the entire count distribution, unlike merely specifying its variance.

Worked Example: A Beta-Binomial Group

Let $P\sim\operatorname{Beta}(2,3)$, and, conditional on $P$, let $Y\mid P\sim\operatorname{Binomial}(10,P)$. Find the mean, variance, induced correlation, and probability of no successes.

Here $s=2+3=5$, so $p=2/5=0.4$, and $\operatorname{Var}(P)=2(3)/\{5^2(6)\}=0.04$. The induced correlation is $0.04/\{0.4(0.6)\}=1/6$.

The mean is $E[Y]=10(0.4)=4$. The variance is $10(0.4)(0.6)\{1+9(1/6)\}=2.4(2.5)=6$. The independent-binomial variance with the same mean is $2.4$.

For no successes, the beta-binomial formula gives $P(Y=0)=B(2,13)/B(2,3)$. Since $B(2,13)=1/(13\cdot14)=1/182$ and $B(2,3)=1/(3\cdot4)=1/12$, this probability is $12/182=6/91$, approximately $0.0659$. The binomial probability would be $0.6^{10}$, approximately $0.00605$, illustrating how the mixing model changes tail probabilities as well as variance.

Same Mean and Variance, Different Count Distributions

A variance-inflation relationship cannot determine tail probabilities or the full shape of the count distribution. The next example compares two latent-probability models with the same first two moments but different probabilities for a group count.

Worked Example: Equal Inflation Does Not Identify the Model

Consider groups of $m=4$ trials with mean success probability $p=0.5$. In one model, $P\sim\operatorname{Beta}(2,2)$. In another, $P$ equals $0.5-\sqrt{0.05}$ or $0.5+\sqrt{0.05}$, each with probability $1/2$; outcomes are conditionally independent given $P$.

Both mixing distributions have $E[P]=0.5$ and $\operatorname{Var}(P)=0.05$. Thus both models give $\operatorname{Var}(Y)=4(0.25)+4(3)(0.05)=1.6$, and both have pairwise correlation $0.05/0.25=0.2$. Their variance inflation factor is $1+3(0.2)=1.6$.

For the two-point model, the probability of zero successes is \[ P(Y=0)=E[(1-P)^4] =\frac{(0.5+\sqrt{0.05})^4+(0.5-\sqrt{0.05})^4}{2} =0.14. \] For the beta model it is $B(2,6)/B(2,2)=(1/42)/(1/6)=1/7$, approximately $0.1429$.

The probabilities differ even though the means, variances, and pairwise correlations agree. Consequently, a common variance factor does not justify treating the count as beta-binomial, nor does it determine probabilities of particular outcomes.

What the Variance Description Does—and Does Not—Say

A quasi-binomial specification uses the binomial mean relationship together with an inflated variance, such as $\phi m_i p_i(1-p_i)$. It is useful when the mean and variance are the quantities needed for an estimating procedure, without asserting a particular count probability law. By contrast, a beta-binomial model makes a stronger claim: it specifies a distribution for a latent probability and, through mixing, a complete distribution for the count.

The distinction matters whenever inference depends on more than the first two moments. For example, likelihood calculations and exact probabilities require a probability model, not only a variance function. A variance description can still be appropriate for mean-based estimation, but it should not be presented as though it established the beta-binomial mechanism.

Dependence: outcomes within a group may be associated, so covariance terms add to the variance of the total.
Latent heterogeneity: groups may have different underlying success probabilities, producing dependence after those probabilities are averaged out.
Variance description: a factor $\phi$ summarizes variance relative to a reference; it does not identify the source or the full distribution.

Common Mistake

Wrong reasoning: “The grouped counts have variance greater than $m p(1-p)$, so they must follow a beta-binomial distribution.” Extra-binomial variance does not imply that specific model. The beta-binomial distribution follows only after assuming a beta-distributed latent probability and conditional binomial sampling. Other dependence structures or other mixing distributions can produce the same mean and variance but different probabilities for particular counts.

Correction: State separately what is assumed about the mean, what is assumed about the variance, and whether a full probability model is being imposed. If only the variance is specified, do not infer model-specific tail probabilities or likelihoods from it.

Key takeaway: Within-group positive dependence and latent heterogeneity both inflate grouped-binomial variance. The variance-inflation factor describes a moment relationship; the beta-binomial model is one specific distributional explanation, not a synonym for extra-binomial variation.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

For $m$ equally likely Bernoulli outcomes with pairwise correlation $\rho$, derive the variance of their sum and identify the variance-inflation factor.
In a latent-probability model, why can trials be conditionally independent but positively correlated marginally?
If $P\sim\operatorname{Beta}(\alpha,\beta)$, express the induced pairwise correlation in terms of $\alpha+\beta$.
What additional assumption, beyond an inflated mean-variance relationship, makes a beta-binomial a specific probability model?
Why can two models with equal grouped-count means and variances still assign different probabilities to a particular count?