Choosing Models for Overdispersed Counts and Grouped Responses

Start With the Response and Its Sampling Unit

A large variance relative to a fitted mean can signal overdispersion, but it does not identify a model by itself. The relevant choice depends on what one observation represents, how its mean changes with predictors or exposure, and whether the variance pattern is consistent with a particular probability distribution. A count recorded once per subject and a number of successes out of a known number of trials require different candidate models.

The comparisons here assume that the observations being modeled are independent across sampling units, conditional on the predictors and any specified group structure. If observations are dependent for another reason, changing the response distribution alone may not address that dependence.

Definition: For a count with conditional mean \(\mu_i\), a Poisson model has \(\operatorname{Var}(Y_i\mid x_i)=\mu_i\). The common negative-binomial (NB2) model has \(\operatorname{Var}(Y_i\mid x_i)=\mu_i+\alpha\mu_i^2\), where \(\alpha>0\) is a dispersion parameter; its Poisson limit is \(\alpha=0\). A quasi-Poisson specification instead sets \(\operatorname{Var}(Y_i\mid x_i)=\phi\mu_i\), with \(\phi>0\), without specifying a full probability distribution.

For grouped successes, \(Y_i\) is the number of successes among \(m_i\) trials. Under a binomial model with success probability \(p_i\), its conditional mean and variance are \(m_i p_i\) and \(m_i p_i(1-p_i)\). A beta-binomial model allows the success probability to vary among groups, producing extra variation in the group totals. A quasi-binomial specification multiplies the binomial variance by a scale parameter \(\phi\), but does not by itself give a binomial or beta-binomial probability distribution.

What the Variance Pattern Can Tell You

Variance patterns are most useful when considered across a range of fitted means or group sizes, not from a single dispersion ratio. For counts, Poisson variance grows linearly with the mean. NB2 variance includes a quadratic term, so its variance-to-mean ratio increases with the mean. Quasi-Poisson variance is also linear in the mean, but with a multiplier that may exceed one.

Result: At a particular mean \(\mu>0\), NB2 has variance-to-mean ratio \(1+\alpha\mu\), while quasi-Poisson has ratio \(\phi\). These patterns agree at that mean when \(\phi=1+\alpha\mu\), but if \(\alpha>0\), no single constant \(\phi\) can make them agree at two distinct positive means.

Proof. Dividing the NB2 variance by \(\mu\) gives \((\mu+\alpha\mu^2)/\mu=1+\alpha\mu\). The quasi-Poisson ratio is \(\phi\mu/\mu=\phi\). If these ratios agree at distinct \(\mu_1\) and \(\mu_2\), then \(1+\alpha\mu_1=\phi=1+\alpha\mu_2\), so \(\alpha(\mu_1-\mu_2)=0\). Since \(\alpha>0\) and \(\mu_1\ne\mu_2\), this is impossible. \(\square\)

This distinction is a guide, not a reliable model-selection test from a small sample. A quasi-Poisson model may be a practical variance adjustment when the mean model is the main target and a linear variance scaling is adequate. An NB2 model is a distributional choice with a different variance pattern and a specified likelihood, which can support likelihood-based comparisons and prediction from the fitted distribution.

Worked Example: Comparing Count Variance Patterns

Suppose six independent count observations have values \(1,2,3,3,6,9\), and initially treat them as replicates with a common mean. Their sample mean is \(\bar y=(1+2+3+3+6+9)/6=24/6=4\). The sum of squared deviations from 4 is \(9+4+1+1+4+25=44\), so the sample variance is \(44/(6-1)=8.8\).

The Poisson model with mean 4 has variance 4, substantially below this descriptive sample variance. Matching the NB2 variance to 8.8 by a method-of-moments calculation gives \(8.8=4+\alpha(4^2)=4+16\alpha\), hence \(\alpha=4.8/16=0.30\). The corresponding quasi-Poisson variance multiplier at this mean is \(\phi=8.8/4=2.2\).

These calculations show the alternatives clearly: NB2 gives variance \(4+0.30(16)=8.8\), while quasi-Poisson gives variance \(2.2(4)=8.8\). They do not establish that either model fits: six observations provide little information about how variance changes with the mean, and a moment match is not a likelihood fit. With covariates or varying exposures, compare the pattern across fitted means and examine whether the independence and mean assumptions are plausible.

Grouped Successes: Binomial, Beta-Binomial, or Quasi-Binomial?

The number of trials \(m_i\) is essential information. Under independent Bernoulli trials with a common probability \(p_i\) within group \(i\), the binomial variance is \(m_i p_i(1-p_i)\). In a beta-binomial model, the group-specific probability varies across groups. In the common mean-correlation parameterization, \(\rho\) is the intraclass correlation of two trial outcomes in the same group, with \(0\leq\rho<1\). The resulting variance is \(m_i p_i(1-p_i)\{1+(m_i-1)\rho\}\). At \(\rho=0\), it reduces to the binomial variance.

Result: For a beta-binomial group with \(m\geq2\), the ratio of its variance to its binomial variance is \(1+(m-1)\rho\). If \(\rho>0\), this ratio increases with \(m\); a single constant quasi-binomial multiplier cannot match it at two distinct trial counts.

Derivation. Write \(V_{\mathrm{BB}}=mp(1-p)\{1+(m-1)\rho\}\) and \(V_{\mathrm{Bin}}=mp(1-p)\), for \(0<p<1\). Dividing gives \(V_{\mathrm{BB}}/V_{\mathrm{Bin}}=1+(m-1)\rho\). If this ratio equals a constant \(\phi\) at distinct trial counts \(m_1\) and \(m_2\), then \((m_1-m_2)\rho=0\). For \(m_1\ne m_2\), this requires \(\rho=0\). Thus positive beta-binomial correlation produces a trial-count-dependent inflation, unlike a constant quasi-binomial multiplier. \(\square\)

This comparison is especially informative when groups have unequal trial counts. A quasi-binomial model can describe extra variance proportional to the binomial variance, whereas beta-binomial variance increases further with the number of trials when \(\rho>0\). The pattern helps distinguish candidate structures, but it does not prove that group probabilities are beta distributed or that the beta-binomial is the only suitable model.

Worked Example: Unequal Group Sizes and Extra-Binomial Variation

Suppose groups each have 10 trials and the observed success counts are \(0,2,4,6,8,10\). Their mean is \(30/6=5\), so the fitted common success proportion is \(\hat p=5/10=0.5\). The squared deviations from 5 sum to \(25+9+1+1+9+25=70\), giving sample variance \(70/5=14\).

The binomial variance at \(m=10\) and \(p=0.5\) is \(10(0.5)(0.5)=2.5\). The sample variance is therefore \(14/2.5=5.6\) times that value. Matching the beta-binomial variance to 14 gives \(14=2.5\{1+9\rho\}\), so \(1+9\rho=5.6\) and \(\rho=4.6/9\approx0.511\).

A quasi-binomial moment match would instead use \(\phi=14/2.5=5.6\). Both calculations match the variance at this one group size; they differ in their implications at other sizes. For \(m=5\), the beta-binomial multiplier at \(\rho\approx0.511\) is \(1+4(0.511)\approx3.044\), while the quasi-binomial multiplier remains 5.6. The six groups cannot settle which variance pattern is appropriate, but groups of varied sizes would make this distinction empirically relevant.

Use Likelihood Evidence Only When It Exists

The Poisson, negative-binomial, binomial, and beta-binomial models specify probability distributions and therefore have likelihoods. Comparing likelihood evidence requires models for the same observed response on the same observational units; changing from individual trials to grouped totals, dropping observations, or changing the response changes what is being compared. For ordinary AIC comparisons, both candidates need maximized likelihoods for those same data, and the parameter counts must include estimated dispersion parameters.

Quasi-likelihood specifies a relationship between the mean and variance, but not a normalized probability mass function for the response. Consequently, an ordinary likelihood, ordinary likelihood-ratio statistic, and ordinary AIC are not supplied by a quasi-Poisson or quasi-binomial fit. A quasi fit can still be useful for estimating a mean relationship and adjusting uncertainty under its variance assumption; it should not be ranked against a full-likelihood fit by pretending that it has an ordinary likelihood.

Worked Example: Comparing Count Models With AIC

For the same 40 count observations and the same mean-predictor structure, suppose maximized-likelihood output reports a Poisson log-likelihood of \(-78.4\) with one estimated parameter and an NB2 log-likelihood of \(-70.9\) with two estimated parameters. The second parameter is the estimated NB2 dispersion. These are stipulated model-fit outputs; the calculation below illustrates how to compare them, rather than deriving the fits from raw observations.

For a model with \(k\) estimated parameters and maximized log-likelihood \(\ell\), \( \mathrm{AIC}=-2\ell+2k\). Thus the Poisson AIC is \(-2(-78.4)+2(1)=156.8+2=158.8\). The NB2 AIC is \(-2(-70.9)+2(2)=141.8+4=145.8\). The difference is \(158.8-145.8=13.0\) in favor of NB2 among these two likelihood models.

This is evidence about relative fit adjusted for the parameter penalty, not proof that NB2 is the true distribution. It does not compare NB2 with a quasi-Poisson fit by ordinary AIC. Nor should a likelihood-ratio reference distribution be chosen automatically when testing the Poisson limit: the null value \(\alpha=0\) is a boundary value, so boundary-test methods are relevant.

A Practical Selection Sequence

Identify the response unit.
Use a count distribution for counts, and retain the number of trials when the response is a grouped success count.

Specify the mean structure.
Assess candidate variance patterns only after the mean and any exposure or trial-count information have been represented appropriately.

Compare variance implications.
Ask whether extra variance appears roughly proportional to the mean or binomial variance, or increases in the way NB2 or beta-binomial structure predicts.

Use the evidence available.
Compare likelihood-based candidates on identical data using suitable likelihood methods; use quasi-likelihood when its variance adjustment is the intended tool, not as though it supplied an ordinary likelihood.

For example, Poisson versus NB2 is a count-model comparison; binomial versus beta-binomial is a grouped-response comparison. Quasi-Poisson and quasi-binomial are variance specifications, not alternative full probability distributions. In either setting, selection should also consider whether the sampling structure supports the candidate model and whether the practical purpose is mean estimation, distributional prediction, or likelihood-based comparison.

Common Mistake

Wrong reasoning: “The observed variance is more than twice the Poisson variance, so the negative-binomial model must be correct.” A single variance ratio does not reveal whether overdispersion follows a quadratic NB2 pattern, a linear quasi-Poisson pattern, omitted mean structure, dependence, or some other feature of the data.

Correction: Treat a dispersion calculation as evidence to investigate, not a verdict. Compare how candidate variances change across fitted means or trial counts, respect the observational and grouping structure, and use likelihood comparisons only for models that specify likelihoods for the same response data.

Key takeaway: Choose among Poisson, negative-binomial, binomial, beta-binomial, and quasi-likelihood specifications by combining response structure, variance-pattern implications, and valid likelihood evidence. No single dispersion estimate makes the choice automatically.

Check Your Understanding

Use the model assumptions and variance patterns, not just the size of a dispersion estimate.

At two distinct positive means, why can a quasi-Poisson model with one fixed \(\phi\) not match an NB2 model with \(\alpha>0\) at both means?
What additional information must accompany a grouped success count to assess binomial versus beta-binomial variance?
For groups with unequal trial counts, how does the beta-binomial variance multiplier change as \(m_i\) increases when \(\rho>0\)?
Why is ordinary AIC not available for a quasi-binomial fit?
What conditions should be held fixed when comparing likelihood evidence for two count models?