Choosing an Overdispersion Model for Grouped Counts

Why Model Choice Depends on the Source of Variation

A count dataset can vary more than a Poisson model predicts for several different reasons. Counts may share a latent group-specific rate, or individual count records may have their own unobserved rate heterogeneity. A negative binomial model represents extra variation through a particular count distribution; an observation-level random effect represents it through a lognormal mixture. These choices can yield similar variances while making different assumptions about how the data were generated.

The grouping structure is therefore central. Repeated counts from the same site, person, or other cluster can reveal whether observations move together. A single total per group generally cannot reveal that pattern: its variation alone may be compatible with several mechanisms. Model selection should begin with the sampling design and a defensible account of the variation, not simply with a large variance-to-mean ratio.

Three Models and Their Assumptions

A Poisson random-intercept model assumes that observations are conditionally Poisson once their shared group intercept is known:

The log exposure is an offset, so doubling exposure doubles the conditional expected count when other predictors are held fixed. The model assumes that the group effects are independent across groups and normally distributed, and that counts are conditionally independent given the effects and predictors. It attributes excess variation shared across records in a group to \(b_i\); it does not add a separate source of row-specific overdispersion.

A negative binomial GLMM can instead, or additionally, allow a negative binomial response conditional on group effects. For the NB2 parameterization, its conditional mean and variance are

The NB2 variance grows quadratically in the conditional mean. In a negative binomial GLMM with group intercepts, observations are conditionally independent given the intercepts, but the negative binomial response variation remains after conditioning on them. Thus the model can represent both shared group heterogeneity and extra variation specific to a count record. That flexibility is useful only when the design and data support estimating both sources.

An observation-level random-effect model gives each record its own independent normal effect:

Here \(u_{ij}\) is independent across records and independent of the group effect. Given both effects, the response is Poisson. Integrating over \(u_{ij}\) produces a Poisson-lognormal mixture and extra variation at the observation level. Its mixing distribution differs from the gamma mixing underlying the standard negative binomial, although their means and variances can be close. The observation-level effect belongs to one count record, not to every individual counted within that record.

What Grouping Reveals: A Short Derivation

The key distinction is covariance. Under a Poisson random-intercept model, let \(a_{ij}=E_{ij}\exp(x_{ij}^{\mathsf T}\beta)\), so the conditional mean is \(a_{ij}\exp(b_i)\). For two different observations \(j\) and \(k\) in the same group, conditional independence gives zero conditional covariance. The law of total covariance then gives

The final line follows because \(b_i\sim N(0,\sigma_b^2)\), so \(E\{\exp(b_i)\}=\exp(\sigma_b^2/2)\) and \(\operatorname{Var}\{\exp(b_i)\}=\exp(\sigma_b^2)\{\exp(\sigma_b^2)-1\}\). The covariance is positive when \(\sigma_b^2>0\): counts in the same group tend to be jointly high or jointly low. Independent observation-level effects do not create this cross-record covariance once group effects are accounted for. If each group contributes just one count, there is no within-group pair with which to estimate this signature directly.

Variance Matching Does Not Identify the Mechanism

For one Poisson observation with a lognormal random effect, let the conditional mean be \(a\exp(u)\), with \(u\sim N(0,\sigma_u^2)\). Its marginal mean is \(m=a\exp(\sigma_u^2/2)\), and its marginal variance is

That quadratic-looking variance resembles NB2, whose variance is \(m+m^2/\theta\). Similar variance functions do not make the models interchangeable: their mixture distributions differ, as do their implied tail probabilities and other distributional features. Also, the mean parameterization matters. If the fixed part of the log mean is held constant while a random effect is added, integrating over that effect changes the marginal mean by a factor of \(\exp(\sigma^2/2)\). Comparisons must use a consistent interpretation of the mean.

A Practical Choice for Grouped Counts

The following sequence keeps the choice tied to the assumed source of variation. It presumes a defensible count mean model, including suitable predictors and exposure offsets where needed. If that mean structure is wrong, residual overdispersion can be a symptom of omitted structure rather than a reason to add a dispersion mechanism.

Diagnostics, Alternatives, and Limitations

After fitting a candidate model, check that it has represented both the mean and the dependence structure. Examine residual patterns against fitted means and predictors, and compare observed and model-predicted count distributions at the levels relevant to the design. For clustered records, assess whether residuals still co-move within groups. Check whether estimated random-effect variances are near zero, whether estimates are unstable, and whether the fit depends heavily on a small number of groups. Convergence warnings, singular fits, or implausibly large dispersion estimates call for investigation rather than automatic acceptance of a more complicated model.

Useful alternatives answer different questions. A Poisson model without random effects is appropriate only when conditional equidispersion and independence are credible after accounting for predictors and design. A negative binomial model is an alternative when extra count variation is best represented at the response level; adding group intercepts gives a model with both shared and record-level heterogeneity. An observation-level effect offers a latent log-rate interpretation but can be computationally demanding and, with one count per record, its variance may be difficult to distinguish from other dispersion mechanisms. The models should not be ranked by a single dispersion statistic.

Likelihood criteria can help compare models that are fitted to the same observations with comparable likelihood definitions, but they do not decide which latent-variable story is scientifically credible. Boundary estimates for variance components also complicate conventional likelihood-ratio reference distributions. When possible, inspect predictive performance and model-implied dependence as well as a fit criterion. A model may fit marginal count frequencies yet miss within-group association, or capture association while describing the count tails poorly.

There are important limits to what grouped counts can teach. Aggregation removes within-group information; sparse groups offer weak evidence about covariance; and mean-model misspecification can masquerade as overdispersion. NB2 and lognormal-Poisson mixtures may be hard to distinguish with modest samples. Model choice is therefore an explicit statement about the assumed source and scale of variation, supported—but not mechanically determined—by diagnostics.

Common Pitfall