Why the Direction of Dispersion Matters
For a Poisson count, the variance equals the mean. This equality is a useful benchmark: if a count response varies more than its mean would suggest, the data are overdispersed relative to that benchmark; if it varies less, they are underdispersed. The direction is consequential. A family designed to allow extra variation does not, simply by being flexible, necessarily represent counts whose variation is too small.
In a generalized linear mixed model (GLMM), the comparison must also specify what is being conditioned on. A Poisson response can have variance equal to its mean conditional on covariates and random effects, while its marginal variance is larger because the conditional mean varies across random effects. Thus a large raw variance-to-mean ratio does not by itself establish that the conditional count distribution is overdispersed.
The benchmark is a statement about a distribution or model, not a universal property of a dataset. For independent observations with a common Poisson mean, the sample variance and sample mean provide a rough descriptive comparison. When means differ across observations, pooling all counts can yield a variance that reflects those mean differences as well as variation around each mean. Likewise, in clustered data, pooling across clusters mixes conditional response variation with between-cluster variation.
Describing Dispersion From Counts
For a small set of counts \(y_1,\ldots,y_n\), the sample mean is \(\bar{y}=\sum_i y_i/n\), and the sample variance is \(s^2=\sum_i(y_i-\bar{y})^2/(n-1)\). The ratio \(s^2/\bar{y}\), when \(\bar{y}>0\), is a descriptive variance-to-mean ratio. A ratio above one suggests more variation than the common-mean Poisson benchmark; a ratio below one suggests less. It is not, on its own, a test, a model fit, or proof that a particular family is correct.
The common-mean comparison assumes that the observations are generated under comparable conditions and that the counts being compared have the same mean. For a fitted model with varying means, a more relevant assessment compares observed variation with the model's expected variation at those means. In a GLMM, the conditional Poisson benchmark is \(\operatorname{Var}(Y_{ij}\mid b_j,\mathbf{x}_{ij})=\mu_{ij}\), where \(b_j\) denotes the random effect for cluster \(j\) and \(\mathbf{x}_{ij}\) denotes the covariates. The response distribution and its assumptions must be assessed at the level the model specifies.
Worked Example: A Descriptive Signal of Underdispersion
Suppose a hypothetical pilot records counts \(0,1,1,2,2,2,3,3\) from eight comparable observation periods. Treat the periods as independent observations with a common mean for this descriptive calculation.
Mean. The counts sum to \(14\), so \(\bar{y}=14/8=1.75\) counts per period.
Sample variance. The squared counts sum to \(32\). Using \(\sum_i(y_i-\bar{y})^2=\sum_i y_i^2-n\bar{y}^2\), the sum of squared deviations is \(32-8(1.75)^2=32-24.5=7.5\). Therefore \(s^2=7.5/(8-1)=1.0714\) counts squared.
Comparison. The variance-to-mean ratio is \((7.5/7)/1.75\approx0.6122\). The sample variance is below the sample mean, a descriptive signal of underdispersion relative to the Poisson benchmark. With only eight observations, sampling variability is substantial; this calculation does not establish that the underlying distribution is underdispersed.
Worked Example: A Descriptive Signal of Overdispersion
A hypothetical set of eight comparable observation periods has counts \(0,0,1,1,2,3,5,8\). Assume the periods are independent and have a common mean for the purpose of comparing these summaries.
Mean. The counts sum to \(20\), giving \(\bar{y}=20/8=2.5\) counts per period.
Sample variance. The squared counts sum to \(104\). The sum of squared deviations is \(104-8(2.5)^2=104-50=54\), so \(s^2=54/7=7.7143\) counts squared.
Comparison. The ratio is \((54/7)/2.5=108/35\approx3.086\). The sample variance is substantially larger than the mean, so the data show a descriptive signal of overdispersion relative to a common-mean Poisson model. The high count of \(8\) contributes strongly to the variance, so it is sensible to check the data and the model assumptions before treating the ratio as evidence for a new response family.
Why Random Effects Change the Comparison
A mixed model distinguishes variation within clusters, conditional on their random effects, from variation across clusters. The law of total variance gives a precise reason why the marginal variance can exceed the Poisson mean even when the conditional response is exactly Poisson.
Here \(B\) represents a random effect. If \(Y\mid B\sim\operatorname{Poisson}(\Lambda)\), where \(\Lambda\) is the conditional mean determined by \(B\) and any covariates, then \(\operatorname{Var}(Y\mid B)=\Lambda\) and \(E(Y\mid B)=\Lambda\). Substituting into the identity yields
Consequently, the marginal variance is at least the marginal mean, and it is strictly larger when the conditional mean varies across random effects. This is not evidence that the conditional Poisson assumption fails: it is variation explained by the random-effects structure. In a GLMM, diagnose dispersion relative to the fitted conditional model, not by assuming that the pooled, marginal counts should themselves have a Poisson variance.
Worked Example: Marginal Overdispersion From Cluster Heterogeneity
Imagine a hypothetical population with two equally common cluster types. Conditional on its type, each observation is Poisson with mean \(1\) count for the first type and mean \(4\) counts for the second type. Within either type, the conditional Poisson variance equals its conditional mean.
Marginal mean. The average conditional mean is \(E(\Lambda)=(1+4)/2=2.5\) counts.
Variation in conditional means. The variance of the two equally likely means is \(\operatorname{Var}(\Lambda)=((1-2.5)^2+(4-2.5)^2)/2=(2.25+2.25)/2=2.25\) counts squared.
Marginal variance. By the law of total variance, \(\operatorname{Var}(Y)=E(\Lambda)+\operatorname{Var}(\Lambda)=2.5+2.25=4.75\) counts squared. The marginal variance-to-mean ratio is \(4.75/2.5=1.90\). The pooled counts are overdispersed relative to a single Poisson distribution with mean \(2.5\), even though the response is Poisson within each type. A GLMM that adequately represents this cluster heterogeneity can therefore have a conditional Poisson response and a marginal variance larger than its marginal mean.
What the Direction Suggests About Family Choice
The observed direction helps identify which model assumptions merit scrutiny; it does not select a family automatically. A Poisson model is appropriate when the conditional count distribution has the specified mean structure and conditional variance equal to that mean. If the conditional variance remains greater than the mean after accounting for the mean structure and random effects, an overdispersion-capable model may be needed. If the conditional variance is less than the mean, an extra-variation-only model cannot represent the discrepancy in the needed direction.
| Model or approach | Dispersion direction it can address | Important qualification |
|---|---|---|
| Poisson | Benchmark: conditional variance equals conditional mean | Does not allow a different conditional variance-mean relationship. |
| Negative binomial, including NB1 and NB2 | Overdispersion | These families allow extra variance; they are not a general remedy for underdispersion. |
| Conway-Maxwell-Poisson | Can represent underdispersion or overdispersion under suitable parameterization | Its dispersion flexibility does not remove the need to check support and model fit. |
| Quasi-Poisson | Can inflate the variance relative to the Poisson mean | It specifies a mean-variance relationship rather than a full probability likelihood. |
Negative-binomial models address overdispersion through variance functions that exceed the Poisson variance. In common parameterizations, NB1 has a variance proportional to the mean with an added dispersion factor, while NB2 adds a term proportional to the squared mean. These alternatives differ in how extra variance changes with the mean. The Conway-Maxwell-Poisson family offers flexibility in both directions. Quasi-Poisson can adjust variance for overdispersion but does not supply a full probability distribution. These distinctions matter when the analysis requires a likelihood, predictions for counts, or a variance pattern that changes with the mean.
Underdispersion may arise when counts are constrained or unusually regular, or when the response process discourages large departures from a typical count. Overdispersion can reflect unmodeled heterogeneity, clustering not adequately represented, dependence, excess zeros, or a misspecified mean relationship. These are possible mechanisms, not diagnoses: the same variance pattern can have different causes, and a flexible distribution cannot correct a mistaken mean structure or an inappropriate independence assumption.
Diagnostics, Failure Modes, and a Practical Sequence
A variance-to-mean ratio is most interpretable for comparable counts with a common mean. In a regression or GLMM, examine whether apparent dispersion remains after the predictors, exposure, and random-effects structure have been accounted for. A practical sequence is:
State whether the Poisson benchmark is conditional on covariates and random effects. Do not compare a pooled marginal variance directly with a conditional Poisson variance.
Review fitted means, exposure definitions, influential counts, and whether observations or clusters are dependent in ways the model does not represent.
Compare observed variability with what the fitted model implies. A persistent excess suggests overdispersion; persistently less variability suggests underdispersion. A single raw ratio is only a screening description.
Consider whether the conditional variance pattern, response support, and scientific response process are compatible with the proposed family. Reassess fit after changing the model.
One failure mode is treating every large pooled variance as a reason to replace Poisson with a negative binomial. If cluster heterogeneity explains the marginal excess, the issue may instead be whether the random-effects structure is adequate. Another is interpreting a ratio slightly below one as definitive underdispersion: finite samples fluctuate, and a small number of observations can strongly affect the sample variance. Conversely, a ratio close to one does not prove that a Poisson model fits; it says little about the mean structure, dependence, or distributional shape.
Check Your Understanding
Answer each question by identifying the benchmark and the level of variation being discussed.
- A set of comparable counts has sample mean \(3\) and sample variance \(1.8\). What is the descriptive variance-to-mean ratio, and what direction does it suggest?
- Counts are conditionally Poisson given cluster random effects, but the conditional means vary across clusters. Can the marginal variance exceed the marginal mean? Explain.
- After fitting a conditional mean model, a persistent conditional variance larger than the fitted mean remains. Which direction of dispersion is indicated, and why might a negative-binomial family be considered?
- Why is a variance-to-mean ratio calculated from pooled counts with different expected means not a clean test of conditional Poisson dispersion?
Solutions. (1) The ratio is \(1.8/3=0.60\), suggesting underdispersion descriptively; it is not conclusive without considering sampling variability and the model. (2) Yes. The law of total variance gives \(\operatorname{Var}(Y)=E(\Lambda)+\operatorname{Var}(\Lambda)\), so variation in the conditional means can make the marginal variance exceed the marginal mean. (3) It indicates conditional overdispersion. A negative-binomial family may be considered because it can allow variance above the Poisson mean, provided the mean structure and random-effects assumptions have also been checked. (4) The pooled variance reflects both variation within observations at their expected means and variation among those means; it therefore does not isolate variability around the conditional Poisson model.