Why Extra-Poisson Variation Needs a Cause
For a Poisson GLMM, equidispersion is a conditional claim: given the modeled random effects and covariates, the count has variance equal to its conditional mean. It does not say that counts pooled across groups, or repeated counts from the same group, must look like independent Poisson observations. Latent differences between groups can widen the pooled distribution; shared effects or temporal structure can make repeated counts dependent; and the chosen conditional count distribution can fail to describe the data even after modeled effects are taken into account.
These mechanisms matter because they call for different responses. Adding a random effect is not a general cure for every large variance-to-mean ratio. It may account for persistent differences between groups, but not necessarily for short-lived dependence between visits or a conditional distribution that has the wrong shape. A useful analysis first specifies what is conditioned on, then asks which remaining pattern is inconsistent with that specification.
Three Mechanisms to Separate
Latent heterogeneity means that the conditional rate varies across units because of an unobserved or partially observed group characteristic represented by a random effect. A Poisson distribution may fit within each group conditional on its rate, while the mixture of groups has greater variability than a single Poisson distribution with the pooled mean. When the same latent effect applies across repeated occasions, it can also induce within-unit association: occasions from a persistently high-rate unit tend to be high together.
Dependence means that repeated counts are associated beyond the independence structure specified by the model. Dependence can arise from shared events, serial processes, or a common time-specific shock. It is possible for each occasion's marginal count to have a Poisson distribution while the vector of repeated counts is not independent. Thus, checking only a separate variance-to-mean ratio for each occasion can miss this mechanism.
Conditional count-distribution inadequacy means that even after conditioning on the specified covariates and random effects, the assumed Poisson distribution does not adequately describe the count outcomes. For example, the model may fail to capture a concentration of extreme counts, or its mean structure may omit an important predictor or exposure adjustment. Such problems can resemble dispersion, but the remedy depends on the particular failure: an omitted predictor, a missing dependence structure, and an unsuitable conditional distribution are not interchangeable explanations.
Worked Examples
Worked Example: Persistent Differences Between Groups
Suppose, hypothetically, that each of many groups has repeated counts with a constant group-specific rate. Half the groups have rate \(0.5\) events per occasion and half have rate \(2.5\) events per occasion. Conditional on the group rate, occasions are independent Poisson counts. Find the pooled mean and variance for a randomly selected occasion, and the covariance between two occasions from the same group.
Let \(\lambda\) be the group-specific rate. Its mean is \((0.5+2.5)/2=1.5\), and its variance is \(\{(0.5-1.5)^2+(2.5-1.5)^2\}/2=1\). For a randomly selected occasion, the mean count is \(E(Y)=E(\lambda)=1.5\). The conditional variance is \(\lambda\), so \(E\{\operatorname{Var}(Y\mid\lambda)\}=E(\lambda)=1.5\). The variation in conditional means is \(\operatorname{Var}\{E(Y\mid\lambda)\}=\operatorname{Var}(\lambda)=1\). Therefore the pooled variance is \(1.5+1=2.5\), exceeding the pooled mean of \(1.5\).
For two different occasions in the same group, conditional independence gives zero conditional covariance. Their rates are nevertheless the same \(\lambda\), so their marginal covariance is \(\operatorname{Var}(\lambda)=1\). The pattern is consistent with persistent group heterogeneity: conditional Poisson counts can have pooled overdispersion and positive within-group association. In an applied analysis, the assumed grouping and rate structure would still need substantive justification.
Worked Example: Shared Shocks Create Dependence
Consider two hypothetical repeated counts for one unit. Let \(U\sim\operatorname{Poisson}(0.8)\) be a count of events affecting both occasions, and let \(V_1,V_2\sim\operatorname{Poisson}(1.2)\) be occasion-specific counts. Assume \(U,V_1,V_2\) are mutually independent and set \(Y_1=U+V_1\), \(Y_2=U+V_2\). Find each count's mean and variance, their covariance, and the variance of their sum.
Each count is a sum of independent Poisson variables, so \(Y_1\) and \(Y_2\) each have mean and variance \(0.8+1.2=2\). Their covariance is \(\operatorname{Cov}(U+V_1,U+V_2)=\operatorname{Var}(U)=0.8\), because the occasion-specific components are independent of one another and of \(U\). Thus the counts are positively associated even though each occasion separately has variance equal to its mean.
The variance of the total is \(\operatorname{Var}(Y_1+Y_2)=2+2+2(0.8)=5.6\). If the counts were independent with the same marginal variances, that total variance would be \(2+2=4\). The extra \(1.6\) comes from the covariance term. This example is a dependence mechanism, not evidence that either occasion's marginal Poisson variance is too small.
Worked Example: A Conditional Distribution That Fits Poorly
Suppose eight hypothetical, independent units have the same exposure and covariates, and no group effect is specified. Their observed counts are \(0,0,0,1,1,2,4,8\). Under a common-rate Poisson model, estimate the rate and calculate a Pearson dispersion summary. What can the result establish?
The estimated common rate is the sample mean: \(\bar{Y}=(0+0+0+1+1+2+4+8)/8=16/8=2\) counts per unit. The Pearson sum is \(\sum_i (Y_i-2)^2/2\). The squared deviations sum to \(3(4)+2(1)+0+4+36=54\), so the Pearson sum is \(54/2=27\). With one estimated rate, a commonly used rough dispersion ratio is \(27/(8-1)=3.86\), substantially above 1.
This is a warning that the fitted model may not capture the observed spread; it is not a diagnosis of its cause. In particular, eight observations provide little basis for a reliable large-sample reference calculation, and the pattern could reflect an omitted group structure, dependence, an inadequate mean specification, or a conditional distribution that does not fit. The counts and the design must be examined before choosing a model change.
Diagnostics: Match the Check to the Mechanism
A useful diagnostic plan respects the repeated-measures structure. First, examine whether the model's conditional mean is plausible across relevant predictors, exposures, and groups. A wrong mean can create systematic residual patterns that look like a variance problem. Second, compare observed within-group or within-unit patterns with those expected under the fitted dependence structure. Third, inspect conditional residuals and the frequency of unusually large counts after accounting for modeled effects. A single overall dispersion statistic compresses all these patterns into one number and cannot distinguish their causes.
For repeated counts, plots by group or occasion can reveal whether high counts persist within particular units, cluster at particular times, or occur as isolated extremes. Persistent differences support investigating latent heterogeneity. Similar marginal spreads but stronger-than-expected similarity between adjacent occasions point toward dependence. Excessive conditional tails or systematic residual patterns after the mean and random-effects structure are considered suggest conditional-distribution or mean-model inadequacy. These clues are not proofs: finite samples can make all three patterns look noisy, and more than one mechanism can operate at once.
Before interpreting diagnostics, check the assumptions that make them meaningful. Counts must refer to the intended observational units and occasions; exposure should be comparable or appropriately represented; the fitted mean and grouping structure should reflect the design; and any conditional independence claim must be stated relative to the modeled effects. If occasions share units, treating all rows as independent without justification can understate uncertainty, even if the marginal count variance seems reasonable.
Choosing a Modeling Response
| Observed concern | Mechanism to investigate | First modeling question |
|---|---|---|
| Persistent differences in rates across groups | Latent heterogeneity | Does the design support a group-level random effect, and is its level correctly identified? |
| Repeated counts move together beyond modeled group effects | Residual dependence | Is there a shared event, serial pattern, or other dependence structure missing from the model? |
| Conditional residuals retain unusual spread or tail patterns | Conditional distribution or mean misspecification | Are important predictors, exposures, or features of the count distribution inadequately represented? |
These responses are alternatives to investigate, not automatic substitutions. A random effect represents variation associated with a defined grouping level; it should not be added merely because a dispersion statistic is large. A dependence structure targets association across observations, whereas changing the conditional count distribution targets the remaining conditional spread or shape. Mean-model revision is also essential when residual patterns track a predictor or exposure. Standard GLM methods provide useful background on count means and dispersion, but the key GLMM question is what remains after conditioning on the specified random effects.
A principal limitation is that these sources may be hard to distinguish with few groups, few repeats, or little variation in exposure. Different models can yield similar marginal distributions, and a flexible model may fit observed counts without identifying the process that generated them. Conclusions should therefore describe the model-based evidence and uncertainty, not claim that one mechanism has been proven from overdispersion alone.
Common Mistake
- Wrong reasoning: “The observed variance exceeds the mean, so the Poisson distribution is wrong and I should replace it.”
- Correction: The comparison may be marginal, while the model's Poisson assumption is conditional. First check whether latent heterogeneity, repeated-count dependence, mean misspecification, or exposure differences explain the pattern. Only then assess whether the conditional count distribution itself remains inadequate.
- Another warning: A fitted random intercept does not automatically remove all dependence. It captures the particular persistent shared variation represented by that effect; additional dependence may remain.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For a Poisson GLMM, what equality defines conditional equidispersion, and why does it not guarantee marginal equidispersion?
- Two repeated counts each have marginal variance 3, but their covariance is 0.6. What is the variance of their sum?
- A large dispersion ratio is found after fitting a model. Name two distinct mechanisms that could produce it and one diagnostic question for each.
- Why is a large variance-to-mean ratio alone insufficient evidence that the conditional Poisson distribution is inadequate?
Solutions. (1) Conditional equidispersion is \(\operatorname{Var}(Y\mid b)=E(Y\mid b)=\mu\). Mixing over effects that change \(\mu\) can add variability in the conditional means, so the marginal variance need not equal the marginal mean. (2) \(\operatorname{Var}(Y_1+Y_2)=3+3+2(0.6)=7.2\). (3) Persistent latent heterogeneity can be investigated by asking whether rates differ systematically across groups; residual dependence can be investigated by asking whether repeated counts remain associated after accounting for modeled effects. (4) The ratio may be marginal or may reflect dependence, omitted structure, or mean misspecification. It does not by itself test the conditional distribution after all relevant structure has been accounted for.