Why a Worked Diagnostic Analysis Needs Several Checks
A count GLMM can represent clustering through random effects, but that does not guarantee that it reproduces the observed count distribution. A model may match the overall mean while generating too few zeros, too few unusually large counts, or too little variation among observations within clusters. These patterns can have different explanations, so a useful diagnosis compares several features of the observed data with outcomes generated under the fitted model.
The analysis below uses a hypothetical study recording event counts in repeated observation periods at clinics. The goal is not to select a remedy from one statistic. It is to check whether the fitted model can reproduce the features that matter, then describe any discrepancies that remain.
Start With a Dispersion Summary, Not a Verdict
A quick descriptive check compares observed variation with a Poisson benchmark. For a set of counts with sample mean \(\bar{x}\) and sample variance \(s^2\), the ratio \(s^2/\bar{x}\) is near 1 when the observations are consistent with a simple Poisson distribution having a common mean. A ratio well above 1 is a signal to investigate, not proof of a particular model failure. In clustered data, the observations need not share a common marginal mean, and the random effects themselves induce marginal variation.
Worked Example: A Descriptive Dispersion Ratio
Suppose a hypothetical set of 10 independent observation periods has event counts \(0,0,0,1,1,2,2,4,5,5\). The sum is 20, so the sample mean is \(\bar{x}=20/10=2\). The sum of squared counts is 76. Therefore,
The sample variance is twice the sample mean, which flags more variation than the equal-mean Poisson benchmark suggests. This small calculation is descriptive: it neither accounts for covariates nor separates within-cluster variation from variation between clusters. For a GLMM, the model-based comparison must reflect the fitted means, cluster structure, and estimation uncertainty.
One common model-based summary is the Pearson dispersion ratio. With fitted conditional means \(\hat{\mu}_{ij}\), define Pearson residuals \(r_{ij}=(y_{ij}-\hat{\mu}_{ij})/\sqrt{\hat{\mu}_{ij}}\). Their squared sum is \(X^2=\sum_{ij}r_{ij}^2\); dividing by a chosen residual degrees-of-freedom value gives a dispersion summary. In a GLMM, the exact degrees-of-freedom convention and the treatment of estimated random effects matter. The ratio is therefore best used as a compact diagnostic alongside simulations, rather than interpreted using a simple reference distribution without justification.
Replicate the Features That Matter
A simulation-based check compares a statistic calculated from the observed data with the same statistic calculated from replicated data under the fitted model. For each replication, generate outcomes with the observed design and covariates, including the exposure offsets where relevant. A parametric bootstrap can also refit the model to each replicate, so the reference distribution reflects parameter estimation as well as outcome variability.
Conditional replications hold cluster effects at their fitted values and regenerate outcomes within those clusters. Unconditional replications regenerate cluster effects from the fitted random-effects distribution as well as regenerating outcomes. The first asks whether the within-cluster outcome model fits at the estimated cluster effects; the second also checks whether the fitted distribution of clusters can reproduce between-cluster variation. A large discrepancy in one type but not the other helps localize the concern, though estimates of cluster effects are uncertain and should not be treated as known constants.
Worked Example: Comparing Observed Zeros and Upper-Tail Counts
Return to a hypothetical study with 80 clinic-period observations. A Poisson random-intercept GLMM is fitted with a treatment indicator and an exposure offset. Suppose the observed data contain 31 zeros and 9 counts of at least 8. In \(B=1000\) unconditional parametric-bootstrap replications, the zero count has a central 95% predictive range of 11 to 27, and 18 replications produce at least 31 zeros. For counts at least 8, the central 95% range is 0 to 7, and 9 replications produce at least 9 such counts.
For the zero-count discrepancy, the simulation-based upper-tail proportion with the usual plus-one correction is
For the upper-tail discrepancy, the corresponding calculation is
Both observed statistics exceed their respective 95% predictive ranges, and both upper-tail proportions are small. Under this fitted model and simulation scheme, the observed data show more zeros and more large counts than the model commonly generates. These are diagnostic comparisons, not probabilities that the model is true or false, and the two statistics are related features of the same data rather than independent tests.
Why Cluster Heterogeneity Can Look Like Overdispersion
Even when the conditional distribution is Poisson, a random intercept makes the marginal distribution more variable than a Poisson distribution with one fixed mean. This follows from the law of total variance. Conditional on \(b_j\), the variance equals \(\mu_{ij}\); across possible random effects, the conditional mean also varies.
For a fixed covariate pattern, write \(\mu_{ij}=\exp(\eta_{ij}+b_j)\), where \(\eta_{ij}\) contains the fixed-effect and offset terms. Since \(b_j\sim N(0,\tau^2)\), the normal moment-generating function gives \(E\{\exp(b_j)\}=\exp(\tau^2/2)\) and \(E\{\exp(2b_j)\}=\exp(2\tau^2)\). Thus the marginal mean is \(m=\exp(\eta_{ij}+\tau^2/2)\), and
This derivation explains why a marginal variance-to-mean ratio above 1 is not, by itself, evidence that the conditional Poisson assumption fails: the fitted random intercept already creates extra marginal variation. Conversely, a random intercept does not guarantee adequate fit. It represents a particular form of cluster-level heterogeneity and may not capture remaining dependence, an unsuitable mean relationship, or an outcome distribution with a different shape.
Worked Example: Locating a Remaining Discrepancy
Suppose a second check of the same hypothetical clinic study gives a Pearson dispersion ratio of 1.78. A parametric-bootstrap reference distribution for that ratio has a central 95% range of 0.82 to 1.64, with 14 of 1000 refitted replications at least as large as 1.78. The plus-one upper-tail proportion is \((1+14)/(1000+1)=0.01499\). This supports a concern about overall outcome variability beyond what the fitted model commonly reproduces, while the separate zero and tail checks show where part of that discrepancy occurs.
Now compare adjacent-period counts within clinics. Suppose the observed within-clinic association summary is 0.29, while its unconditional-replication central 95% range is \(-0.12\) to 0.18; only 7 of 1000 replications are at least 0.29. This is evidence that the model also under-reproduces the observed within-clinic pattern. The random intercept allows observations in one clinic to share a common cluster effect, but it does not automatically model additional time-related dependence between adjacent periods.
The conclusion is therefore specific: this fitted model under-reproduces the zero count, the upper tail, the overall dispersion summary, and the chosen within-clinic association feature. The checks do not identify a unique correction. A revised model could be motivated by the study design—for example, a different conditional count distribution or a structure for within-clinic dependence—but each candidate would need to be refitted and checked against the same prespecified features. Merely adding complexity because one statistic is large would not establish a better description.
Organizing the Diagnostic Evidence
A disciplined analysis records both what was checked and what each check can say. The key comparison is always between the observed feature and its reference distribution under the fitted model, with the replication scheme made explicit.
Confirm that counts, exposure, covariates, cluster membership, and repeated-measure ordering are represented as intended. A data or offset error can mimic model misfit.
Include a dispersion summary and features relevant to the scientific question, such as the number of zeros, counts above a meaningful threshold, or a cluster-level pattern.
Simulate under the fitted model using the observed design. State whether cluster effects are held at fitted values or regenerated, and whether each replicate is refitted.
Report the observed statistic, its predictive reference range, and a simulation-based tail proportion when useful. Treat the collection of checks as diagnostic evidence, not as a set of independent hypothesis tests.
The principal alternatives are not interchangeable. A Pearson dispersion ratio is compact but compresses many possible patterns into one number. Zero and upper-tail counts target distributional features directly but depend on the chosen thresholds and can be correlated. Replicated outcomes show how several statistics behave under the full fitted model and design, but their conclusions depend on the model and simulation scheme being examined. A single omnibus summary may be convenient, yet it can conceal whether the problem lies in zeros, extreme counts, cluster variation, or dependence.
Common Mistakes and Limitations
- Calling any variance-to-mean ratio above 1 a failed Poisson model. Random effects can raise marginal variance even when the conditional count distribution is Poisson. Compare the data with the fitted GLMM, not only with a common-mean Poisson benchmark.
- Using a small simulation tail proportion as the probability the model is wrong. It is the fraction of simulated statistics at least as extreme as the observed statistic, under the specified fitted-model simulation procedure. It is not a probability that the model is false.
- Checking only the total count or mean. A model can reproduce the mean and miss the number of zeros, the upper tail, or the arrangement of counts within clusters. Use complementary summaries.
- Confusing conditional and unconditional checks. Holding fitted cluster effects fixed does not test the same feature as regenerating cluster effects. Name the replication scheme and interpret it accordingly.
- Treating predictive ranges as universal pass/fail limits. They depend on the model, design, chosen statistic, and simulation procedure. A discrepancy prompts investigation; it does not uniquely prescribe a replacement model.
Simulation checks also have finite Monte Carlo precision. With \(B\) replications, the estimated tail proportion can be unstable when only a few simulated statistics are as extreme as the observed one; increasing \(B\) improves its resolution but does not repair a misspecified simulation model. Sparse data can make particular summaries highly variable, and multiple checks can flag chance discrepancies. Report the pattern and context, not just whether a threshold was crossed.
Check Your Understanding
Use the model assumptions and diagnostic logic in this tutorial to answer the questions. Brief solutions follow.
- A count sample has mean 3 and sample variance 6. Calculate its descriptive variance-to-mean ratio. What does the result establish, and what does it not establish for clustered GLMM data?
- In 999 replications, 24 zero counts are at least as large as the observed zero count. Calculate the plus-one upper-tail proportion and interpret it.
- Why can a Poisson random-intercept GLMM have a marginal variance greater than its marginal mean even when its conditional distribution is Poisson?
- An observed upper-tail count is outside the conditional-replication range but inside the unconditional-replication range. What distinction should be considered before concluding that the count model fails?
Solutions. (1) The ratio is \(6/3=2\). This flags variation above the equal-mean Poisson benchmark but does not show whether a fitted GLMM can reproduce the data or identify the source of variation. (2) The proportion is \((1+24)/(999+1)=0.0250\). Under the specified simulation scheme, about 2.5% of replications have a zero count at least this large; this is diagnostic evidence, not the probability that the model is false. (3) The law of total variance gives \(\operatorname{Var}(Y)=E(\mu)+\operatorname{Var}(\mu)\); variation in the random intercept makes the conditional mean vary across clusters, adding the nonnegative term \(\operatorname{Var}(\mu)\). (4) Conditional replications hold cluster effects at fitted values, whereas unconditional replications regenerate them. The contrast may indicate that the discrepancy concerns the fitted cluster effects or their distribution; it does not by itself identify a unique model change.