Detecting Extra-Poisson Variation With Predictive Checks

Why Compare Observed and Replicated Variability?

A Poisson GLMM allows counts from the same cluster to be associated through shared random effects. That dependence can create substantial marginal variation across observations, even when the model assumes a Poisson distribution conditional on the random effects. The diagnostic question is therefore not simply whether the raw counts vary a great deal. It is whether the fitted model, including its random-effects structure, can generate variability like the variability seen in the data.

A predictive check addresses this by simulating count data under the fitted model and comparing selected summaries of those simulated data with the same summaries calculated from the observations. Here the focus is variation at two scales: individual observations and clusters. An unusually large observed variability summary can signal extra-Poisson variation not captured by the fitted GLMM, but it can also indicate a poor mean model, an incorrect random-effects structure, or an influential observation.

A predictive check requires a correctly specified simulation target: hold the observed covariates, offsets, and cluster sizes fixed, then generate counts under the fitted model, using the model’s random-effects distribution when assessing variation across clusters. For each simulated data set, calculate the same diagnostic summary as for the observations. This comparison assumes that the conditional count distribution, fixed-effects mean structure, random-effects distribution, and dependence structure are reasonable representations of the data-generating process. In practice, the parameters are estimated, so the resulting comparison is a diagnostic reference rather than an exact, parameter-free test.

Summaries for Observation-Level and Cluster-Level Variation

One useful observation-level summary is the Pearson-type discrepancy $$ T_{\mathrm{obs}}=\sum_{j}\sum_i \frac{\{y_{ij}-\widehat{\mu}_{ij}\}^2}{\widehat{\mu}_{ij}}, $$ where \(\widehat{\mu}_{ij}\) is the fitted mean on the scale appropriate to the check. Large values indicate that squared deviations are large relative to the fitted means. For known Poisson means, each term has expectation one: since \(E(Y_{ij}-\mu_{ij})^2=\operatorname{Var}(Y_{ij})=\mu_{ij}\), it follows that \(E[(Y_{ij}-\mu_{ij})^2/\mu_{ij}]=1\). This gives the statistic an intuitive scale, but it does not mean that the statistic has a simple reference distribution in a fitted GLMM. Parameter estimation, sparse counts, and clustering all affect its distribution. The model-replicated distribution supplies a more relevant comparison.

A cluster-level summary can reveal whether clusters are more variable than the fitted model predicts. For example, let \(Y_j=\sum_i y_{ij}\) be the observed total in cluster \(j\), and let \(\widehat{m}_j\) be the model’s expected cluster total for the chosen prediction target. One simple summary is the sample variance of the centered totals \(Y_j-\widehat{m}_j\). Its units are squared counts. Calculate this same variance for each replicated data set; do not compare it with a generic cutoff that ignores the fitted model and cluster sizes.

The observed statistic is interpreted relative to the simulated values. If it lies near the upper end of the replicated distribution, the observed data are more variable than the fitted model commonly generates for that summary. A lower-tail discrepancy can instead indicate less variability than predicted. Neither result identifies a unique cause: a wrong mean structure can produce large residuals, and an inappropriate random-effects distribution can produce cluster-level mismatches.

Checking Variability Across Clusters

An observation-level summary can identify discrepancies within the collection of counts, but an across-cluster summary asks a distinct question: are cluster totals more dispersed than the fitted GLMM predicts? To make the comparison meaningful, the replicated data must reproduce the fitted model’s cluster-level variation as well as its conditional count variation. In particular, cluster sizes and predictors should match the observed design. A check based only on fitted cluster effects may not address the variability expected across new clusters; the simulation target must match the question being asked.

Interpreting the Check and Its Limitations

Predictive checks complement residual plots and dispersion summaries rather than replacing them. A Pearson dispersion statistic compresses discrepancies into one number, while a predictive check compares that number—or another chosen summary—with variation expected under the fitted GLMM. Residual plots can help locate patterns by fitted value, covariate, or cluster; a single predictive statistic cannot show where a discrepancy occurs. Alternative count models may be considered if the discrepancy persists, but changing the model should follow investigation of mean structure, exposure, clustering, and influential observations.

Useful diagnostics include examining the simulated distribution of the chosen summary, identifying which observations or clusters dominate it, and checking whether the conclusion changes under a scientifically reasonable alternative summary. Sparse data make these checks especially sensitive to the chosen statistic: a few large counts can dominate a squared-residual measure, while many zero counts may require a summary that reflects the feature of concern. With few clusters, cluster-level variability is estimated imprecisely and the simulated reference may be broad. Plug-in parameter estimates can also make the simulated distribution too narrow if estimation uncertainty is substantial.

The central practical principle is to match the statistic to the scale of the concern. Use an observation-level summary to ask whether individual counts depart excessively from fitted means; use a cluster-level summary to ask whether cluster totals are more dispersed than the fitted model predicts. Report the summary, its units, the simulation scheme, and how extreme the observed value is among replicated values. That makes the conclusion interpretable and keeps a diagnostic comparison from being mistaken for a definitive test.