Why Check the Zeros Directly?
A count GLMM can describe the broad outcome distribution reasonably well and still predict too few or too many zero counts. This can matter when zeros are common, when conclusions depend on the frequency of no events, or when the model is used to predict counts for individual clusters. The useful question is not whether a zero seems surprising in isolation, but whether the fitted model tends to generate as many zeros, and a similar distribution of zeros across clusters, as the data show.
The check is a simulation-based predictive comparison. First define a zero-count summary of the observed outcomes. Then generate replicated outcomes under the fitted model, calculate the same summary in each replicate, and compare the observed value with the resulting reference distribution. The comparison is diagnostic: it can reveal a mismatch, but by itself it does not identify its cause.
The Model-Implied Reference
For clarity, consider a Poisson-log-link count GLMM. Conditional on the random effects \(b_j\), observations are modeled as independent Poisson variables with means \(\mu_{ij}\), where \(\log(\mu_{ij})=x_{ij}^{\mathsf T}\beta+z_{ij}^{\mathsf T}b_j+\log(e_{ij})\) if an exposure offset \(e_{ij}\) is used. The fixed-effects design vector is \(x_{ij}\), the random-effects design vector is \(z_{ij}\), and the random effects follow the distribution specified by the model, commonly \(b_j\sim N(0,\Sigma)\). The Poisson response has conditional variance equal to its conditional mean. The random-effects distribution, conditional independence, link, exposure specification, and other distributional assumptions must be suitable for the data and the intended predictions.
For a Poisson count with mean \(\mu_{ij}\), the conditional probability of zero is \(\exp(-\mu_{ij})\). Therefore, if the fitted conditional means are treated as fixed, the model-implied expected total number of zeros is the sum of those probabilities. This expected count is a useful check, but it is not a complete reference distribution: replication also shows the variability of the zero count and, when simulated appropriately, the dependence induced by shared cluster effects.
Here is a short justification for why the simulation distribution matters. Let \(Z_{ij}=I(Y_{ij}=0)\). Conditional on the random effects, the \(Z_{ij}\) are independent Bernoulli variables under the stated Poisson model, with success probabilities \(p_{ij}=\exp(-\mu_{ij})\). Thus the conditional variance of their sum is \(\sum p_{ij}(1-p_{ij})\). If random effects are also varied across replications, the total variance includes variation in the conditional expected zero count:
The second term reflects changes in zero probabilities as cluster effects vary. Consequently, treating all observations as independent Bernoulli trials with one pooled zero probability can give the wrong reference spread. Replications from the fitted GLMM preserve the model’s specified cluster structure and provide a direct comparison on the scale of the chosen statistic.
Worked Example: Expected Zeros From Fitted Means
Suppose a hypothetical fitted model gives four conditional Poisson means, for four observations in different clusters, of 0.2, 0.4, 1, and 2. The observed outcomes are all zero. First calculate the model-implied expected number of zeros at these fitted means.
The four zero probabilities are \(\exp(-0.2)\approx0.8187\), \(\exp(-0.4)\approx0.6703\), \(\exp(-1)\approx0.3679\), and \(\exp(-2)\approx0.1353\). Summing the unrounded probabilities gives:
Thus about two zeros are expected on average, compared with four observed. A more direct tail comparison is possible here because the observations are conditionally independent Poisson counts at fixed means. The probability that all four are zero is:
This calculation suggests that four zeros would be unusual under these fixed means. In practice, however, the fitted model has estimated parameters and may include shared random effects. Simulating the full fitted model gives a more appropriate reference when those features are relevant; this hand calculation is a check on the basic zero-probability arithmetic, not a substitute for that simulation.
How to Compare Observed and Replicated Counts
For each replicated data set, calculate \(T_0\) and every \(T_{0j}\) in exactly the same way as for the observed data. Keep the observation design aligned: retain the observed covariates, exposures, cluster membership, and number of observations unless the prediction target explicitly calls for a different design. The simulation must reproduce the parts of the model whose predictive behavior is being checked.
Use the overall zero count or proportion, and define cluster-specific counts or proportions if the distribution across clusters is important.
For each replication, draw counts under the fitted model. Decide whether the target is conditional on estimated cluster effects or includes new draws of the random effects; state that choice because it changes the predictive reference.
Apply the same zero-count definitions to every replicate, preserving the original cluster sizes and covariates.
Report the observed value alongside the simulated distribution, such as its mean, interval, and an appropriate tail proportion. Look for both an overall shift and clusters that repeatedly fall in an extreme part of their own reference distributions.
A plug-in simulation holds fitted parameters fixed and is straightforward, but it omits uncertainty in their estimates. A predictive procedure that also accounts for parameter uncertainty gives a wider or otherwise changed reference when that uncertainty is material. Conditional replication holds estimated cluster effects fixed; unconditional replication draws cluster effects from the fitted random-effects distribution. The former asks how the fitted conditional means reproduce outcomes within these clusters; the latter also asks how outcomes vary across clusters under the model. Match the choice to the predictive question rather than treating one version as universally correct.
Worked Example: Overall Zero Count in Replications
In a hypothetical study, a fitted count GLMM is used for 30 observations. The observed data contain 14 zeros. A simulation procedure matched to the target generates 1,000 replicated data sets, each with the same covariates, exposures, and observation count. The replicated zero counts have mean 10.8, a central 95% simulation interval from 6 to 16, and 87 values at least as large as the observed count.
The estimated upper-tail proportion is \(87/1000=0.087\). The Monte Carlo standard error of this estimated proportion is approximately:
The observed value 14 lies inside the 95% simulation interval, and the estimated upper-tail proportion is about 0.087. These summaries do not indicate a pronounced overall excess of zeros in this example, although they do not establish that the model is correct. The Monte Carlo standard error describes simulation noise in the estimated tail proportion; it is not the standard error of the observed zero count.
The central interval and the tail proportion answer related but different descriptive questions. The interval locates the observed statistic relative to the middle of the simulated distribution. The tail proportion counts how often the simulated statistic reaches or exceeds the observed one. With a discrete count, ties are common, so define the tail convention explicitly and use it consistently.
Worked Example: Locating Zero Excess Across Clusters
Consider four hypothetical clusters, each with 10 observations. The observed cluster zero counts are 8, 7, 5, and 3, for a total of \(8+7+5+3=23\). In 2,000 fitted-model replications, the mean cluster counts are 6.0, 5.1, 4.0, and 2.4; their sum is 17.5, the mean replicated total. The simulated central 95% interval for the total is 12 to 23, and 62 replications have a total of at least 23. Cluster-specific upper-tail proportions for reaching or exceeding the observed counts are 0.20, 0.12, 0.35, and 0.40, respectively.
For the total, the estimated upper-tail proportion is \(62/2000=0.031\). The observed total is at the upper endpoint of the stated central interval, and the tail proportion indicates that totals this large or larger were uncommon in these replications. Yet the cluster-specific summaries do not show one cluster with an exceptionally large upper-tail discrepancy: all four cluster upper-tail proportions are at least 0.12. The overall signal may reflect a modest increase spread across clusters rather than a single cluster driving the total.
This distinction is why the total alone is not enough when cluster behavior matters. Conversely, testing many clusters separately can produce apparently unusual values just by chance. Treat the cluster display as a pattern check, account for the number of clusters when making formal claims, and avoid interpreting one small tail proportion as a definitive finding.
Interpretation, Alternatives, and Failure Modes
A simulation tail proportion is not automatically a conventional hypothesis-test p-value. It summarizes how often the chosen discrepancy is at least as extreme in a specified direction under the replication scheme. Its value depends on the statistic, the direction, the parameter and random-effect treatment, and the simulated design. A small value is evidence of a predictive mismatch for that summary under those choices; it is not the probability that the model is true or that a particular alternative mechanism is present.
The analytic expected zero count is a useful alternative when fitted means are fixed and the zero probability is simple to compute. It can quickly flag a difference in averages. It does not show the full predictive spread, and a calculation that ignores shared random effects can misstate that spread. Residual summaries can also reveal observations or clusters with poor fit, but a direct replicated zero-count check has the advantage of assessing the exact feature of interest on its natural count scale. None of these summaries replaces checking other aspects of fit.
Several practical failures can make the comparison misleading:
- Using the wrong replication target. Holding cluster effects fixed and redrawing them answer different questions. State which is used and why.
- Breaking the design. Changing exposure, covariates, or cluster sizes across replicates can make the reference incomparable with the observed statistic.
- Ignoring cluster dependence. A pooled calculation based on independent, identical zero indicators may not represent the fitted GLMM’s distribution.
- Using too few replications. Tail proportions in the far tail are imprecise with a small simulation count. Increase the number of replications when a conclusion depends on a rare simulated event.
- Overreading an overall match. A plausible total can conceal a cluster pattern that the model misses; inspect cluster-specific counts or proportions as well.
- Overreading one cluster. With many clusters, at least one can look unusual by chance. Look for coherent patterns and consider the multiplicity of the inspection.
- Confounding the symptom with its cause. Too many zeros relative to replications identifies a predictive discrepancy for zero frequency. This check alone does not determine whether the issue is the mean structure, the random-effects specification, another form of variation, or a distinct zero-generating process.
The check is limited to the selected discrepancy and fitted prediction scheme. A model can reproduce the total zero count but fail to predict positive counts, upper tails, or dependence within clusters. A satisfactory zero check is therefore one piece of a broader diagnostic assessment, not a certificate of overall adequacy.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- For a Poisson observation with fitted mean 0.7, what is its fitted probability of being zero?
- In 500 replications, 35 replicated overall zero counts are at least as large as the observed count. What is the estimated upper-tail proportion?
- Why can a model match the total number of zeros but still show a cluster-level discrepancy?
- What does a small upper-tail proportion establish, and what does it not establish?
- When would conditional and unconditional replications answer different predictive questions?
Solutions. (1) The probability is \(\exp(-0.7)\approx0.4966\). (2) The estimated proportion is \(35/500=0.070\), or 7.0%. (3) The same total can arise from different allocations of zeros across clusters; one cluster may have many more zeros than its own replications typically produce, while another has fewer. (4) It indicates that the observed discrepancy was uncommon under the specified replication scheme and statistic. It does not give the probability that the model is true, identify the cause of misfit, or establish a conventional test result without a testing framework. (5) Conditional replications hold estimated cluster effects fixed and assess outcomes around those effects; unconditional replications also draw cluster effects and assess predictions that include between-cluster variation.