Why Count Diagnostics Need More Than a Zero Count
A count model can reproduce the average count and still miss important features of the data. It may predict too few zeros, too little variability, or too few large counts. These patterns can occur together, but they do not mean the same thing: an excess of zeros is not by itself proof of zero inflation, and a large sample variance is not by itself proof that a negative-binomial model is needed.
For mixed-effects count models, the relevant comparison is between the observed data and data simulated from the fitted model, including its estimated random-effects distribution and the observation design. This matters because random effects can create marginal overdispersion and change the marginal zero frequency even when the conditional response model is Poisson.
Set the Comparison Up Correctly
For a Poisson response with mean \(\lambda\), the conditional variance equals the mean and the probability of zero is \(e^{-\lambda}\). Those are useful reference properties, but in an NLME model they usually apply conditional on the individual random effects and covariates. If \(Y_{ij}\) is an observation for individual \(i\) at occasion \(j\), and \(b_i\) denotes that individual’s random effects, a conditional Poisson model has
The marginal distribution integrates over the random effects. The law of total variance gives
Thus the marginal variance can exceed the marginal mean even when the conditional response is Poisson. Likewise, the marginal probability of zero is \(E\{\exp[-\mu_{ij}(b_i)]\}\), not generally \(\exp[-E\{\mu_{ij}(b_i)\}]\). A diagnostic must therefore be compared with predictions from the fitted NLME model, rather than with a homogeneous Poisson formula applied indiscriminately.
A reliable simulation check follows a few principles. Keep the observed covariates, exposure, and observation schedule fixed; simulate random effects and responses according to the model’s intended predictive target; and calculate the same summary on the observed and every simulated data set. For population-level checks, replicate random effects as well as responses. For checks conditional on estimated individual effects, state that target clearly. Repeated observations within an individual must not be treated as independent when constructing the predictive distribution.
Three Complementary Summaries
The zero frequency is the number of zero observations divided by the total number observed. The count variance describes overall spread, while an upper-tail summary such as the number with \(Y\geq 4\) focuses on large counts. No single summary captures the whole distribution. In particular, many zeros can coexist with a long positive tail, while a model can match the zero proportion and still underpredict large counts.
For each summary, compare the observed value with its distribution across simulated data sets. A predictive interval is a range containing a specified fraction of replicated summaries under the fitted model; it is a reference for model checking, not a confidence interval for a parameter. A value outside that range is evidence of a mismatch worth investigating, not a standalone formal test. If many diagnostics or thresholds are examined, some unusual comparisons can arise by chance.
Worked Example: A Homogeneous Poisson Reference
Suppose a hypothetical sample has 100 independent count observations. The frequency table is: 40 zeros, 30 ones, 15 twos, 8 threes, 4 fours, 2 fives, and 1 six. First summarize the data and then compare them with a homogeneous Poisson reference fitted by its sample mean.
The frequencies sum to \(40+30+15+8+4+2+1=100\). The total count is \(30+2(15)+3(8)+4(4)+5(2)+6(1)=116\), so the fitted mean is \(\bar{x}=116/100=1.160\). The sum of squared counts is \(30+4(15)+9(8)+16(4)+25(2)+36(1)=312\). Therefore, the sample variance is
The observed zero proportion is \(40/100=0.400\), and there are \(4+2+1=7\) counts at least four. Under a homogeneous Poisson model with \(\lambda=1.160\), the expected zero proportion is \(e^{-1.160}\approx0.3135\), or about 31.35 zeros in 100 observations. The Poisson upper-tail probability is \(P(Y\geq4)=1-\sum_{k=0}^{3}e^{-1.160}1.160^k/k!\approx0.0304\), corresponding to about 3.04 counts at least four.
The sample variance exceeds the fitted mean, and the observed zero and upper-tail frequencies both exceed their Poisson reference expectations. These are descriptive discrepancies, not proof of a particular alternative. This calculation also assumes independent observations from one common Poisson distribution; a mixed-effects model must instead be checked against simulations that incorporate its random effects and design.
Use Simulations to Assess the Size of a Discrepancy
An expected count alone does not show how much variation is ordinary under the fitted model. Simulated data sets provide that reference distribution. For each simulated replicate, record the zero count, variance, and upper-tail count—or whichever summaries address the scientific concern. A plot of observed statistics against the simulated distribution can make the discrepancy and its uncertainty visible.
Worked Example: Is the Zero Count Unusually High?
In a hypothetical analysis, a fitted count NLME model is used to simulate 200 replicated data sets, each with the same 120 observation units and design as the data. The observed data contain 54 zeros. Across simulations, the zero count has mean 36.4, standard deviation 5.0, and a central 95% range from 27 to 46. One of the 200 replicates has at least 54 zeros.
The observed count is \(54-36.4=17.6\) above the simulated mean, or \(17.6/5.0=3.52\) simulated standard deviations above it. It also exceeds the stated 95% simulation range. The empirical fraction of replicates with at least 54 zeros is \(1/200=0.005\). This tail fraction is coarse because only 200 simulations were run; it is a simulation-based measure of model discrepancy, not the probability that the model is true or a conventional null-hypothesis p-value.
The zero frequency is poorly reproduced under this predictive check, so the model warrants investigation. The result does not establish zero inflation: unmodeled heterogeneity, a misspecified mean structure, dependence, or a different count distribution may also contribute. Inspecting variance and upper-tail summaries, and checking the model’s predictive target, helps determine whether the problem is isolated to zeros or more broadly distributed.
Read the Pattern Across Zeros, Variance, and Tails
The following patterns are clues rather than unique diagnoses. All comparisons should be made with the fitted models’ simulated distributions. A negative-binomial model can represent extra conditional variation relative to a Poisson model; zero-inflated and hurdle models allow additional structure for zero versus positive outcomes. The earlier material on Zero-Inflated Count Models and Hurdle Models for Excess-Zero Counts develops those model forms. Here the question is what their predictive checks reveal.
| Observed pattern relative to simulations | Possible interpretation | What to check next |
|---|---|---|
| Zeros, variance, and upper tail all exceed Poisson predictions | Extra heterogeneity or a more dispersed count distribution may be missing | Whether random effects and covariates adequately capture heterogeneity; compare dispersion-model simulations |
| Zeros are high, but positive-count variance and upper tail are reproduced | A separate zero process may be plausible | Whether zero-inflated or hurdle simulations reproduce both zero frequency and positive counts |
| Zeros are reproduced, but large counts are underpredicted | Tail behavior or mean structure may be inadequate despite a reasonable zero fit | Upper-tail calibration, influential observations, exposure, and covariate structure |
| Observed summaries fall within predictive ranges | No obvious discrepancy for those summaries | Other distributional features, covariate patterns, and within-individual dependence |
A zero-inflated model combines a probability of an extra-zero state with a count distribution that can itself produce zeros. A hurdle model separates the probability of zero from the distribution of positive counts, using a zero-truncated count distribution for the latter. Consequently, matching the observed number of zeros does not distinguish these approaches. Their simulated positive-count distributions and upper tails also need to agree with the data.
Worked Example: A Zero Match Does Not Guarantee a Tail Match
Consider a hypothetical data set of 100 observations with 45 zeros, sample variance \(2.00\), and 12 counts at least four. Three fitted models are checked by 500 simulations each. Suppose the observed zero count lies inside each model’s central 95% simulated range. For the Poisson model, the simulated median number at least four is 5 (range 1–10); for the negative-binomial model it is 10 (range 5–17); and for a hurdle model it is 11 (range 5–18). These are hypothetical simulation summaries, not universal reference values.
The observed upper-tail count, 12, is above the Poisson model’s stated range but lies inside the ranges for the other two models. The Poisson model’s zero frequency may look adequate, yet its upper tail is not reproduced in these simulations. The negative-binomial and hurdle models are not thereby proven correct: their variance, positive-count distribution, covariate-specific predictions, and dependence structure still require checking. Because all three zero-count checks passed, the zero summary alone cannot choose between the alternatives.
Diagnostics, Failure Modes, and Limits
A useful diagnostic sequence makes the simulated comparison interpretable. First verify that observed and replicated summaries use the same units and denominator. Then check whether simulations include the random effects and exposure structure appropriate to the predictive target. Finally, examine summaries jointly: an excess-zero finding is more informative when considered alongside variance and upper-tail discrepancies.
Choose a zero proportion, variance, and tail threshold that address the model question. State the threshold and denominator explicitly.
Simulate from the fitted model while preserving the covariates, exposure, and repeated-observation structure relevant to the check.
Locate each observed summary in its simulated distribution and inspect plots for systematic or subgroup-specific patterns.
Consider mean misspecification, random-effect heterogeneity, dispersion, zero mechanism, and dependence; after any model change, repeat the predictive checks.
Several mistakes can make a diagnostic misleading. Comparing observed zeros with \(e^{-\bar{x}}\) in a mixed-effects setting ignores variation in individual means. Comparing an observed sample variance directly with a mean ignores sampling variation and the model’s marginal random-effects distribution. Simulating each repeated observation independently erases within-individual dependence and can make predictive ranges too narrow. Finally, trying many tail thresholds and reporting only the most unusual one exaggerates the apparent strength of evidence.
Simulation checks have limits. They depend on the fitted model and parameter estimates, and finite simulation runs give imprecise tail estimates. A model can pass the selected checks while failing in an unexamined feature. Conversely, a discrepancy can reflect a few unusual observations or a poorly specified mean or exposure relationship, not necessarily the response distribution. These checks guide model criticism; they do not replace scientific reasoning or identify a unique data-generating mechanism.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Why can a conditional Poisson NLME model have a marginal variance greater than its marginal mean?
- A data set has more zeros than the fitted model’s central 95% predictive range, but its upper-tail count is well reproduced. What does this establish, and what does it not establish?
- Why is comparing the observed zero proportion only with \(e^{-\bar{x}}\) potentially misleading in a model with random effects?
- What is the diagnostic value of checking counts above a threshold as well as the zero frequency?
- Why must simulations preserve repeated observations within individuals when the fitted model includes individual random effects?
Solutions. (1) The marginal variance equals the mean conditional variance plus the variance of the conditional mean; variation in individual means adds a nonnegative component. (2) It establishes a mismatch in zero frequency relative to that model’s simulations, but does not uniquely establish zero inflation or identify its cause. (3) Random effects make the marginal zero probability an average of conditional zero probabilities, \(E\{\exp[-\mu(b_i)]\}\), which need not equal \(\exp[-E\{\mu(b_i)\}]\). (4) The upper tail can reveal dispersion or tail misfit even when the model reproduces zeros. (5) Independent simulation of repeated records would remove the modeled within-individual dependence and distort the predictive distribution of the summaries.