Apparent Zero Inflation From Unmodeled Heterogeneity

Why a Poisson Model Can Predict Too Few Zeros

A dataset can contain more zero counts than a fitted Poisson model predicts even when every observation comes from an ordinary count process and no unit belongs to a distinct, permanently-zero class. The key possibility is unmodeled heterogeneity: observations may have different underlying rates because of omitted predictors, group differences, or variation not captured by the assumed count distribution. Low-rate observations produce zeros often; averaging them with high-rate observations can yield a high overall zero fraction.

The distinction matters because a large zero residual is a feature of the observed distribution, not by itself evidence of a particular mechanism. A zero-inflated model represents one explanation by adding a structural-zero component. But heterogeneity in an ordinary count model can also create excess zeros relative to a homogeneous Poisson benchmark. Model interpretation should therefore follow investigation of the rate structure and count distribution, not just a comparison of the observed zero fraction with one Poisson prediction.

Rate Heterogeneity Raises the Marginal Zero Probability

Suppose a count \(Y_i\) is Poisson conditional on its rate \(\Lambda_i\), with \(Y_i\mid\Lambda_i\sim\operatorname{Poisson}(\Lambda_i)\). Each unit has a positive rate, but rates vary across units. Conditional on a particular rate, the probability of a sampling zero is \(e^{-\Lambda_i}\). Averaging over the distribution of rates gives the marginal zero probability:

The function \(e^{-x}\) is strictly convex because its second derivative is \(e^{-x}>0\). Jensen's inequality therefore gives

The right-hand side is the zero probability of a Poisson distribution whose rate equals the average rate. Thus, holding the marginal mean fixed, variation in the Poisson rate cannot lower the zero probability below that homogeneous Poisson benchmark. If the rate is genuinely variable, the inequality is strict. This argument does not say that every heterogeneous model has more zeros than every fitted model; it compares the mixture to a Poisson distribution with the same mean.

In a GLMM, one common source of rate variation is a group random intercept. For counts \(Y_{ij}\) in group \(j\), a conditional model might be

Here \(x_{ij}\) is the covariate vector, \(\beta\) is the fixed-effect coefficient vector, and \(b_j\) is a group-specific random intercept. A standard formulation assumes the random effects are independent across groups, have mean zero and a specified distribution (often normal), and are independent of the covariates; conditional on the random effects and covariates, the counts are independent Poisson. The marginal zero probability averages \(e^{-\lambda_{ij}}\) over the random-effect distribution. Unmodeled group variation can therefore make zeros more frequent than a Poisson model that uses only a common rate or fixed effects predicts. With repeated counts within a group, a shared \(b_j\) also induces marginal dependence between those counts.

Three Ways Heterogeneity Can Be Missed

The rate mixture argument applies to several practical situations. First, omitting an important predictor can leave observations with different conditional means pooled together. Second, ignoring group-level variation can make the fitted model treat distinct groups as if they had the same rate. Third, an incorrect count distribution can impose too little variability. For example, a negative binomial count distribution can represent overdispersion relative to a Poisson distribution without introducing structural zeros. The relevant question is not simply whether zeros are numerous, but whether the model has represented the sources of variation that determine their probability.

A Gamma-Mixed Poisson Gives an Exact Illustration

A second illustration makes the role of continuous rate variation explicit. Suppose \(\Lambda\) has a gamma distribution with shape \(k\) and rate \(k/\mu\), so \(E(\Lambda)=\mu\). Conditional on \(\Lambda\), let \(Y\) be Poisson. This is a gamma-Poisson mixture, whose marginal count distribution is negative binomial. Its zero probability can be derived directly from the gamma density:

The integral uses the gamma integral \(\int_0^\infty \lambda^{k-1}e^{-a\lambda}\,d\lambda=\Gamma(k)/a^k\) for \(a>0\). The marginal variance is \(\operatorname{Var}(Y)=\mu+\mu^2/k\), which exceeds the Poisson variance \(\mu\) for finite positive \(k\). As \(k\) increases, the rate becomes less variable and the mixture approaches a Poisson model.

Diagnosing an Apparent Excess of Zeros

A useful diagnostic begins with a clearly stated reference model and its assumptions. For a fitted Poisson model, compare observed zeros with the expected number under its predicted probabilities, but also inspect the whole count distribution and the model's residual patterns. In grouped data, check whether residuals or zero rates vary systematically by group. Check whether plausible omitted covariates predict differences in exposure or event rates. If dispersion remains after relevant predictors and grouping are represented, consider whether the conditional Poisson distribution is still too restrictive.

A simple observed-versus-expected comparison is descriptive, not a stand-alone test of a structural-zero mechanism. Its uncertainty depends on fitted parameters, dependence among observations, and the model-fitting process. When a formal calibration is needed, a parametric bootstrap can be used: simulate datasets under the fitted reference model, refit the model in each simulation using the same procedure, and compare the observed zero discrepancy with the simulated discrepancies. This accounts for estimation and can preserve the grouping structure if the simulations reproduce it.

Distinguishing Explanations and Recognizing Limits

A zero-inflated Poisson model and a heterogeneous Poisson or negative binomial model can both predict more zeros than a homogeneous Poisson model. They differ in their assumptions about how the distribution arises. A zero-inflated model adds a component in which an observation is in a structural-zero state; the ordinary count component can also generate sampling zeros. A mixed count model instead allows rates to vary, with zeros arising from the conditional count distribution. A negative binomial model is one standard way to represent extra-Poisson variation in the marginal count distribution. These models may fit similar marginal zero frequencies while implying different latent explanations.

A practical comparison should consider more than the number of zeros. Assess predicted frequencies across the full count range, variation among groups, conditional relationships with covariates, and whether model assumptions are scientifically plausible. The earlier material on identifiability in zero-inflated count models is relevant: similar observed distributions can make distinct latent explanations difficult to distinguish. A model-selection result cannot, by itself, prove that a latent structural-zero state exists.

• Do not equate excess zeros with structural zeros. The zero fraction can be high because many observations have low rates.
• Do not rely on the pooled mean alone. Two populations with the same mean can have very different zero probabilities when their rates vary differently.
• Do not ignore dependence. Shared group effects can make counts within a group dependent, invalidating calculations that assume independent observations.
• Do not choose a model from one diagnostic alone. Zero counts, dispersion, covariate patterns, and group structure all provide evidence about fit.

The heterogeneity explanation also has limits. A random-effects distribution may be misspecified; a gamma mixture, a lognormal random intercept, and omitted fixed predictors do not imply identical marginal distributions. Sparse data may provide little information about the shape of that distribution. Moreover, adding complexity can make estimates unstable or difficult to interpret. The objective is not to select the most flexible model automatically, but to use a plausible structure that represents the design and the scientific sources of variation.