Zeros at the Observation and Group Levels

Why the Level of a Zero Mechanism Matters

A zero-inflated count model can assign a structural-zero mechanism to each observation. When observations are grouped, however, another possibility is that the group has a persistent zero state: every observation in that group is zero because the group is structurally unable to produce events. These mechanisms can give the same probability of zero for an individual observation but very different probabilities for an entire group to be all zero.

The distinction is about the level at which the latent mechanism operates, not simply about how many zeros appear in the data. For example, a set of sites might each have a chance of being unavailable on a particular occasion, or a site might be persistently unavailable across all occasions. Those statements imply different probability models and different dependence among observations from the same group.

Three Related but Distinct Models

First consider observation-level zero inflation. Let \(Z_{ig}=1\) indicate that observation \(i\) in group \(g\) is in the structural-zero state, with \(P(Z_{ig}=1)=\pi_{ig}\). If \(Z_{ig}=1\), then \(Y_{ig}=0\); otherwise \(Y_{ig}\) is drawn from the base count distribution. Thus

If the structural indicators are independent across observations conditional on their probabilities and the base counts are conditionally independent, this model does not add a shared zero state to the group. Each observation can be structurally zero independently.

For a persistent group-level state, let \(S_g=1\) mean the whole group is structurally zero, with \(P(S_g=1)=\rho_g\). If \(S_g=1\), then \(Y_{ig}=0\) for every \(i\) in group \(g\). If \(S_g=0\), the observations are generated from their base count distributions. Assuming conditional independence in the nonstructural state, the group likelihood contribution is

Here \(\mathbf{1}\{\cdot\}\) equals one when the condition inside the braces holds and zero otherwise. A group with any positive count cannot be in the persistent structural-zero state. An all-zero group can arise either from that state or by chance from the base count distribution.

A third model gives a group its own zero propensity without declaring that the whole group is necessarily in an all-zero state. For example, let

and use \(\pi_g\) as the structural-zero probability for each observation in group \(g\). Conditional on \(\pi_g\), observation-level structural indicators may be independent. The shared propensity makes zeros more common in some groups than others, but a group with high \(\pi_g\) can still have positive counts. This is not the same as a shared persistent state \(S_g\), which makes every count in a structural group exactly zero.

What Shared Group States Imply

A group-level state changes the joint distribution, not only the marginal probability of zero. Suppose, for simplicity, that every observation in a group has the same base probability \(f_0=P(Y_{ig}=0\mid S_g=0)\). Write \(A_{ig}=1\{Y_{ig}=0\}\). Under the persistent-zero model, \(P(A_{ig}=1)=\rho_g+(1-\rho_g)f_0\). For two distinct observations in the same group, conditional independence given \(S_g\) gives

To see this, conditional on \(S_g=1\), both zero indicators equal one. Conditional on \(S_g=0\), each has mean \(f_0\) and the indicators are independent. Therefore \(E(A_{ig}A_{jg})=\rho_g+(1-\rho_g)f_0^2\). Subtracting \(\{\,\rho_g+(1-\rho_g)f_0\,\}^2\) gives the displayed covariance. It is positive when \(0<\rho_g<1\) and \(f_0<1\). Thus an excess of all-zero groups can be a signature of shared latent states, although it does not by itself establish that this mechanism is appropriate.

The model requires substantive and statistical assumptions. The structural group state must make all counts in that group zero; groups must be independent of one another conditional on modeled covariates and any specified shared effects; and, within a nonstructural group, the base count model must describe the counts and their conditional dependence. If a group has varying exposures or covariates, its base probabilities \(f(0;\mu_{ig},\theta)\) need not be equal. The group likelihood still uses their product, but the simplified covariance expression above no longer applies unchanged.

Group-Specific Zero Propensities Are Not All-Zero States

In a group-specific propensity model, each observation retains its own structural-zero indicator. The group shares a probability \(\pi_g\), not necessarily a single state that forces all its counts to zero. For example, set \(\operatorname{logit}(\pi_g)=\gamma_0+u_g\), where \(u_g\) is a group random effect. Conditional on \(u_g\), a count has zero probability \(\pi_g+(1-\pi_g)f(0;\mu_{ig},\theta)\). The random effect allows groups to differ in their tendency to produce structural zeros. When \(u_g\) is integrated out, observations in a group can be dependent marginally, because they share the same uncertain propensity.

Model Choice, Diagnostics, and Limits

A practical formulation begins by asking what is shared. If each observation can independently be unavailable or not at risk, an observation-level mixture may be appropriate. If a group can be persistently incapable of producing counts throughout the observation period, a group-level structural state represents that claim. If groups differ in their rates of observation-level structural zeros but can still contain positive counts, a group-specific propensity is more apt. These are alternative scientific descriptions, not merely different computational options.

The base distribution matters as well. A Poisson base model may be inadequate when counts remain overdispersed after accounting for covariates and group structure; a negative binomial base distribution is one alternative. Ordinary random effects in the count mean can also create group heterogeneity and many zeros without any structural-zero state. Hurdle models, by contrast, specify a zero-versus-positive process and a positive-count distribution. Their interpretation differs from a mixture in which zeros can arise from both the structural component and the base count distribution.

Useful checks include comparing observed and model-predicted numbers of all-zero groups, groups with at least one positive count, and within-group zero patterns. Also examine whether predicted counts and zero frequencies vary appropriately with exposure and covariates. A model that predicts the overall zero fraction well but misses the frequency of all-zero groups may have the wrong dependence structure. Conversely, a large observed number of all-zero groups is not proof of a persistent structural state: omitted group heterogeneity, an inadequate base distribution, or errors in exposure and group definitions may produce similar patterns.

These models have limits. A group-level structural state can be difficult to distinguish from a group with a very small but nonzero count rate, particularly when groups have few observations. Estimating group-specific propensities can also be unstable when there are few groups or sparse counts. The latent states are not directly observed; their interpretation depends on model assumptions and cannot be read off from an observed zero. A model should therefore be justified by the sampling process and checked against patterns beyond the overall proportion of zeros.