Specifying an Extension for Sparse Grouped Counts

From a sparse count table to a model

A grouped count study can have many zeros, substantial variation between groups, and a small number of large counts. Those patterns do not by themselves identify a data-generating mechanism. A negative binomial GLMM, a zero-inflated GLMM, and a hurdle GLMM can all produce more zeros and more variable counts than a basic Poisson model, but they do so in different ways. The purpose of specifying candidates is to make those differences explicit before fitting or interpreting them.

Use \(j=1,\ldots,J\) for groups, such as clinics, and \(i=1,\ldots,n_j\) for count observations within group \(j\). Let \(Y_{ij}\) be the count in observation \(i\) of group \(j\), \(E_{ij}>0\) its exposure, and \(\mathbf{x}_{ij}\) its count-component covariates. The examples below are hypothetical. A group random intercept represents residual group-to-group variation after accounting for the recorded predictors; it does not, by itself, explain why any particular observation is zero.

Definition: A grouped-count GLMM specifies a conditional distribution for \(Y_{ij}\) given covariates and latent group effects, and a distribution for those effects. In the models below, observations are conditionally independent within and across groups given the stated latent effects. Integrating over shared group effects generally makes observations in the same group marginally dependent.

An exposure offset is appropriate when the expected count is proportional to exposure at a fixed rate. In that case its coefficient is fixed at one: it enters a log-mean as \(\log(E_{ij})\). If exposure does not have this proportional relationship, an offset is not automatically justified.

Three candidate specifications

The negative binomial candidate uses one count-generating process. For precision, use the NB2 parameterization: conditional on its mean \(\mu\) and dispersion \(\kappa>0\), a negative binomial count has variance \(\mu+\kappa\mu^2\). Its probability of zero is \((1+\kappa\mu)^{-1/\kappa}\). The GLMM specification is

Y_{ij}\mid b_j \sim \operatorname{NB2}(\mu_{ij},\kappa), \qquad \log(\mu_{ij})=\log(E_{ij})+\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+b_j, \qquad b_j\sim N(0,\sigma_b^2).

Here the count distribution, including its zeros, is negative binomial conditional on \(b_j\). The random intercept shifts the log expected count for every observation in a group. The model assumes the specified negative binomial variation is adequate after conditioning on predictors and group effect; it does not posit a separate structural-zero state.

A zero-inflated Poisson model adds a latent state. Let \(S_{ij}=1\) denote the structural-zero state, and let \(S_{ij}=0\) indicate that the observation is generated by a Poisson count process. One possible grouped specification is

\begin{aligned} P(S_{ij}=1\mid u_j)&=\pi_{ij}, & \operatorname{logit}(\pi_{ij})&=\mathbf{z}_{ij}^{\mathsf T}\boldsymbol{\gamma}+u_j,\\ Y_{ij}\mid S_{ij}=0,b_j&\sim\operatorname{Poisson}(\lambda_{ij}), & \log(\lambda_{ij})&=\log(E_{ij})+\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+b_j,\\ Y_{ij}\mid S_{ij}=1&=0, & b_j&\sim N(0,\sigma_b^2),\quad u_j\sim N(0,\sigma_u^2). \end{aligned}

For this candidate, the two group effects are assumed independent, and observations are independent conditional on them and the covariates. Other dependence assumptions are possible, but they must be specified rather than left implicit. Conditional on \(b_j,u_j\), the zero probability is \(\pi_{ij}+(1-\pi_{ij})\exp(-\lambda_{ij})\), while the conditional mean is \((1-\pi_{ij})\lambda_{ij}\). A zero can therefore arise from either latent state.

A hurdle model separates the probability of crossing from zero from the distribution of positive counts. Let \(q_{ij}\) be the probability of a positive observation. One negative binomial hurdle GLMM is

\begin{aligned} \operatorname{logit}(q_{ij})&=\mathbf{z}_{ij}^{\mathsf T}\boldsymbol{\gamma}+v_j, &v_j&\sim N(0,\sigma_v^2),\\ X_{ij}\mid b_j&\sim\operatorname{NB2}(\mu_{ij},\kappa), &\log(\mu_{ij})&=\log(E_{ij})+\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+b_j,\\ Y_{ij}\mid b_j,v_j&= \begin{cases} 0,&\text{with probability }1-q_{ij},\\ X_{ij}\mid X_{ij}>0,&\text{with probability }q_{ij}. \end{cases} \end{aligned}

Here \(X_{ij}\) is the underlying negative binomial count used to define the positive-count distribution; it is not the observed hurdle response \(Y_{ij}\). The zero probability is \(1-q_{ij}\). Given that the observation is positive, its count follows a zero-truncated negative binomial distribution. In this specification \(b_j\) and \(v_j\) are independent normal effects, and the observations are conditionally independent given the effects.

Key distinction: The zero-inflated model allows zeros from both its structural-zero state and its count distribution. A hurdle model assigns every zero to the zero component; its count distribution is truncated to positive values. The NB GLMM has no separate zero component.

Why the hurdle positive-count mean has this form

The truncation in a hurdle model changes the mean of its positive-count component. Let \(f_0=P(X_{ij}=0\mid b_j)\) for the underlying NB2 variable. Since \(X_{ij}=0\) contributes nothing to its expectation,

\begin{aligned} E(X_{ij}\mid X_{ij}>0,b_j) &=\frac{E\{X_{ij}\mathbf{1}(X_{ij}>0)\mid b_j\}} {P(X_{ij}>0\mid b_j)}\\ &=\frac{E(X_{ij}\mid b_j)}{1-f_0} =\frac{\mu_{ij}}{1-f_0}. \end{aligned}

The observed hurdle response is \(Y_{ij}\), not \(X_{ij}\). Its positive-count distribution is defined to be the distribution of \(X_{ij}\) conditional on \(X_{ij}>0\). Consequently,

E(Y_{ij}\mid Y_{ij}>0,b_j,v_j)=\frac{\mu_{ij}}{1-f_0}, \qquad E(Y_{ij}\mid b_j,v_j)=q_{ij}\frac{\mu_{ij}}{1-f_0}.

Equivalently, the second equality follows directly from the hurdle response: the positive event has probability \(q_{ij}\), and its conditional mean is \(\mu_{ij}/(1-f_0)\). This derivation keeps the underlying negative binomial variable distinct from the observed hurdle response.

Worked Example: A Negative Binomial Candidate for Sparse Claims

Suppose a hypothetical insurer records claim counts for policy-months nested in six regions. A policy-month has exposure \(E_{ij}=1\), and covariates include policy type. A fitted candidate uses \(\kappa=1.5\) and gives a conditional mean of \(\mu_{ij}=0.50\) for a reference policy-month at \(b_j=0\). The proposed NB2 conditional zero probability is

\(\displaystyle P(Y_{ij}=0\mid b_j=0)=(1+\kappa\mu_{ij})^{-1/\kappa}=(1+1.5(0.50))^{-1/1.5}=1.75^{-2/3}\approx 0.689.\)

For comparison, a Poisson count with the same mean has zero probability \(\exp(-0.50)\approx0.607\). The NB2 candidate allows greater conditional variance, \(\mu+\kappa\mu^2=0.50+1.5(0.50)^2=0.875\), and therefore permits a larger zero probability here. This calculation makes the candidate distribution concrete; it does not establish that negative binomial variation is the actual claim-generating mechanism.

For this specification to be defensible, claim counts must be nonnegative integers, the exposure offset must represent proportional time at risk, and residual group differences must be reasonably represented by the normal random intercept. Conditional independence is assumed after conditioning on that intercept and the covariates. Repeated policy-months with additional serial dependence, or many more zeros than this model predicts, would call for further investigation rather than automatic acceptance of this candidate.

Worked comparisons of zero mechanisms

Worked Example: Distinguishing Zero-Inflated and Hurdle Probabilities

Consider a hypothetical clinic-level observation with Poisson count mean \(\lambda=0.80\) and structural-zero probability \(\pi=0.35\), conditional on its group effects. The zero-inflated model gives

\(\displaystyle P(Y=0)=\pi+(1-\pi)e^{-\lambda}=0.35+0.65e^{-0.80}\approx0.642.\)

Its conditional mean is \((1-\pi)\lambda=0.65(0.80)=0.520\). Given a positive response, the count-process state must have been active, so the expected count is \(0.80/(1-e^{-0.80})\approx1.453\). This is both the active Poisson mean conditional on positivity and the mean of the full zero-inflated response conditional on positivity.

Now specify a hurdle Poisson model with the same underlying \(\lambda=0.80\), but set \(q=0.35\). Its zero probability is \(1-q=0.65\), and its positive-count mean is \(0.80/(1-e^{-0.80})\approx1.453\). Thus its unconditional mean is \(q\{0.80/(1-e^{-0.80})\}\approx0.508\). The two models have the same positive-count distribution in this setup, but different zero probabilities and unconditional means: the zero-inflated model allows ordinary Poisson zeros, whereas the hurdle model assigns all zeros to its zero component. A single observed zero cannot tell which mechanism generated it.

These values are conditional on the group effects. In a grouped GLMM, the marginal probabilities and means require averaging over the random-effects distribution; substituting a random effect of zero does not generally give that marginal average.

Worked Example: Specifying a Negative Binomial Hurdle GLMM

Suppose hypothetical hospital data contain many zero visit counts and a separate process is scientifically plausible: some hospital-months have no opportunity for the service, while other months can have one or more visits. For an illustrative hospital-month, take \(\mu=0.60\), \(\kappa=0.80\), and \(q=0.25\), conditional on group effects. The underlying NB2 zero probability is

\(\displaystyle f_0=(1+\kappa\mu)^{-1/\kappa}=(1+0.80(0.60))^{-1/0.80}=1.48^{-1.25}\approx0.613.\)

The underlying positive probability is \(1-f_0\approx0.3874\), so the hurdle response has \(P(Y=0)=1-q=0.750\). The underlying NB2 mean conditional on being positive is \(\mu/(1-f_0)=0.60/(1-f_0)\approx1.549\). Hence the observed hurdle response has \(E(Y)=q(1.549)\approx0.387\), not \(0.60\). The latter is the mean of the underlying, untruncated negative binomial variable.

The model requires a defensible positive-versus-zero process, an adequate NB2 distribution among positive counts after truncation, and an appropriate exposure and group-effect structure. The excess of zeros alone does not establish the proposed opportunity mechanism. If the count covariates also belong in the hurdle component, that is a substantive modeling choice: the two components describe different conditional quantities.

Assumptions, diagnostics, and candidate choice

The formulas specify conditional models. Their interpretations depend on assumptions about how data were generated and recorded, not just on the observed frequency of zeros. Before treating any candidate as plausible, check the following:

Observation unit and exposure: Each \(Y_{ij}\) must represent a count for a well-defined unit and period. An offset assumes proportionality between expected count and exposure.
Grouping: A random intercept assumes a particular form of residual group heterogeneity. Conditional independence is required after accounting for the chosen effects; repeated observations may require more than a single intercept if their dependence remains.
Component distributions: The NB candidate requires an adequate NB2 conditional distribution. The zero-inflated candidate requires a meaningful latent structural-zero state and an adequate count distribution for the non-structural state. The hurdle candidate requires a suitable binary positive-count process and a suitable zero-truncated count distribution.
Covariates and random effects: Each component needs its own defensible predictors and effect structure. Omitting group variation from a component that varies across groups can distort its estimated probabilities. Assuming independent component random effects is an assumption, not a consequence of writing two equations.

Useful checks compare observed and model-predicted zero proportions, positive-count frequencies, upper tails, and patterns across exposure levels and groups. For a GLMM, comparisons should account for the fact that observations in a group share latent effects. A discrepancy concentrated among positive counts suggests a different problem from a discrepancy concentrated at zero. Likewise, group-specific residual patterns may indicate that the specified random-effects structure is inadequate. These checks can reveal lack of fit, but they cannot prove a particular latent explanation.

The principal alternatives answer different questions. The NB GLMM attributes extra variation to its count distribution and group effects. The zero-inflated model distinguishes a structural-zero state from a count-generating state. The hurdle model models whether a count is positive separately from the size of a positive count. A model with more components is not automatically a better scientific description: added complexity can be weakly identified when groups are few or positive counts are rare. If data are reported only as totals over several units with differing covariates or exposures, the total may not have the same likelihood as a collection of unit-level counts; specify the aggregation process rather than silently treating a total as one ordinary observation.

Common pitfall: “There are more zeros than a Poisson model predicts, so the data must be zero-inflated.” This conclusion does not follow. Negative binomial variation, group heterogeneity, omitted predictors, or a hurdle process can also produce many zeros. The correction is to state candidate data-generating assumptions, check implications for zeros and positive counts, and use subject-matter knowledge as well as model fit to assess plausibility.

Check Your Understanding

Answer each question, then compare with the solutions below.

A proposed NB2 GLMM has \(\mu=0.40\) and \(\kappa=1\) for an observation at its group-effect value. What is its conditional zero probability?
In a zero-inflated Poisson model, explain why a zero does not identify the structural-zero state.
A hurdle model has \(q=0.20\), and its underlying count has mean \(\mu=0.50\) and zero probability \(f_0=0.60\). Find the positive-count conditional mean and the unconditional hurdle-response mean.
Why does a random intercept mean that observations in the same group are generally not marginally independent?

Solutions. (1) \(P(Y=0)=(1+\kappa\mu)^{-1/\kappa}=1.4^{-1}\approx0.714\), conditional on the stated group effect. (2) A structural-state observation is zero, but a Poisson count in the active state can also be zero; the model assigns probability to both routes. (3) The positive mean is \(\mu/(1-f_0)=0.50/0.40=1.25\). The hurdle-response mean is \(q(1.25)=0.20(1.25)=0.25\). (4) Two observations share the same latent group effect, so integrating over that shared random quantity induces dependence even when they are conditionally independent given it.