Diagnosing Excess Zeros in Count Data

Why Diagnose Zeros?

In count data, zero is often a common outcome: a person may report no events during a period, or a firm may record no incidents in a quarter. The relevant question is not simply whether there are many zeros. It is whether the number and pattern of zeros are surprising under a specified fitted count model, given the model’s predicted means and distribution.

A Poisson or negative-binomial model implies a probability of zero for each observation. Comparing those probabilities with the observed zeros is a focused diagnostic. A mismatch can flag inadequate distributional assumptions, omitted covariates, incorrect exposure measurement, or a distinct process that produces zeros. The comparison alone does not identify which explanation is right, and it does not establish that a separate zero-generating mechanism exists.

Definition: For observation \(i\), let \(Y_i\) be its observed count and let \(\hat{q}_i\) be the fitted model’s predicted probability that \(Y_i=0\). The observed zero count is \(O=\sum_{i=1}^n I(Y_i=0)\); the expected zero count under the fitted model is \(E=\sum_{i=1}^n \hat{q}_i\). The zero-count discrepancy is \(D=O-E\). A positive \(D\) means more zeros were observed than the fitted model predicts.

Predicted Zero Probabilities

For a Poisson regression with fitted conditional mean \(\hat{\mu}_i\), the fitted distribution assigns probability \(e^{-\hat{\mu}_i}\) to zero. Thus, the model’s expected zero count is the sum of these individual probabilities. It is generally not appropriate to calculate a single zero probability using the average fitted mean: the exponential function is nonlinear, so observations with different fitted means have different zero probabilities.

\hat{q}_i^{\,P}=P(Y_i=0\mid X_i;\hat{\mu}_i)=e^{-\hat{\mu}_i}, \qquad E_P=\sum_{i=1}^{n}e^{-\hat{\mu}_i}.

For the NB2 parameterization, the conditional mean is \(\mu_i\) and the conditional variance is \(\mu_i+\alpha\mu_i^2\), where \(\alpha>0\) is the dispersion parameter. Its zero probability is \((1+\alpha\mu_i)^{-1/\alpha}\). With fitted values, sum this probability observation by observation. NB1 uses a different variance structure and has a corresponding distribution-specific zero probability; do not apply the NB2 formula to an NB1 fit.

\hat{q}_i^{\,NB2}=(1+\hat{\alpha}\hat{\mu}_i)^{-1/\hat{\alpha}}, \qquad E_{NB2}=\sum_{i=1}^{n}(1+\hat{\alpha}\hat{\mu}_i)^{-1/\hat{\alpha}}.

These formulas require a distributional model, not just a model for the conditional mean. Poisson quasi-maximum likelihood can estimate a conditional mean consistently under appropriate conditions even when the Poisson distribution is wrong. That mean-model result alone does not make the Poisson zero probability correct. A zero diagnostic based on the Poisson distribution is therefore a diagnostic of the distributional assumption, not merely of the fitted mean.

Worked Example: Poisson Predictions for a Common Fitted Mean

Suppose a hypothetical Poisson model assigns each of \(100\) observations a fitted mean of \(0.5\) events. The data contain \(72\) observed zeros. Calculate the expected zero count and the discrepancy.

For each observation, the predicted probability of zero is \(e^{-0.5}\approx0.606531\). Therefore,

E_P=\sum_{i=1}^{100}e^{-0.5} =100(0.606531) \approx60.6531.

The observed count is \(O=72\), so \(D=72-60.6531=11.3469\), or about \(11.35\) more zeros than predicted. This is a descriptive discrepancy, not yet a calibrated significance test. In particular, a fitted mean and dispersion estimated from these same observations introduce uncertainty that a simple fixed-probability calculation does not include.

Residuals and a Plug-In Standardization

For each observation, define the zero indicator \(Z_i=I(Y_i=0)\). The zero residual is \(r_i=Z_i-\hat{q}_i\): it is \(1-\hat{q}_i\) for an observed zero and \(-\hat{q}_i\) otherwise. Summing the residuals gives the discrepancy, since \(\sum_i r_i=O-E\). Looking at individual or grouped residuals can reveal where the mismatch occurs, rather than only whether the total is positive.

If the probabilities \(q_i\) were known and observations were independent under the model, then \(Z_i\) would be Bernoulli with probability \(q_i\). A Bernoulli variable has variance \(q_i(1-q_i)\). Independence makes the variance of the sum equal to the sum of the variances. This motivates a plug-in standardization using fitted probabilities:

\operatorname{Var}(O\mid q_1,\ldots,q_n) =\sum_{i=1}^{n}q_i(1-q_i), \qquad Z_{\text{plug-in}} =\frac{O-\sum_{i=1}^{n}\hat{q}_i} {\sqrt{\sum_{i=1}^{n}\hat{q}_i(1-\hat{q}_i)}}.

This standardization is a useful scale for a discrepancy, but it is not automatically a valid hypothesis test. Its denominator treats fitted probabilities as fixed. Estimating regression coefficients and dispersion from the same data can change the variability of the zero count; dependence or clustering can change it as well. A plug-in value should therefore be interpreted cautiously, especially near a decision threshold.

Justification: If \(Z_i=I(Y_i=0)\), then \(E(Z_i)=q_i\) and \(\operatorname{Var}(Z_i)=q_i(1-q_i)\). For independent observations, \(\operatorname{Var}(\sum_i Z_i)=\sum_i\operatorname{Var}(Z_i)\). Replacing the unknown \(q_i\) values by fitted \(\hat{q}_i\) gives the plug-in variance in the standardization above. Because the fitted probabilities are estimated, this substitution does not by itself account for parameter-estimation uncertainty.

Worked Example: NB2 Predictions Across Different Fitted Means

Consider a hypothetical NB2 fit with \(\hat{\alpha}=0.8\). There are \(80\) observations with fitted mean \(0.5\), \(80\) with fitted mean \(1.5\), and \(40\) with fitted mean \(3\). Suppose the observed total is \(96\) zeros. Calculate the expected count and the plug-in standardized discrepancy.

For the three fitted means, the predicted zero probabilities are \((1+0.8(0.5))^{-1/0.8}=1.4^{-1.25}\approx0.656659\); \((1+0.8(1.5))^{-1/0.8}=2.2^{-1.25}\approx0.373226\); and \((1+0.8(3))^{-1/0.8}=3.4^{-1.25}\approx0.216596\). The expected total is

E_{NB2}=80(0.656659)+80(0.373226)+40(0.216596) \approx91.0546.

Using those displayed probabilities, the plug-in variance is \(80(0.656659)(1-0.656659)+80(0.373226)(1-0.373226)+40(0.216596)(1-0.216596)\approx43.5382\). Its square root is approximately \(6.59835\). Thus, \(D=96-91.0546\approx4.9454\), and the standardized discrepancy is \(4.9454/6.59835\approx0.749\) to three decimal places.

The result is a modest discrepancy on this plug-in scale, not evidence by itself that the NB2 model should be rejected or that a particular alternative mechanism is present. The calculation also illustrates why fitted means should be handled separately: averaging the three means first and then calculating one zero probability would not give the expected count shown here.

Look for Patterns, Not Only a Total

An aggregate comparison can conceal important structure. A model might predict the right total number of zeros while systematically underpredicting zeros for one subgroup and overpredicting them for another. Examine \(r_i\), or summarize observed and expected zeros by covariate groups, fitted-mean ranges, exposure, time, or other scientifically relevant dimensions. Such summaries are descriptive diagnostics; groups chosen after inspecting the data should not be treated as independent confirmatory tests.

Worked Example: Locating a Group-Specific Discrepancy

A hypothetical Poisson model predicts a zero probability of \(0.4\) for \(50\) observations in one exposure group and \(0.2\) for \(50\) observations in another. The observed zero counts are \(30\) and \(14\), respectively. Compare the groups and the total.

The first group has \(50(0.4)=20\) expected zeros, so its discrepancy is \(30-20=10\). The second has \(50(0.2)=10\) expected zeros, so its discrepancy is \(14-10=4\). Together, \(O=44\), \(E=30\), and \(D=14\). The plug-in variance is \(50(0.4)(0.6)+50(0.2)(0.8)=12+8=20\), giving a standardized discrepancy of \(14/\sqrt{20}\approx3.130\).

Both groups contribute positive discrepancies, and the first contributes \(10\) of the total \(14\). That pattern may prompt a closer check of the exposure definition, omitted group characteristics, or whether the model describes the count distribution adequately in each group. It does not establish that either group has a separate zero-generating process. As in the earlier example, the standardization is a plug-in diagnostic rather than a fully calibrated test.

Assessing Evidence and Its Limits

A careful diagnosis combines the numerical comparison with knowledge of how the outcome is recorded and generated. First check data coding: a recorded zero should represent no events, not a missing value, an unobserved period, or a value assigned by a reporting rule. Confirm that exposures and observation windows are appropriate. Then inspect whether zero discrepancies cluster in substantively meaningful ways, and whether the fitted model captures relevant covariate variation and dispersion.

When formal calibration is needed, a parametric bootstrap can incorporate parameter estimation. Fit the chosen count model, simulate datasets under that fitted model with the same covariates and observation structure, refit the model to each simulated dataset using the original fitting procedure, and recalculate the chosen zero-discrepancy statistic each time. Compare the observed statistic with this simulated reference distribution. Refitting matters: it represents the estimation step that the fixed-probability plug-in calculation ignores. The bootstrap still assesses the specified model and statistic; it cannot remedy incorrect sampling assumptions or prove a substantive mechanism.

Excess zeros are compatible with multiple explanations. The count distribution may be misspecified, covariates may be missing, observations may be dependent, or some units may have no opportunity to experience the event. Distinguishing such explanations requires design knowledge and model assessment, not just the sign or size of \(O-E\). In particular, a positive discrepancy is motivation to investigate, not automatic justification for a zero-inflated specification.

Common Mistakes

Calling any large zero count “excess zeros.” A raw count has no reference point. Compare it with the expected count under a clearly stated fitted model.
Using the mean of fitted means to predict zeros. Calculate each \(\hat{q}_i\) and sum them; nonlinear zero-probability formulas do not generally commute with averaging.
Treating a mean model as a complete distribution. A conditional mean alone does not determine \(P(Y_i=0)\). Use a specified distributional model for that probability.
Reading a plug-in standardized discrepancy as a definitive test. Its simple variance calculation holds probabilities fixed and assumes independence. Parameter estimation, clustering, and other design features may require additional calibration.
Concluding that excess zeros prove a special zero process. They are a model diagnostic, and several forms of misspecification or data-collection features can produce the same pattern.

Key takeaway: Diagnose zeros by comparing observed counts with the sum of model-implied zero probabilities, then examine residual patterns and the data-generating context. Treat the discrepancy as evidence about model fit—not, by itself, as a diagnosis of its cause.

Check Your Understanding

Use these questions to check whether you can calculate and interpret a zero diagnostic.

A Poisson model gives two observations fitted means of \(0.2\) and \(2\). Write the expected zero count for those observations. Why is it not generally equivalent to apply the Poisson zero formula to their average fitted mean?
State the NB2 variance function and its fitted zero-probability formula. What does the dispersion parameter represent in the variance function?
For independent Bernoulli zero indicators with known probabilities \(q_i\), derive the variance of their sum.
What does a positive value of \(O-E\) mean? Give one reason it does not prove a distinct zero-generating process.
Why can refitting each simulated dataset be important when using a parametric bootstrap to assess a zero-count discrepancy?