Why a Large Number of Zeros Is Not Enough
Count data often contain many zeros. A policy may have no claims during the observation period; a customer may make no purchases in a month; a device may record no failures. A high zero count can look surprising, but the relevant question is not whether zeros are common in isolation. It is whether the observed number of zeros is surprising given the means predicted by the specified Poisson GLM.
A Poisson model with a small fitted mean predicts a high probability of zero. Conversely, a model assigning high means to many observations predicts fewer zeros. When fitted means vary across observations, even the overall zero probability cannot be calculated from one average mean: each observation contributes its own probability of zero.
Expected Zeros Under a Poisson GLM
Suppose the responses \(Y_1,\ldots,Y_n\) are conditionally independent and \(Y_i\sim\operatorname{Poisson}(\mu_i)\), where \(\mu_i>0\) is the mean specified by the fitted model for observation \(i\). Under a Poisson GLM, the mean is linked to the linear predictor; with the log link, for example, \(\log(\mu_i)=\eta_i\). The probability of a zero for observation \(i\) is
The expected number of zeros is therefore the sum of those individual probabilities. Define \(I_i=1\) when \(Y_i=0\), and \(I_i=0\) otherwise. The total zero count is \(Z=\sum_{i=1}^n I_i\). Since \(E[I_i]=P(Y_i=0)\), linearity of expectation gives
This result does not require equal means. Under the additional assumption of conditional independence, the indicators are independent Bernoulli variables, so
The formulas describe a model whose means are treated as known. In practice, a GLM’s means are estimated from the same data being checked. As a result, a simple comparison of observed and expected zeros, or a standardized difference using the variance formula, is a screening diagnostic, not automatically an exact test with a calibrated significance level. A formal assessment must account for model fitting and the particular analysis.
Worked Example: Is a High Zero Count Plausible at a Low Mean?
A hypothetical portfolio contains 100 independent policy-periods. A Poisson model assigns every period a mean claim count of \(0.2\), and 85 periods have no claims. Assess whether 85 zeros are obviously inconsistent with the model using the expected count and a standardized comparison. Treat the specified means as fixed for this calculation.
Method. Calculate each period’s probability of zero, then the expected zero count and its standard deviation under independence.
For each period,
Thus the expected number of zero periods is
The variance and standard deviation of the zero count are approximately
The observed count is \(85\), which is \(85-81.87=3.13\) above expectation. Its standardized difference is approximately \(3.13/3.85=0.81\). That is not a large discrepancy by this rough comparison. In particular, 85 zeros is not compelling evidence of excess zeros merely because it sounds like a large count.
Check. The expected zero proportion is \(0.8187\), so an expected count near 82 out of 100 is consistent with the low mean \(0.2\). The calculation avoids the mistake of treating a majority of zeros as inherently suspicious.
Different Fitted Means Require Different Zero Probabilities
When a GLM assigns different means to observations, calculate \(\exp(-\mu_i)\) for each mean and add the results. Substituting an overall average mean into the zero-probability formula generally gives the wrong expected zero count. The exponential function is nonlinear, so the average of \(\exp(-\mu_i)\) need not equal \(\exp(-\bar{\mu})\).
Worked Example: Combining Two Groups With Different Means
In a hypothetical study, 50 independent units in group A have specified Poisson mean \(0.1\), and 50 independent units in group B have specified mean \(0.9\). There are 77 observed zeros. Treat both group means as fixed and use a standardized comparison to assess the discrepancy.
Method. Calculate the expected zeros and variance separately by group, then add them. This respects the different fitted means.
For group A, the zero probability is \(\exp(-0.1)=0.9048\). Its expected zero count is \(50(0.9048)=45.24\). For group B, the probability is \(\exp(-0.9)=0.4066\), giving \(50(0.4066)=20.33\) expected zeros. Therefore,
The variance contributions are \(50(0.9048)(1-0.9048)=4.31\) for group A and \(50(0.4066)(1-0.4066)=12.06\) for group B. Hence,
The observed count exceeds expectation by \(77-65.5704=11.4296\), or approximately \(11.4296/4.05=2.82\) standard deviations. Under the fixed-mean, independence calculation, this is a noticeably larger discrepancy than in the previous example and merits investigation. It is still a diagnostic comparison, not a formal test that accounts for estimating the means.
Check. The group-specific expected counts sum to about \(65.57\). Using the overall mean \(0.5\) for all 100 units would instead give \(100\exp(-0.5)=60.65\), which is not the correct expectation for these two groups. The mistake is replacing heterogeneous means by their average before applying the exponential.
What a Standard Poisson Fit Can Miss
A standard Poisson GLM specifies a full conditional distribution, not just a relationship between predictors and the expected count. Conditional on its mean \(\mu_i\), it assigns probabilities to zero, one, two, and higher counts according to the Poisson distribution. A model can therefore have a reasonable fitted average count while predicting the wrong distribution of counts, including too few zeros.
For an intercept-only Poisson model, the maximum-likelihood estimate of the common mean is the sample mean. This follows from the likelihood: for observed counts \(y_1,\ldots,y_n\), the part of the log-likelihood depending on \(\mu\) is \(\sum_i y_i\log(\mu)-n\mu\). Differentiating gives \(\sum_i y_i/\mu-n=0\), so the maximizing value is \(\hat{\mu}=\sum_i y_i/n\). The fit must balance all observed counts; it does not choose its mean solely to reproduce the observed number of zeros.
Worked Example: A Fitted Mean Can Conceal a Poor Count Distribution
A hypothetical intercept-only data set has 100 independent observations: 70 counts are zero and 30 counts are exactly two. Fit a standard Poisson model with one common mean. Calculate the fitted mean, compare observed and expected frequencies, and calculate the Poisson deviance.
Method. Estimate the intercept-only mean by the sample mean, then use the fitted Poisson probabilities and the deviance formula to examine distributional fit.
The total count is \(70(0)+30(2)=60\), so the sample mean and fitted Poisson mean are
The fitted probabilities for counts zero, one, and two are
The model therefore predicts approximately 54.88 zeros, 32.93 ones, and 9.88 twos among 100 observations. The observed frequencies are 70 zeros, no ones, and 30 twos. Although the fitted mean equals the observed mean, the predicted distribution differs sharply from the observed pattern.
For counts \(y_i\) with fitted means \(\hat{\mu}_i\), the Poisson deviance is
Each of the 70 zeros contributes \(2(0.6)=1.2\), for a total of \(84\). Each of the 30 counts equal to two contributes
The positive counts contribute \(30(2.0159)=60.48\), so the total deviance is approximately \(84+60.48=144.48\). This is a substantial discrepancy to investigate, considered alongside the model’s degrees of freedom and other diagnostics; it is not, by itself, a universal cutoff or a formal test.
Check. The fitted mean is \(0.6\), but that does not mean the Poisson model reproduces the observed zero and positive-count frequencies. The common mistake is to judge fit from the mean alone. This example also shows why excess zeros need not be the only mismatch: the distribution of positive counts can disagree with the Poisson specification as well.
Interpretation, Consequences, and Next Steps
A standard Poisson GLM fitted to data with more zeros than its distribution predicts may still estimate an average count, but that does not ensure that its probability predictions or likelihood-based fit are adequate. If the zero pattern differs across predictor groups, the fitted mean relationship may also be inadequate. Inferences based on the Poisson variance assumption can be unreliable when the conditional count distribution is substantially different from the assumed one.
An excess-zero diagnostic does not identify the cause. Potential explanations include omitted predictors, unaccounted-for differences in exposure or risk, dependence between observations, or a process that genuinely has a distinct zero-generating component. A zero-inflated model represents a mixture in which some observations are in a zero-only state and others follow a count distribution. A hurdle model separately represents whether the count is zero or positive and then models the positive counts. These are different assumptions about the data-generating process; neither should be selected solely because the observed zero count is large.
The earlier material on overdispersion is relevant because both extra zeros and other forms of heterogeneity can lead to variance patterns that conflict with a simple Poisson model. However, a variance comparison alone does not establish that zeros are excessive: compare the observed zero count with the probabilities implied by the fitted means, and consider the rest of the count distribution as well.
Confirm that the response is a count, that the exposure and predictors are represented appropriately, and that the fitted Poisson means correspond to the observations being assessed.
For each fitted mean \(\hat{\mu}_i\), calculate \(\exp(-\hat{\mu}_i)\), then sum to obtain the expected zero count.
Consider the size of the observed-versus-expected difference, the uncertainty in that comparison, and whether other count frequencies or residual diagnostics also indicate a mismatch.
Describe evidence of model inadequacy without claiming that a particular zero-generating mechanism has been proven. Any alternative model should have a defensible interpretation and be assessed on its assumptions and fit.
Common Exam Traps
- Calling every high zero count “excess.” A small Poisson mean can make zeros entirely expected. Calculate the model-implied probability first.
- Using one average mean for heterogeneous observations. Add \(\exp(-\hat{\mu}_i)\) across observations; do not generally use \(n\exp(-\bar{\mu})\).
- Treating a screening statistic as a formal test. Fitted means are estimated, and the simple variance calculation assumes independent observations with fixed means.
- Assuming excess zeros prove a special zero process. The discrepancy may instead reflect predictors, exposure, dependence, heterogeneity, or broader distributional misspecification.
- Checking only the mean. Matching the average count does not guarantee that the model reproduces the frequency of zeros or positive counts.
Check Your Understanding
For each problem, use the stated Poisson means and independence assumptions. Round expected counts to four decimal places where applicable.
- A hypothetical set of 40 independent observations has a common Poisson mean of \(0.1\). Calculate the expected number of zeros. (Answer: \(36.1935\) expected zeros)
- A study has 25 independent observations with Poisson mean \(0.4\) and 25 with mean \(1.2\). Calculate the expected total number of zeros using the group-specific means. (Answer: \(24.2879\) expected zeros)
- For 80 independent observations with fixed Poisson mean \(0.5\), 55 responses are zero. Calculate the expected number of zeros and state whether the observed count is above or below expectation. (Answer: \(48.5225\) expected zeros; observed is above expectation by \(6.4775\))
- An intercept-only Poisson model is fitted to 50 observations: 35 are zero and 15 are one. Calculate the fitted mean and the model’s expected number of zeros. Is the observed zero count above or below that expectation? (Answer: fitted mean \(0.3000\); \(37.0409\) expected zeros; observed is below expectation by \(2.0409\))
- A model assigns fixed means of \(0.2\) to 20 independent observations and \(0.8\) to 30 independent observations. If 35 responses are zero, calculate the expected total number of zeros and the observed-minus-expected difference. (Answer: \(29.8545\) expected zeros; observed-minus-expected difference \(5.1455\))