Why the Number of Zeros Matters
A count model can reproduce an overall average and still describe the observed counts poorly. One important check is whether it predicts a plausible number of zero counts. For an insurer studying hypothetical claim counts, for example, a model might predict the average number of claims per policy accurately but predict far fewer no-claim policies than are observed. That difference is a warning to examine the model’s assumptions and predictors.
The comparison must use the fitted mean for each observation, not automatically one pooled mean. Different exposures or predictor values can imply different probabilities of zero. This tutorial focuses on that comparison: calculating the zeros a Poisson model predicts, reading the observed count pattern, and interpreting a mismatch cautiously.
A Poisson model is a plausible description only when its assumptions are reasonable for the counts being studied. In particular, conditional on the predictors and exposure, the model assumes a Poisson distribution, whose conditional mean and variance are both \(\mu_i\). It also assumes that the included predictors and exposure adequately represent systematic differences in expected counts. A mismatch in zeros can signal a problem with these assumptions, but cannot by itself identify which assumption failed.
From a Fitted Mean to an Expected Number of Zeros
For a single observation with fitted mean \(\mu_i\), the Poisson probability mass function gives
Setting \(y=0\) gives the probability of a zero. To find the expected number of zeros among \(n\) observations, define \(I_i=1\) if \(Y_i=0\), and \(I_i=0\) otherwise. The total number of observed zeros is \(Z=\sum_{i=1}^n I_i\). Since the expectation of a sum is the sum of the expectations, the expected number of zeros is the sum of the individual zero probabilities.
This result does not require independence: linearity of expectation is enough. If the counts are conditionally independent and the fitted means are treated as fixed, then \(Z\) is a sum of independent Bernoulli variables with probabilities \(e^{-\mu_i}\), and
When the means are estimated from the same data, this variance is not a complete calibration for a formal goodness-of-fit test because it ignores parameter estimation. It is still useful for understanding the expected zero count. Formal assessment should account for fitting the model, for example by simulating counts under the fitted model and refitting it in each simulation.
The reason is that the probability of zero is a nonlinear function of the mean. The second derivative of \(e^{-x}\) is \(e^{-x}>0\), so the function is convex. For different fitted means, averaging \(e^{-\mu_i}\) will generally give a different result from applying \(e^{-x}\) to their average. In fact, Jensen’s inequality gives
Thus, using the overall average fitted mean in place of the individual means generally underestimates the expected zero proportion when the means vary.
Worked Example: A Homogeneous Poisson Count Pattern
A hypothetical research team records the number of equipment faults at each of 100 identical monitoring sites over the same period. Suppose the counts are modeled as independent Poisson random variables with common mean \(\lambda\). The observed total number of faults is 120, and 48 sites have no faults. Calculate the fitted expected number of zeros and compare the observed count pattern with the model.
Method. For a common-mean Poisson model, the maximum-likelihood estimate of \(\lambda\) is the sample mean. The fitted mean is \(120/100=1.2\) faults per site. The model assumptions here are a common mean, Poisson counts, and independence across sites.
Expected zeros. The fitted zero probability for one site is
Therefore, the expected number of zero-fault sites is
The observed number is 48, or 48% of the sites, compared with a fitted zero probability of about 30.12%. The difference is \(48-30.1194\ldots=17.8806\ldots\), approximately 17.88 sites.
Check the broader pattern. One possible observed frequency table consistent with the stated total has 48 zeros, 20 ones, 12 twos, 10 threes, 6 fours, 2 fives, and 2 sixes. These frequencies sum to \(48+20+12+10+6+2+2=100\). Their total faults are \(20+24+30+24+10+10+12=130\), not 120, so that table is not consistent with the stated total. A consistent frequency table is 48 zeros, 20 ones, 12 twos, 10 threes, 6 fours, 2 fives, and 2 sixes only if the total is 130; it cannot be used here. The zero comparison itself uses the stated total and fitted mean, so no additional frequency table is needed to make that comparison.
The correct conclusion is that the zero count is substantially above the model’s expected count and warrants further checking. It is not a formal rejection by itself: a formal test would need an appropriate reference distribution that accounts for estimating \(\lambda\). Common mistake avoided: treating an expected count as a guaranteed count, or claiming a definitive model failure solely because observed and expected zeros differ.
Worked Example: Unequal Exposures and Zero Probabilities
A hypothetical portfolio has 100 policies. Forty policies each have 0.5 years of exposure, and 60 each have 1.5 years. A fitted Poisson rate is 0.4 claims per policy-year, so the fitted mean for a policy is its rate multiplied by its exposure. Suppose 74 policies have zero claims. Calculate the model’s expected number of zeros and identify the error in a pooled calculation.
Method. The model is \(Y_i\sim\operatorname{Poisson}(\mu_i)\), with \(\mu_i=0.4t_i\), where \(t_i\) is exposure in years. The fitted means are \(0.4(0.5)=0.2\) for each of the 40 shorter-exposure policies and \(0.4(1.5)=0.6\) for each of the 60 longer-exposure policies.
Expected zeros using individual exposures. The zero probability in the first group is \(e^{-0.2}=0.8187308\ldots\), giving \(40e^{-0.2}=32.7492\ldots\) expected zeros. In the second group it is \(e^{-0.6}=0.5488116\ldots\), giving \(60e^{-0.6}=32.9287\ldots\) expected zeros. Therefore,
The observed number, 74, is \(74-65.6779\ldots=8.3221\ldots\), approximately 8.32 above the fitted expectation.
Common mistake and correction. The total fitted mean is \(40(0.2)+60(0.6)=44\), so the average fitted mean is \(44/100=0.44\). Using \(100e^{-0.44}\) gives \(100(0.644036\ldots)=64.40\) expected zeros. This is the wrong calculation because it applies the zero-probability formula to an average mean instead of to the two fitted means. The correct expected count is 65.68. In this case, the shortcut understates the expected zeros, as convexity predicts.
Worked Example: Checking Zeros Within Risk Groups
A hypothetical model assigns fitted Poisson means of 0.2 to 50 low-risk locations, 1.0 to 30 medium-risk locations, and 2.0 to 20 high-risk locations. The observed zero counts are 48, 15, and 5 in those respective groups. The observed total count across all locations is 80. Compare the zero pattern with the fitted model, including the model’s expected total count.
Method. Apply \(e^{-\mu}\) within each group, then multiply by the number of locations in that group. This preserves the fitted risk differences instead of replacing them with one pooled mean.
For the low-risk group, the expected number of zeros is \(50e^{-0.2}=50(0.8187308\ldots)=40.9365\ldots\), approximately 40.94. For the medium-risk group, it is \(30e^{-1}=30(0.3678794\ldots)=11.0364\ldots\), approximately 11.04. For the high-risk group, it is \(20e^{-2}=20(0.1353353\ldots)=2.7067\ldots\), approximately 2.71. The total is
There are \(48+15+5=68\) observed zeros, about 13.32 more than the fitted expectation. The expected total count is \(50(0.2)+30(1.0)+20(2.0)=10+30+40=80\), which equals the observed total. Thus, matching the overall count total does not ensure that the model matches the zero frequency. The excess occurs within each risk group: 48 versus 40.94, 15 versus 11.04, and 5 versus 2.71.
This pattern is a clear reason to investigate the model, but it does not establish a single explanation. The groups may still contain meaningful unmodeled differences, counts may not be conditionally independent, or a Poisson distribution may not describe the conditional variation adequately. Common mistake avoided: deciding that a model fits because its expected total count matches the observed total.
Interpreting an Excess of Zeros
An observed zero count above the fitted expectation is often described as excess zeros. The phrase describes a discrepancy; it does not name its cause. Before concluding that the response needs a different distribution, check whether the model has the right exposure, whether important predictors or group differences have been omitted, and whether observations that were treated as independent could be related.
The rest of the count pattern matters as well. A model that predicts too few zeros may also predict too many small positive counts or too few large counts. Comparing observed and expected frequencies across several count values can reveal whether the problem is concentrated at zero or reflects a broader mismatch. The zero count is a useful diagnostic, not a substitute for examining the counts and the model specification.
Nor does every difference between observed and expected zeros indicate a meaningful failure. Under a model, random variation is expected. There is no universal cutoff that turns a difference into proof of a problem. When a formal goodness-of-fit assessment is required, its reference distribution should reflect how the model was fitted; simulation under the fitted model is one way to do this. A simple binomial calculation is appropriate only for a common zero probability with fixed parameters, and is not automatically valid after estimating parameters or when probabilities differ by observation.
Use each observation’s predictors and exposure to find its fitted Poisson mean \(\mu_i\).
For each observation, calculate \(e^{-\mu_i}\), or calculate the probability once per group when fitted means are identical within groups.
Add the fitted zero probabilities to get the expected number of zeros, then compare it with the observed number overall and, where useful, by group.
Inspect the broader count pattern and model assumptions. Treat excess zeros as a warning to investigate, not as proof of a particular cause.
Check Your Understanding
For each problem, calculate or interpret the fitted zero pattern using the information given.
- A hypothetical group of 80 identical devices has counts modeled as independent Poisson variables with mean 0.5 failures per device-period. Find the fitted zero probability and expected number of zero-failure devices. (Answer: zero probability \(e^{-0.5}=0.6065\); expected zeros \(80e^{-0.5}=48.52\).)
- A hypothetical policy group has 30 policies with 0.25 years of exposure and 20 policies with 1 year of exposure. The fitted rate is 0.8 claims per policy-year. Find the expected number of policies with zero claims, using the correct exposure-specific means. (Answer: \(30e^{-0.2}+20e^{-0.8}=33.55\) expected zero-claim policies.)
- A fitted model gives mean 0.4 for 25 observations and mean 1.4 for another 25 observations. Calculate the expected total number of zeros. (Answer: \(25e^{-0.4}+25e^{-1.4}=22.92\) expected zeros.)
- In the setting of question 3, suppose 34 zeros are observed. State the discrepancy and what conclusion is justified. (Answer: 34 is about \(34-22.92=11.08\) above expectation; this is a warning to investigate the model, not proof of a specific cause or a formal rejection by itself.)
- A hypothetical sample has 50 observations and a total count of 40. A common-mean Poisson model is fitted, and 28 observations are zero. Calculate the fitted mean and expected number of zeros, then compare with the observed count. (Answer: fitted mean \(40/50=0.8\); expected zeros \(50e^{-0.8}=22.47\); the observed count is about 5.53 above expectation, warranting further model checking.)