Count Prediction and Calibration

From a Fitted Count Model to a Forecast

A fitted count regression gives more than a coefficient table: it can forecast an expected count for a specified covariate profile and, under a distributional model, describe the range of counts that could occur. These are different outputs. A forecast of 2.2 events is a conditional mean; an individual outcome must still be a whole number, and may be zero, one, or several events.

Calibration asks whether forecasts agree with outcomes at the level being assessed. A model can have a plausible mean forecast for one case and still be poorly calibrated across cases, groups, or the full count distribution. The comparison must therefore match the question: totals for aggregate calibration, and count frequencies for distributional calibration.

Definition: For observation \(i\), let \(Y_i\) be the count outcome and \(x_i\) the covariate vector. A fitted count model produces a conditional mean forecast \(\hat{\mu}_i\), an estimate of \(E(Y_i\mid x_i)\). A predictive distribution assigns probabilities to possible values of \(Y_i\), conditional on the covariates and fitted model. The mean forecast is the center of that distribution, not a guarantee about the realized count.

For a Poisson regression with a log link, the fitted mean is \(\hat{\mu}_i=\exp(x_i'\hat{\beta})\). If the model includes exposure \(t_i\) as an offset, it is \(\hat{\mu}_i=\exp(x_i'\hat{\beta}+\log t_i)\). The offset makes the expected count proportional to exposure, holding the other covariates fixed. The exposure must be available or specified for the case being forecast.

A Poisson predictive distribution assumes \(Y_i\mid x_i\sim\text{Poisson}(\mu_i)\), with probability mass function \(P(Y_i=y\mid x_i)=\exp(-\mu_i)\mu_i^y/y!\), for \(y=0,1,2,\ldots\). For forecasting, replace \(\mu_i\) with \(\hat{\mu}_i\). The conditional mean and variance in this model are both \(\mu_i\). In a negative-binomial regression, the fitted conditional mean is also used to forecast, but the count probabilities come from the fitted negative-binomial distribution and its estimated dispersion parameter. The particular negative-binomial parameterization matters for those probabilities; below, the examples use the size-parameter form.

Worked Example: Poisson Probabilities for One Forecast

Suppose a fitted Poisson regression gives a hypothetical customer a conditional mean forecast of \(\hat{\mu}=2.2\) service calls during the next month. Treating the covariates and fitted parameters as fixed for this calculation, find the probabilities of zero, one, and two calls, and the probability of at least three.

For a Poisson count, \(P(Y=y)=\exp(-2.2)2.2^y/y!\). Thus

\begin{aligned} P(Y=0)&=\exp(-2.2)\frac{2.2^0}{0!}\approx 0.1108,\\ P(Y=1)&=\exp(-2.2)\frac{2.2^1}{1!}\approx 0.2438,\\ P(Y=2)&=\exp(-2.2)\frac{2.2^2}{2!}\approx 0.2681. \end{aligned}

The probabilities for zero, one, and two calls sum to approximately \(0.6227\). Therefore \(P(Y\geq 3)=1-\{P(Y=0)+P(Y=1)+P(Y=2)\}\approx 0.3773\), using unrounded probabilities in the calculation. These probabilities describe possible monthly counts under the fitted Poisson model. The mean forecast remains 2.2 calls per month; it is not a forecast that this customer will make exactly 2.2 calls.

Prediction Summaries Under the Negative-Binomial Model

A predictive distribution can summarize uncertainty in an individual count more fully than its mean alone. For a discrete count, one useful summary is a predictive interval: a range of integer values whose model-based predictive probability is at least a stated level. Because a count distribution is discrete, the achieved probability may exceed the nominal level. This interval concerns a future count conditional on the fitted model; it is not a confidence interval for a regression coefficient.

For a negative-binomial distribution in the size-parameter form, let the conditional mean be \(\mu\) and the size parameter be \(r>0\). Its count probabilities can be written as

P(Y=y\mid x)= \frac{\Gamma(y+r)}{\Gamma(r)y!} \left(\frac{r}{r+\mu}\right)^r \left(\frac{\mu}{r+\mu}\right)^y,\qquad y=0,1,2,\ldots .

Here \(\Gamma(\cdot)\) is the gamma function, which satisfies \(\Gamma(k)=(k-1)!\) for positive integers \(k\). In this parameterization the conditional variance is \(\mu+\mu^2/r\). A fitted model supplies \(\hat{\mu}_i\) and an estimate of \(r\); substituting them into the probability formula gives forecasts of individual count probabilities. This calculation does not explain how the negative-binomial model is derived or estimated.

Worked Example: A Negative-Binomial Prediction Summary

Consider a hypothetical case with fitted mean \(\hat{\mu}=2\) and fitted size parameter \(r=2\). Calculate the probabilities of zero, one, and two events, the probability of three or more events, and the model-based conditional standard deviation.

The probability formula has \(r/(r+\hat{\mu})=2/4=0.5\) and \(\hat{\mu}/(r+\hat{\mu})=0.5\). For zero events,

P(Y=0)=\left(\frac{2}{4}\right)^2=0.25.

For one event the combinatorial factor is \(\Gamma(3)/\{\Gamma(2)1!\}=2\), so \(P(Y=1)=2(0.5)^2(0.5)=0.25\). For two events the factor is \(\Gamma(4)/\{\Gamma(2)2!\}=3\), so \(P(Y=2)=3(0.5)^2(0.5)^2=0.1875\). Consequently,

P(Y\geq 3)=1-(0.25+0.25+0.1875)=0.3125.

The conditional variance is \(\hat{\mu}+\hat{\mu}^2/r=2+4/2=4\), so the conditional standard deviation is \(\sqrt{4}=2\) events. For an equal-tailed 95% predictive interval, the lower endpoint is zero: the probability below zero is zero, and the cumulative probability through zero is 0.25. The upper endpoint is seven: the cumulative probability through six is approximately 0.9648, below 0.975, while the cumulative probability through seven is approximately 0.9805. Thus a discrete equal-tailed interval is 0 to 7 events, with model-based coverage approximately 98.05%. Its coverage exceeds 95% because the possible outcomes are integers.

These summaries condition on the fitted parameters. They do not automatically include uncertainty from estimating regression coefficients or dispersion. If that uncertainty is material, a forecasting procedure can account for it by drawing parameter values from an estimated sampling distribution, recalculating the predictive distribution for each draw, and combining the resulting forecasts. That additional uncertainty is distinct from the variation in the future count itself.

Aggregate Calibration

For a set of \(n\) cases with fitted means \(\hat{\mu}_i\), the sum of the forecasts is the predicted total, \(\sum_{i=1}^n\hat{\mu}_i\). The observed total is \(\sum_{i=1}^n y_i\). Comparing these totals checks whether the model systematically predicts too many or too few counts overall in the cases being assessed. The predicted total need not be an integer: it is an expected count total.

The justification is linearity of expectation. If each forecast is the conditional mean, then

E\left(\sum_{i=1}^nY_i\mid x_1,\ldots,x_n\right) =\sum_{i=1}^n E(Y_i\mid x_1,\ldots,x_n) =\sum_{i=1}^n\mu_i.

This result does not require independence of the outcomes. Independence, or a model for dependence, does matter for calculating the variance of the aggregate total and for constructing a prediction interval for that total. A close observed and predicted total is evidence only about aggregate calibration for the cases considered; it does not show that every subgroup or count probability is well calibrated.

Worked Example: Comparing Observed and Predicted Totals and Frequencies

For six hypothetical cases, a fitted Poisson model gives means \(0.5, 1, 1.5, 2, 2.5,\) and \(3\). The observed counts are \(0, 0, 1, 2, 4,\) and \(3\). Compare the totals and the observed and predicted count frequencies in four categories: zero, one, two, and three or more.

The observed total is \(0+0+1+2+4+3=10\) counts. The predicted total is \(0.5+1+1.5+2+2.5+3=10.5\) counts. The ratio of observed to predicted totals is \(10/10.5\approx 0.9524\), so the observed total is about 4.76% below the predicted total. With only six cases, this is a descriptive comparison, not a basis for a broad claim about model performance.

For the distributional comparison, calculate the Poisson probability in each category at each case’s fitted mean, then sum probabilities across the six cases. For example, the expected number of zeros is

\sum_{i=1}^6 \exp(-\hat{\mu}_i) =\exp(-0.5)+\exp(-1)+\exp(-1.5)+\exp(-2)+\exp(-2.5)+\exp(-3) \approx 1.4647.

Repeating the calculation for the other categories gives expected frequencies of approximately 1.6311 for one count, 1.2620 for two counts, and 1.6422 for three or more. These expected frequencies sum to 6, the number of cases. The observed frequencies are 2, 1, 1, and 2, respectively.

Count category	Observed frequency	Expected frequency from fitted Poisson distributions
0	2	1.4647
1	1	1.6311
2	1	1.2620
3 or more	2	1.6422

The total comparison looks close, while the observed zero frequency is higher than its expected frequency. These results illustrate why matching totals alone cannot establish that a model reproduces the full distribution. The expected frequency in a category is the sum of the case-specific probabilities of falling in that category; it is generally not obtained by applying a single distribution to the average fitted mean.

Calibration in Practice

Calibration can be examined at several resolutions. For an overall check, compare observed and predicted totals. To see whether forecasts align across the prediction range, group cases by predicted mean and compare each group’s observed average count with its average predicted mean. For distributional calibration, compare observed frequencies with the sums of predicted probabilities for chosen count categories. The grouping and categories should be chosen so that the comparison is informative, not so finely that nearly every cell contains very few cases.

Whenever possible, evaluate calibration on data not used to fit the model. In-sample comparisons can look favorable partly because the model was estimated using those outcomes. In a validation sample, apply the fitted model to covariates and exposures for cases not used in fitting, then compare their outcomes with their forecasts. A validation comparison still has sampling variability; small differences between observed and predicted counts do not necessarily indicate meaningful miscalibration.

The forecast population also matters. If the cases being forecast have different covariate values, exposure, or selection rules from the model’s estimation sample, the comparison may reflect that change as well as model performance. State the population, time period, and unit of exposure clearly. Predicted counts for a month should not be compared directly with observed counts for a year without accounting for the differing exposure.

Common Mistakes

Treating a conditional mean as an individual outcome. A forecast of 2.2 events does not mean that 2.2 events will occur. It is a mean; the predictive distribution assigns probabilities to integer counts.
Using a Poisson probability formula for a negative-binomial forecast. A fitted mean by itself does not determine negative-binomial probabilities. Use the fitted dispersion parameter and the chosen parameterization as well.
Comparing only aggregate totals. Similar totals can coexist with errors in particular groups or in the probabilities of zero and larger counts. Check the level of calibration relevant to the intended use.
Calling the expected total a prediction interval. The sum of fitted means is an expected total, not a range for the realized total. A range requires a predictive distribution for the aggregate, including any dependence across cases.
Interpreting calibration as causation. Agreement between forecasts and outcomes concerns predictive performance under the assessed conditions; it does not show that changing a covariate would cause the predicted change in counts.

Key takeaway: A fitted count model can supply both conditional mean forecasts and predictive probabilities. Assess calibration by comparing observed outcomes with the corresponding predictions at a clearly specified level: total, group, or count category.

Check Your Understanding

Try these questions without looking back.

A Poisson model forecasts a conditional mean of 1.8 events. What does this number say, and what does it not say, about the count for one case?
In the negative-binomial size-parameter form, what information besides the fitted mean is needed to calculate count probabilities?
For several cases, explain why the expected total count is the sum of the case-specific conditional means.
Why can observed and predicted totals agree even when the model’s count-distribution calibration is imperfect?
What additional issue must be addressed before using a sum of fitted means as a prediction interval for the realized aggregate total?