Calibration of Poisson Counts and Rates

Why Calibrate Poisson Predictions?

A Poisson GLM can predict how many events to expect for each observation, but a good fit summary does not by itself show whether those predictions agree with observed counts in a useful way. Calibration asks whether predicted event totals correspond to observed event totals, especially across groups with different predicted rates. The comparison must respect exposure: ten events over a thousand person-years do not indicate the same rate as ten events over one hundred person-years.

Let \(Y_i\) be the observed count for unit \(i\), and let \(E_i>0\) be its exposure. Under the usual Poisson rate model, \(Y_i\) has mean \(\mu_i=E_i\lambda_i\), where \(\lambda_i\) is the event rate per exposure unit. A fitted model gives a predicted mean count \(\hat{\mu}_i=E_i\hat{\lambda}_i\). The offset \(\log E_i\) in a log-link model ensures that the predicted count incorporates exposure.

Groups are commonly formed by sorting observations by predicted rate or predicted count and dividing them into bins. Whatever the grouping rule, the count comparison is \(O_g\) against \(M_g\). The rate comparison is \(O_g/E_g\) against \(M_g/E_g\), not against an unweighted average of the individual predicted rates.

Aggregating Counts and Rates Correctly

The following result explains why group totals are natural for Poisson calibration. It assumes independent counts with fixed means conditional on the covariates and exposures. “Fixed” here means that the predictions are treated as given for the calculation; it does not account for the uncertainty from estimating model parameters.

To verify the aggregation claim, use the probability-generating function of a Poisson random variable. For \(Y_i\sim\operatorname{Poisson}(\mu_i)\), \(E[t^{Y_i}]=\exp\{\mu_i(t-1)\}\). Independence gives

This is the probability-generating function of a Poisson variable with mean \(\sum_{i\in g}\mu_i\), proving the first statement. A Poisson variable has variance equal to its mean. Under individual-mean calibration, \(\sum_{i\in g}\mu_i=M_g\); dividing \(O_g\) by the fixed \(E_g\) divides its mean by \(E_g\) and its variance by \(E_g^2\). Thus the claimed rate mean and variance follow. \(\square\)

The group predicted rate \(M_g/E_g\) is an exposure-weighted average of the individual predicted rates:

An observation with greater exposure contributes more to both the predicted total and the group rate. This is why averaging the \(\hat{\lambda}_i\) values without weights generally gives the wrong benchmark.

Discrepancies and Poisson Count Variation

A useful table reports \(O_g\), \(M_g\), \(E_g\), the observed and predicted group rates, and a signed discrepancy such as \(O_g-M_g\). A positive discrepancy means more events were observed than predicted; a negative one means fewer. This signed information matters: a large discrepancy in either direction is a potential calibration concern, while a sequence of positive discrepancies across higher-prediction groups may suggest a systematic pattern.

Raw discrepancies are not comparable across groups of very different sizes. If a group is expected to have many events, its ordinary count fluctuations are larger in absolute terms. Under the Poisson model with a fixed known mean \(M_g>0\), a standardized residual is

The denominator is the Poisson standard deviation of the group count under calibration. It gives a scale for judging a discrepancy against ordinary count variation; it does not, by itself, provide a calibrated significance test when predictions are estimated from the same data.

For the first two statements, \(E[R_g]=(E[O_g]-M_g)/\sqrt{M_g}=0\), and \(\operatorname{Var}(R_g)=\operatorname{Var}(O_g)/M_g=M_g/M_g=1\). For the Pearson contribution,

The identity uses the general decomposition \(E[(X-a)^2]=\operatorname{Var}(X)+(E[X]-a)^2\); under the stated Poisson calibration assumptions, the second term is zero. \(\square\)

This result is about repeated sampling under a specified mean, not a rule that every observed contribution should equal one. For fixed \(M_g\), \(R_g\) is not generally exactly normal, and its squared value is not exactly chi-squared with one degree of freedom. When \(M_g\) is large, the Poisson central limit theorem gives an approximately standard normal \(R_g\); with a fixed number of independent groups whose means grow, the sum of squared standardized residuals is approximately chi-squared with one degree of freedom per group. These approximations need not be reliable for small expected counts.

Calibration Is Not the Same as Count Variability

Calibration concerns the mean: does the expected count agree with the predicted count? Poisson variability concerns the distribution around that mean. Under the Poisson model, a group with predicted count \(M_g\) has variance \(M_g\); individual realizations will therefore differ from \(M_g\), even when its mean is correctly specified. A single observed count above prediction is not, on its own, proof of mean miscalibration.

The distinction also works in the other direction. A model can have correct group means but counts that vary more than the Poisson assumption allows. For example, if the true variance is approximately \(2M_g\) while the mean remains \(M_g\), the usual residual denominator \(\sqrt{M_g}\) understates the variability. With \(M_g=100\) and \(O_g=120\), the Poisson-standardized residual is \(20/\sqrt{100}=2\); using a variance of \(200\) gives \(20/\sqrt{200}\approx1.41\). This arithmetic illustrates the effect of a larger variance; it does not show that this variance pattern is present in any particular data set.

Interpreting a Grouped Check

A grouped table or plot is a diagnostic summary, not a complete proof of model adequacy. It can reveal systematic discrepancies across ranges of prediction, but grouping sacrifices detail: different individual predictions and outcomes are compressed into each bin. Calibration of a group total does not imply that each observation in the group has a correct mean prediction.

A simple reference calculation based on fixed \(M_g\) also omits the fact that GLM predictions are usually estimated. When the same data are used both to fit the model and to assess its calibration, observed counts and fitted means are statistically dependent. The groups may themselves be determined by the fitted predictions. Consequently, the fixed-mean Poisson residual calculations describe a benchmark, not an automatic reference distribution for a formal test. Predictions evaluated on independent data are closer to the fixed-prediction setting, although uncertainty and the design still matter.

Use the check to describe the direction and scale of discrepancies, always alongside exposure and predicted counts. If the mean pattern looks poor, that is different from a concern that counts fluctuate more or less than the Poisson model specifies. Keeping these questions separate prevents ordinary random variation from being mislabeled as systematic mean error, and prevents a correct overall total from concealing calibration differences across prediction groups.