Calibration Checks for Sparse Counts

Why Check Calibration by Groups?

A count model can give reasonable predictions overall while systematically underpredicting some observations and overpredicting others. Comparing a single observed total with a single predicted total may hide that pattern. A grouped calibration check asks a more targeted question: among observations with similar fitted means, do the observed counts add up to roughly what the fitted GLMM predicts?

This check is especially useful for sparse counts. Individual observations may be zero most of the time, and a residual for one observation can be difficult to interpret. Aggregating observations into groups can make a pattern visible. But aggregation does not make the observations independent: the model’s cluster structure still matters when deciding whether a discrepancy is unusually large.

Definitions and the Prediction Target

Write \(Y_{ij}\) for the count for observation \(i\) in cluster \(j\). Let \(\hat{\mu}_{ij}\) be its fitted expected count under the prediction target being checked. For example, a check may assess predictions for the observed clusters or predictions intended to average over the random-effects distribution. The target must be chosen explicitly, and the fitted means and simulations must represent the same target.

The group expected total is a sum of fitted means, not an average of them. If a group contains many observations with small expected counts, its total expected count can still be substantial. Groups should be formed from fitted means, not from observed counts: sorting observations by their outcomes and then comparing totals would use the very information the check is meant to assess.

There is no universal best number of groups. Too few groups can hide a trend across the fitted-mean range; too many can leave unstable totals, particularly when expected counts are very small. Choose a grouping rule that leaves enough observations and expected count in each group to make the summaries interpretable. Report the cut points and group sizes so the check can be reproduced.

Worked Totals and What They Do Not Show

Why the GLMM Structure Must Be Preserved

The variability of a group total depends on more than its expected total. For a group with observations indexed by \(k\), the variance of its total satisfies

The second sum contains the within-group covariances. In a GLMM, observations in the same cluster may be dependent through their shared random effects. Treating every count as an independent Poisson variable would generally omit this dependence, so it could give the wrong variability for the group totals even if the expected totals were correct.

A simulation check uses the fitted GLMM to generate replicated count data while retaining the observed design: covariates, offsets or exposure values, cluster membership, and the count model’s structure. Each replicate must be generated according to the prediction target being checked. Keep the grouping rule fixed as specified for the check; if the rule itself is data-adaptive, apply that same rule in each replicate. The point is not to simulate a collection of unrelated counts, but to ask what grouped totals the fitted model would typically produce for this design.

One useful discrepancy is the largest standardized departure across groups. Let \(D_g^{(b)}=O_g^{(b)}-E_g\) be the discrepancy in replicate \(b\), calculated against the original fitted expected total. From the \(B\) replicates, calculate the simulated mean \(\bar{D}_g^*\) and standard deviation \(s_g^*\) for each group. Then define

For each replicate, calculate \(T_b\) using the same simulated centers and standard deviations. A large \(T_{\mathrm{obs}}\) relative to the replicated \(T_b\) values indicates that at least one group departs more than is typical under the fitted model. The maximum statistic accounts for looking across the groups as a set; simply reporting the most unusual group without considering that search can exaggerate the evidence.

Worked Simulation Check

A Short Justification for Simulation-Based Scaling

If a group’s replicate discrepancies have simulated mean \(\bar{D}_g^*\) and standard deviation \(s_g^*\), the standardized observed discrepancy compares its departure from the model’s simulated center with the variability the model produces for that group. This is preferable to dividing every discrepancy by the same number: groups with different sizes and expected counts generally have different scales of variation.

The maximum over groups then summarizes whether any group is unusually far from its simulated center. To see why the replication step matters, write a group total as \(S_g=\sum_{k\in G_g}Y_k\). Its variance includes both the individual variances and the pairwise covariances shown above. Replicated data generated under the fitted GLMM reproduce those terms through the model’s joint structure. An independent-count shortcut would set the covariance terms to zero, which is not justified when the GLMM implies dependence. This argument does not guarantee a perfect diagnostic: it depends on the fitted model and simulation procedure being appropriate for the intended prediction target.

Worked Example: Unequal Exposure and Group Construction

Reading the Check and Recognizing Failure Modes

A useful calibration display reports, for each fitted-mean group, its cut points, number of observations, observed total, expected total, difference, and often the observed-to-expected ratio. Showing simulated intervals or the simulated distribution of a chosen discrepancy adds the reference needed to judge magnitude. The display should make clear whether the check concerns conditional predictions for the observed clusters or a target averaged over random effects.

Several patterns deserve attention. A steadily increasing positive discrepancy across higher fitted-mean groups can indicate systematic underprediction at the upper end. Persistent negative discrepancies can indicate overprediction. An isolated discrepancy may instead reflect ordinary count variability, a small group, or a few influential observations. Grouped totals can locate a broad calibration problem, but they cannot identify its cause.

• Too many narrow groups: Sparse counts can make each total highly variable and the ratios unstable. Use fewer groups or a grouping rule that leaves interpretable totals, and report how the rule was chosen.
• Grouping on observed counts: This can manufacture apparent differences. Form groups from fitted means, not from the outcomes.
• Ignoring cluster dependence: Independent simulations may produce an inappropriate reference distribution. Retain the fitted clustering structure.
• Changing the prediction target mid-check: Conditional fitted means compared with simulations for a different target do not form a coherent calibration check. State the target and use it throughout.
• Treating a ratio as a test: A ratio can be large when the expected total is tiny. It is descriptive; use a model-based reference distribution to assess whether the discrepancy is surprising.
• Ignoring failed or implausible simulations: Check that replicated data are generated as intended and that group totals are finite and calculable. A faulty simulation pipeline cannot provide a trustworthy benchmark.

Grouped calibration is complementary to residual plots and dispersion summaries. Residual displays preserve observation-level patterns that grouping can hide; an overall dispersion summary compresses variation into a different single-number view. In contrast, this check focuses on whether aggregate counts align with fitted means across the prediction range. Its limits are equally important: results depend on the chosen bins, the fitted model, and the simulation setup. A good grouped check cannot establish that every individual prediction is accurate, and an apparently good check cannot prove the model is correct.