Why Rare-Event Calibration Needs Care
A model can predict a low event probability for most observations and still be poorly calibrated. With rare outcomes, a small number of events can move an observed group rate substantially, while a group with no events may be entirely plausible even when its predicted event count is positive. A useful check therefore compares observed and predicted frequencies alongside their uncertainty, rather than treating every difference as evidence of model failure.
For a generalized linear mixed model (GLMM), the comparison also has to match the prediction target. A prediction conditional on an estimated cluster effect concerns observations in that particular cluster; a prediction marginalized over random effects concerns the population represented by the random-effects distribution. As in the earlier material on Residuals Marginalized Over Random Effects, these are distinct quantities. Use the one that corresponds to the calibration question, and state it.
The groups are often risk bands—for example, low, medium, and high predicted risk. Define them using predictions, not observed outcomes, and report how many observations and events each contains. The mean predicted risk is preferable to a single representative prediction when risks vary within a group: summing the probabilities gives the model’s expected number of events in that group.
Uncertainty in a Grouped Frequency
If outcomes in a group are independent Bernoulli observations with a common event probability, then \(O_g\) has a binomial distribution with size \(n_g\). A Wilson interval is a useful interval for that group’s event probability. For observed rate \(\hat r=O_g/n_g\), its two-sided \(100(1-\alpha)\%\) Wilson interval is
where \(z\) is the standard normal quantile for the chosen confidence level; for a 95% interval, \(z\approx1.96\). Comparing \(\bar p_g\) with this interval is a visual or descriptive check, not a test that proves calibration. The interval describes uncertainty in the observed rate under its sampling assumptions; it does not give a 95% probability that the fixed predicted risk lies inside it.
The common-probability, independent-outcome assumptions may not hold in a GLMM. Even when the probabilities differ within a group, independent outcomes have a Poisson-binomial count distribution, not exactly a binomial one. More importantly, outcomes sharing a random effect can be dependent after that effect is integrated out. A simple binomial interval can then understate uncertainty. For a grouped check where this matters, simulate counts from the fitted GLMM while preserving its cluster structure.
Worked Example: Comparing Risk-Band Rates With Intervals
Suppose a hypothetical GLMM is used to predict a rare binary outcome for 300 patients. For this illustration, each risk band contains 100 independent patients with a common predicted risk within the band. The table gives the predicted risk and observed event count.
| Risk band | Predicted risk | Expected events \(E_g\) | Observed events \(O_g\) | Observed rate | Approximate 95% Wilson interval |
|---|---|---|---|---|---|
| Low | 2% | 2 | 3 | 3% | 1.03% to 8.45% |
| Medium | 6% | 6 | 4 | 4% | 1.57% to 9.84% |
| High | 12% | 12 | 14 | 14% | 8.53% to 22.14% |
For the low-risk band, \(E_g=100(0.02)=2\), while \(\hat r_g=3/100=0.03\). Substituting \(n_g=100\), \(\hat r=0.03\), and \(z=1.96\) into the Wilson formula gives approximately 0.0103 to 0.0845, or 1.03% to 8.45%. The 2% predicted risk lies within that interval.
For the medium band, \(E_g=100(0.06)=6\) and \(\hat r_g=4/100=0.04\); its interval is approximately 1.57% to 9.84%, which contains the 6% prediction. For the high band, \(E_g=100(0.12)=12\) and \(\hat r_g=14/100=0.14\); its interval is approximately 8.53% to 22.14%, which contains the 12% prediction. These broad intervals are consistent with the predictions, but they do not establish that calibration is exact. The calculation is illustrative: if patients are clustered or risks vary substantially within a band, a model-based simulation is more appropriate.
Why Cluster-Aware Simulation Helps
Under independent Bernoulli outcomes with fixed probabilities, the variance of a group count is \(\sum_{i\in G_g}\hat p_i(1-\hat p_i)\). With shared random effects, outcomes in the same cluster can move together. For any group count, the variance includes covariances as well:
This identity follows by expanding the variance of a sum: each individual contributes its variance, and each distinct pair contributes twice its covariance. Ignoring positive within-cluster covariance can therefore make an observed count look more surprising than it should.
A practical simulation check fixes the observed covariates and group assignments, generates outcomes under the fitted GLMM, and recalculates the grouped counts for each simulated data set. For a population-level prediction check, the simulations should reflect the random-effects variation relevant to that target, including new random effects where appropriate. Compare the observed group counts with the simulated distributions, and use a joint discrepancy if the question concerns whether the pattern across groups is plausible. This is a plug-in check when fitted parameters are held fixed; it does not automatically include all parameter-estimation uncertainty.
Worked Example: A Cluster-Aware Simulation Check
Consider a hypothetical study with 20 clinics and two risk bands. Each clinic contributes five observations to each band, so each band has 100 observations. A fitted random-intercept GLMM gives marginal expected counts of 2.1 events in the low-risk band and 8.4 in the higher-risk band. The observed counts are 1 and 12. Because outcomes within a clinic may be associated, suppose 5,000 simulated data sets are generated from the fitted model, preserving the clinic memberships and risk-band assignments.
The simulated 2.5th and 97.5th percentiles for the low-risk count are 0 and 6; for the higher-risk count they are 2 and 17. The observed counts, 1 and 12, fall inside their respective simulation intervals. These are predictive ranges for counts under the fitted model, not confidence intervals for a fixed event probability.
For a joint summary, suppose the simulated standard deviations of the two counts are 1.8 and 4.2. A standardized squared discrepancy for the observed data is
Suppose 900 of the 5,000 simulated data sets have a discrepancy at least this large. The estimated simulation tail proportion is \(900/5000=0.180\). Its Monte Carlo standard error is approximately \(\sqrt{0.18(1-0.18)/5000}=0.00543\). This result does not show an unusually large discrepancy under the fitted model. It is not proof of calibration: it says only that this chosen summary is not especially extreme in these simulations. Different calibration failures may require different groupings or summaries.
Overall Agreement Can Conceal Group Miscalibration
Summing over all risk groups can be a useful calibration-in-the-large check, but it can hide errors in opposite directions. An overprediction in one group may cancel underprediction in another. Grouped comparisons are therefore valuable even when the overall observed count is close to the total expected count.
Worked Example: Equal Totals, Different Risk-Group Patterns
In another hypothetical data set, two risk bands each contain 100 observations. The model predicts risks of 8% and 2%, so the expected counts are \(100(0.08)=8\) and \(100(0.02)=2\). The observed counts are 2 in the first band and 8 in the second.
Across all 200 observations, the observed total is \(2+8=10\), and the predicted total is \(8+2=10\). Both overall rates equal \(10/200=0.05\), or 5%. However, in the first band the observed rate is \(2/100=2\%\), well below the 8% prediction. In the second band the observed rate is \(8/100=8\%\), well above the 2% prediction. The Wilson interval for 2 events in 100 is approximately 0.55% to 7.00%; for 8 in 100 it is approximately 4.10% to 15.00%. The first band’s 8% prediction is above its interval, and the second band’s 2% prediction is below its interval. The opposing discrepancies cancel in the overall total but remain visible by risk group.
Diagnostics, Alternatives, and Limitations
A useful calibration display plots observed group rates against mean predicted risks, with uncertainty intervals and a reference line representing equality. Also report group sizes, event counts, and expected counts: a rate alone can be misleading when it is based on only one or two events. Examine whether discrepancies persist across adjacent groups rather than interpreting a single sparse band in isolation.
- Groups with zero events: a zero observed rate is not automatically evidence that predicted risk should be zero. Compare the count with its uncertainty or simulated predictive distribution.
- Overly fine grouping: many narrow groups produce unstable rates and intervals, especially when events are rare. Too few broad groups can conceal systematic patterns. Show the grouping rule and consider whether conclusions change under reasonable alternatives.
- Ignoring clusters: treating all observations as independent can produce intervals that are too narrow. Preserve cluster membership in model-based simulations when dependence matters.
- Using estimated conditional predictions without a clear target: apparent agreement within observed clusters does not by itself establish calibration for new clusters. State whether predictions are conditional or marginalized.
- Interpreting a non-extreme check as confirmation: sparse data may have little power to reveal miscalibration. Failure to detect a discrepancy is not evidence that all relevant calibration errors are absent.
Grouped frequencies are interpretable but depend on how groups are formed. A smooth calibration curve can use the predictions more continuously, but its appearance also depends on smoothing choices and its uncertainty must respect clustering. A single overall comparison of observed and expected totals is simpler, but it cannot reveal offsetting group errors. Simulation can account for the fitted model’s outcome and cluster structure, while a plug-in simulation may still omit parameter uncertainty. These tools complement one another; none makes sparse events informative when the data contain too little information.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A group has 80 observations, each with predicted risk 0.025. What is its expected event count?
- A group has 100 observations and 2 observed events. Why should its 2% rate not be treated as a precise estimate of its event probability?
- Why can an ordinary binomial interval be too narrow for observations clustered within clinics?
- Two groups have observed counts 1 and 9 and expected counts 9 and 1. What does the overall comparison conceal?
- In a simulation check, 120 of 2,000 replicates have a discrepancy at least as large as observed. Calculate the estimated tail proportion and state its interpretation.
Solutions. (1) The expected count is \(80(0.025)=2\) events. (2) With only two events, sampling uncertainty is substantial; a point estimate alone does not show how much variation is plausible. (3) Shared clinic effects induce dependence, so the variance of the total includes positive pairwise covariances that an independent binomial interval omits. (4) Both totals are 10, so an overall comparison can appear exact, while each group has a discrepancy in the opposite direction; the group-specific pattern must be examined. (5) The estimated tail proportion is \(120/2000=0.060\). Under the simulated fitted model, about 6% of replicates had a discrepancy at least this large for the chosen statistic; this is not the probability that the model is true.