Predictive Coverage at the Cluster Level

Why Cluster-Level Coverage Matters

A GLMM prediction interval for a cluster total answers a different question from an interval for an individual outcome. It concerns the total number of events, successes, or other outcomes in a whole cluster, with the cluster's size and covariates taken into account. Checking whether such intervals cover their corresponding observed totals is therefore a direct way to assess a feature of the model's predictions that individual-level residual summaries may not reveal.

Coverage needs careful interpretation when cluster sizes vary. An interval for a cluster of 40 observations may be much wider than one for a cluster of 8, and the two clusters contribute equally to a cluster-level coverage rate. A size-weighted summary answers a different question and can give a different result. Neither summary should be reported without stating what it weights.

Definition and Conditions for Interpretation

The nominal level is a model-based long-run target, not a promise that exactly that fraction of any finite set of clusters will be covered. Its interpretation depends on what is included in the predictive distribution. For a new cluster, prediction generally needs to include variation from a new random effect as well as outcome variation and, when relevant, parameter uncertainty. For an already observed cluster, a prediction conditional on information about its random effect targets a different quantity. The interval and the validation exercise must match the intended target.

Coverage is most interpretable when the fitted GLMM describes the outcome distribution and dependence structure adequately, the random-effects distribution is appropriate for the target population, and the cluster sizes, covariates, and exposures used for prediction correspond to those for the evaluated clusters. If clusters are dependent in ways not represented by the model, a simple interpretation based on independent cluster evaluations may fail. In replicated checks, the replication scheme must also match the prediction target: conditioning on a fitted random effect does not check prediction for an entirely new cluster.

The first quantity gives every cluster equal weight. The second weights each cluster's coverage indicator by its size. It is a descriptive weighted summary, not the coverage probability for each observation within a cluster: a cluster-total interval either contains the total or it does not.

Why Coverage Estimates Vary

Suppose, for illustration, that evaluations are independent and each cluster has the same probability \(c\) of being covered. Then each \(I_j\) has expectation \(c\), so linearity of expectation gives \(E(\widehat C)=c\). Independence also gives \(\operatorname{Var}(\widehat C)=c(1-c)/J\). Thus a finite-sample rate can differ from \(c\) simply through sampling variation. When coverage differs by cluster size or other features, the common-\(c\) assumption is not appropriate: the overall rate averages potentially different cluster-specific coverage probabilities.

This identity explains why a single overall rate may obscure poor performance for a subgroup of clusters. It also shows why a rate calculated over only a few clusters is imprecise evidence about long-run performance. Coverage should be considered together with the number of evaluated clusters, the pattern across cluster sizes, and the interval widths.

Discrete Intervals and Nominal Coverage

For count or binary-outcome totals, the predictive distribution is discrete. One common construction uses generalized-inverse predictive quantiles: for predictive cumulative distribution function \(F\), define \(Q(p)=\inf\{t:F(t)\geq p\}\), and form an equal-tailed interval \([Q(\alpha/2),Q(1-\alpha/2)]\). Because the endpoints are discrete, the interval's predictive inclusion probability need not equal the nominal level exactly. Under this usual inclusive quantile convention, it is at least the nominal level and may be greater. Empirical coverage on a finite set of clusters can nevertheless fall above or below nominal by chance.

Reading Coverage Patterns and Diagnosing Problems

Coverage is a useful diagnostic, but it is not a complete assessment of predictive quality. Undercoverage means observed totals fall outside intervals more often than the stated target would suggest over repeated evaluations. It may arise because intervals are too narrow, because the model misses variation or dependence, or because the prediction target or covariate information was specified incorrectly. Overcoverage may occur because intervals are very wide, because discrete endpoints include extra probability mass, or because the predictive distribution is conservative. A high rate alone does not guarantee useful predictions.

When sizes vary, inspect coverage separately across meaningful size ranges and, where the number of clusters permits, across other relevant design features. Also inspect which endpoint is missed. Repeated totals above the upper endpoint suggest a different predictive failure from misses split between both tails. Report interval widths in context; a method can attain high coverage by producing intervals too broad to be informative. Conversely, close-to-nominal overall coverage can conceal offsetting undercoverage and overcoverage among different cluster types.

The assumptions for interpreting such patterns should be checked. In particular, verify that each observed cluster total is compared with an interval for that same total and prediction target, that cluster size and exposure are represented as intended, and that uncertainty in a new cluster's random effect is included when predicting a new cluster. If many intervals were constructed from a fitted model using the same clusters being assessed, the assessment can be optimistic because the observations influenced the fit. Evaluations on clusters not used to fit the model offer a more direct assessment of prediction to new clusters; the design and interpretation of those checks require care.

What Coverage Adds—and What It Does Not

Coverage complements calibration checks, comparisons of observed and replicated distributions, and residual-dependence checks. Those tools can reveal features such as a mismatch in the overall distribution or remaining within-cluster structure. Coverage instead summarizes whether a specified interval procedure includes the cluster total at its intended rate. It does not identify the cause of failures, and a satisfactory rate does not prove that the full predictive distribution is correct.

An alternative is to compare point predictions with observed totals or to assess predictive scores that reward both accuracy and distributional sharpness. Those summaries answer different questions: point-error measures do not directly check interval inclusion, while scores do not replace a clear account of coverage. A practical report can pair the nominal level, cluster-weighted empirical rate, number of clusters, size-stratified rates, and representative interval widths. State explicitly whether summaries weight clusters equally or by size.

Common Mistake