Why Check the Upper Tail?
A count model can reproduce the typical outcome while failing to reproduce its largest observations. For example, it may fit the many zeros and ones reasonably well but generate too few counts of five or more. Or it may capture individual observations while underpredicting the total number of events in the most affected clusters. Such discrepancies matter when decisions depend on unusually high burdens, not just average outcomes.
A tail-count check makes this concern explicit: choose a threshold and count how many observations reach or exceed it. An extreme-cluster check instead summarizes the largest cluster totals, or counts clusters whose totals exceed a specified level. The observed statistic is then compared with the same statistic calculated from datasets generated under the fitted model. The check asks whether the model can reproduce the chosen kind of extreme—not whether a particular observation is intrinsically anomalous.
Model and Reference Distribution
Suppose the fitted GLMM specifies a distribution for each outcome conditional on covariates and cluster random effects, together with a distribution for those random effects. A model-generated dataset is produced by drawing outcomes according to that specification, with the relevant design features—such as cluster sizes, covariates, and exposures—kept aligned with the observed data. The statistic must be calculated the same way for the observed dataset and every generated dataset.
The check is meaningful only relative to its replication target. For example, generating new cluster effects addresses whether the model reproduces variation across new clusters; retaining fitted cluster effects addresses a different, within-cluster question. The choice should match the diagnostic question and be stated when reporting results. This check does not itself determine which replication target is appropriate.
The usual model assumptions still matter. The specified conditional outcome distributions and random-effects distribution must be adequate for the process being represented, and the dependence structure implied by the GLMM must be appropriate. For a conventional GLMM, outcomes are conditionally independent given the random effects and covariates; marginally, observations in the same cluster can be dependent because they share random effects. If that dependence or the outcome family is wrong, a tail discrepancy may flag model inadequacy without identifying its cause.
Worked Example: Counting Large Individual Outcomes
Suppose a fitted count GLMM is assessed on 60 observations. In the observed data, 8 observations have counts at least 4, so \(T_4(Y_{\mathrm{obs}})=8\). Generate \(B=200\) datasets under the stated model and compute \(T_4\) for each. The simulated frequency table is:
| Replicated tail count | Number of datasets |
|---|---|
| 0 | 26 |
| 1 | 40 |
| 2 | 48 |
| 3 | 36 |
| 4 | 22 |
| 5 | 13 |
| 6 | 7 |
| 7 | 3 |
| 8 or more | 5 |
The empirical upper-tail proportion is the fraction of generated statistics at least as large as observed: $$ \widehat{q}=\frac{\#\{b:T_4(Y^{(b)})\geq 8\}}{B} =\frac{5}{200}=0.025. $$
Thus, only 2.5% of these model-generated datasets had at least 8 observations with counts of 4 or more. This is evidence that the fitted model may underpredict this particular kind of upper-tail frequency. It is not a 2.5% probability that the model is correct, nor a formal test with a universally calibrated significance level. With only 200 replications, the Monte Carlo standard error of this estimated proportion is approximately $$ \sqrt{\frac{0.025(1-0.025)}{200}}=0.0110. $$ The estimate is therefore imprecise; more replications would help distinguish a genuinely small reference proportion from simulation noise.
Why Cluster Dependence Changes the Check
The variability of an upper-tail count depends on the dependence among its indicator terms. Write \(Z_{ij}=I(Y_{ij}\geq u)\). By linearity of expectation, $$ E(T_u)=\sum_{j=1}^{J}\sum_{i=1}^{n_j}P(Y_{ij}\geq u). $$ This holds whether or not observations are independent. For the variance, expanding the variance of a sum gives $$ \operatorname{Var}(T_u)=\sum_{j,i}\operatorname{Var}(Z_{ij}) +2\sum_{(j,i)<(k,\ell)}\operatorname{Cov}(Z_{ij},Z_{k\ell}). $$ Since \(Z_{ij}\) is an indicator, \(\operatorname{Var}(Z_{ij})=P(Y_{ij}\geq u)\{1-P(Y_{ij}\geq u)\}\). The covariance terms need not vanish: observations sharing random effects can have positively associated tail indicators. Consequently, a reference distribution built from independent draws that ignore clusters can be too narrow, making an observed tail count look more surprising than it should. Generating complete datasets under the fitted GLMM preserves the dependence the model specifies.
This justification also explains why an expected number of exceedances is not enough. Two models can have the same expected \(T_u\), yet differ in its variance and therefore in the frequency of unusually large tail counts. Comparing the full simulated distribution of the chosen statistic captures more than comparing its mean alone.
Worked Example: Checking the Largest Cluster Total
Consider 12 clusters. Their observed outcome totals are \(3,5,4,6,2,7,5,4,6,3,16,5\). The observed maximum is therefore \(M(Y_{\mathrm{obs}})=16\). Generate 100 datasets with the same cluster structure under the fitted model, and record the maximum cluster total for each. Suppose the maxima fall into these groups:
| Maximum cluster total | Number of datasets |
|---|---|
| 11 or less | 23 |
| 12 | 26 |
| 13 | 24 |
| 14 | 15 |
| 15 | 10 |
| 16 | 1 |
| 17 | 1 |
There are \(1+1=2\) generated maxima at least as large as 16. The empirical reference proportion is \(2/100=0.020\). In this set of replications, maxima as large as the observed maximum were uncommon. That finding points to possible underprediction of the most extreme cluster total; it does not establish whether the problem lies in the random-effects distribution, the conditional outcome model, an omitted predictor, or another feature.
The statistic uses the maximum and therefore discards information about the other 11 totals. A complementary check could count clusters exceeding a prespecified level. For instance, if \(c=10\), the observed data have \(C_{10}=1\), because only one of the 12 totals is at least 10. A useful threshold should be chosen for a substantive or diagnostic reason, not selected after inspecting which cutoff makes the discrepancy look largest.
Reading the Result and Diagnosing Failure
A small empirical upper-tail proportion means that the observed discrepancy statistic lies in the high end of the model-generated reference distribution. For an upper-tail count or a maximum, this is the direction expected when the model produces too few large observations or cluster totals. A proportion near one-half generally indicates that the observed statistic is not unusual relative to those replications, but it does not prove the model is adequate. A large proportion near one may indicate that the model generates even more extreme values than observed.
Check the analysis before interpreting a small proportion as evidence against the model:
- Confirm that the threshold, cluster definition, and handling of unequal cluster sizes match the observed and generated datasets.
- Verify that the simulated datasets reproduce the intended design and dependence structure. Generating independent outcomes when the fitted GLMM has shared random effects is not an equivalent check.
- Inspect the number of replications and the Monte Carlo uncertainty. A handful of exceedances among few simulations does not locate a reference quantile precisely.
- Consider whether a few influential clusters dominate the maximum. Compare a maximum check with a count of clusters beyond a meaningful cutoff, while recognizing that these are distinct statistics.
- Look at the selected threshold in context. A check at a very high threshold may be driven by one event; a low threshold may mostly describe common outcomes rather than extremes.
Tail-count checks complement, rather than replace, earlier residual displays and dispersion summaries. Pearson dispersion statistics and deviance-based summaries compress discrepancies across many observations into a single measure; a tail statistic targets a specific feature those summaries may obscure. Residual plots can show covariate patterns or isolated observations, while a tail check asks whether the fitted model generates the observed frequency or severity of extremes. Neither approach identifies a cause automatically. Comparing several prespecified, substantively meaningful checks can help characterize a problem, but searching many thresholds and reporting only the most discrepant one exaggerates apparent evidence.
Worked Example: Comparing Two Tail Thresholds
For 40 observed counts, suppose 17 are at least 2 and 6 are at least 4. In 200 model-generated datasets, the number with \(T_2\geq17\) is 90, while the number with \(T_4\geq6\) is 8. The two empirical proportions are $$ \widehat{q}_2=\frac{90}{200}=0.450,\qquad \widehat{q}_4=\frac{8}{200}=0.040. $$
The model-generated datasets often reproduce at least 17 observations of 2 or more, but only 4.0% reproduce at least 6 observations of 4 or more. The evidence therefore points more specifically to a deficit in the more extreme counts than to a general shortage of all counts above 1. This conclusion is conditional on the selected thresholds and replication scheme; it does not mean the model has been shown correct below 4 or that the threshold-4 result is a calibrated hypothesis test.
Scope and Limitations
These checks are targeted diagnostics: their strength is also their limitation. They can reveal that a model misses a chosen upper-tail feature, but cannot by themselves distinguish a wrong random-effects distribution from a wrong conditional variance, an omitted covariate, or another source of misspecification. A small empirical proportion is not proof of overdispersion in every sense; it is evidence of mismatch for the statistic and replication target used.
Parameter estimates are obtained from the same data being checked in many practical workflows. As a result, a plug-in reference distribution is not automatically a formal sampling distribution for a test statistic, and the empirical tail proportion should not be interpreted as a conventional p-value. The number of replications controls simulation precision, not uncertainty from every modeling choice. Report the statistic, threshold, observed value, replication target, number of generated datasets, and the tail proportion together.
A disciplined check begins with a question, not a search over cutoffs: for example, “Does this model reproduce the number of clusters with totals above a clinically relevant level?” Define that statistic in advance when possible, compare it with its model-generated distribution, and use other diagnostics to investigate any discrepancy.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If the observed upper-tail count is 9 and 12 of 300 generated datasets have a count at least 9, what empirical upper-tail proportion is estimated?
- Why can shared random effects change the variability of an upper-tail count even when its expected value is unchanged?
- Suppose the observed maximum cluster total is 14 and 30 of 200 generated maxima are at least 14. Interpret the result without calling it a formal p-value.
- Why should the threshold for a tail-count check not be chosen solely because it gives the smallest empirical proportion?
Solutions. (1) The proportion is \(12/300=0.040\), or 4.0%; it describes the fraction of generated datasets with a statistic at least as large as observed. (2) The variance includes covariance terms between tail indicators. Shared random effects can make exceedances within a cluster dependent, so those covariance terms need not be zero. (3) The proportion is \(30/200=0.150\). The observed maximum is not especially rare in these replications; this check gives limited evidence that the model underpredicts maxima of at least 14. It does not prove adequacy. (4) Trying many thresholds and selecting the most discrepant result makes the apparent evidence more extreme than a prespecified check would suggest. Select thresholds for substantive or diagnostic reasons and report the choice.