Begin With the Diagnostic Question
A predictive check is only informative if its summary targets a feature of the data that could reveal the model's weakness. A check for the total number of zeros asks a different question from one for extreme counts; a check of overall fit can also conceal a systematic mismatch in one covariate range or cluster. The choice of discrepancy statistic therefore comes before interpreting whether the observed data look unusual among replicated data.
The setting here is a fitted GLMM for outcomes \(Y_{ij}\), where \(j=1,\ldots,J\) indexes clusters and \(i=1,\ldots,n_j\) indexes observations within a cluster. The observed outcomes are \(y_{ij}\), and \(y^{\mathrm{rep}}\) denotes a replicated data set generated under the fitted model using the selected replication scheme. The design information and the statistic's definition must be held consistent across the observed and replicated data.
For example, if the question is whether the model reproduces the number of zero outcomes, a natural statistic counts zeros. If the question concerns very large counts, the statistic should instead count observations above a prespecified threshold. Neither statistic is a universal measure of fit: each is aimed at a particular feature. The earlier material on comparing observed and replicated outcome distributions motivates graphical comparisons; here the emphasis is on choosing a scalar summary that makes a diagnostic question explicit.
Simulation Reference and Interpretation
Suppose \(B\) replicated data sets have been generated and \(T_{\mathrm{obs}}=T(y)\) is the observed statistic. For a statistic where larger values indicate more of the feature of concern, count the replicates at least as large as observed. A finite-simulation estimate with a small-sample correction is
Here \(I\{\cdot\}\) equals 1 when its condition is true and 0 otherwise. The added 1 prevents a reported probability of exactly zero when no simulated statistic reaches the observed value. This tail proportion describes how often the chosen statistic is at least as extreme in the specified direction under the simulation reference. It is a diagnostic summary, not automatically a calibrated hypothesis-test p-value: fitted parameters, the replication scheme, and the choice of statistic affect its behavior.
A small upper-tail proportion is evidence that the model, under the selected replication scheme, often produces fewer of the targeted feature than were observed. A large proportion is not proof of adequate fit; the statistic may simply be insensitive to another kind of misfit. If unusually small values are also concerning, define a two-sided discrepancy around an appropriate reference center, such as a distance from that center, rather than mechanically doubling a one-sided proportion. The center and direction should follow the diagnostic question.
Worked Examples: Four Targeted Questions
Worked Example: Is the Model Reproducing the Number of Zeros?
Suppose a hypothetical sparse-count data set contains 60 observations, of which 18 are zero. The diagnostic question is whether the fitted model reproduces the overall number of zero outcomes. Define \(T_0(y)=\sum_{j,i} I(y_{ij}=0)\). The number of zero outcomes is an integer count, and larger values directly mean more zeros; thus this statistic matches the question.
Across \(B=500\) replicated data sets, suppose 3 have at least 18 zeros. Then
Under this simulation reference, so few replicates have as many zeros as the observed data that the fitted model appears to underproduce zeros overall. This is a warning about that feature, not proof of a particular alternative mechanism. For instance, this statistic alone cannot tell whether the mismatch is concentrated in a subgroup or linked to another form of variation.
Worked Example: Are Extreme Counts Too Frequent?
In a hypothetical study of 80 count outcomes, define an extreme count in advance as a value of at least 5. The observed number of such outcomes is 9, so use \(T_5(y)=\sum_{j,i} I(y_{ij}\geq 5)\). This asks about the frequency of counts in the specified upper tail, not about the largest count alone.
Suppose \(B=199\) replicated data sets are generated and 8 have at least 9 outcomes of 5 or more. The calculation is
The observed tail count is near the upper edge of this simulated reference: 9 of 200, after correction, is the estimated upper-tail proportion. This points to possible underproduction of counts at least 5. The threshold must have a meaningful justification, such as an operational definition of a large count, and should not be repeatedly adjusted until the observed result looks unusual. If several thresholds are examined, their results are a set of related diagnostics, not independent formal tests.
Worked Example: Does Any Comparable Cluster Have an Unusually Large Total?
Consider a hypothetical data set with 12 clusters, each having the same number of observations, equal exposure, and a comparable covariate profile. Their observed cluster totals are 6, 7, 8, 9, 10, 10, 11, 11, 12, 13, 14, and 16. A targeted question is whether the largest cluster total is unusually large. Define \(T_{\max}(y)=\max_j\sum_i y_{ij}\). The common cluster sizes, exposures, and profiles make a raw maximum a defensible summary for this example; without that comparability, it could mostly reflect cluster size or expected rate.
The observed maximum is \(T_{\max,\mathrm{obs}}=16\). Suppose 7 of \(B=499\) replicated data sets have a maximum of at least 16. Then
The observed largest total is rarely reached in these replications, suggesting that the model may not reproduce the most extreme cluster total. This statistic does not say that all clusters fit poorly, nor does it identify the cluster-level cause. If cluster sizes or exposures differ substantially, a raw maximum can answer the wrong question; a cluster-specific standardized discrepancy or a statistic accounting for expected totals may be more appropriate.
Worked Example: Is Fit Systematically Poor in a Covariate Group?
Suppose a hypothetical binary-outcome GLMM is being checked across two covariate groups defined before examining outcomes. In the group with higher exposure, there are 40 observations and 12 observed events. The fitted model's expected event count in that group, calculated by summing its predicted probabilities, is 7.2. A direct discrepancy for excess events in this group is \(T_g(y)=\sum_{(i,j)\in g} y_{ij}\). The difference between observed and expected counts is \(12-7.2=4.8\) events, but whether that difference is unusual depends on the simulated reference distribution.
Suppose \(B=399\) replicated data sets are generated and 11 have at least 12 events in the same prespecified group. Then
This indicates that the observed group event count is relatively high compared with the replications. The group definition must be applied identically to every replicate. If the goal is to compare groups with very different expected counts or uncertainty, a standardized residual summary can be more informative than a raw count difference. Grouping should also be substantively defensible: data-driven cut points chosen after inspecting the outcome can exaggerate apparent discrepancies.
Why the Statistic Must Match the Question
The basic justification is that a statistic compresses the data to the feature relevant to the diagnostic question. Let \(\mathcal{A}\) be an event such as “at least 18 zero outcomes.” For the zero-count statistic, \(T_0(y)\), the event can be written exactly as \(T_0(y)\geq18\). The simulated frequency of that event estimates how often the fitted model produces the observed-or-larger zero count. The calculation is useful precisely because its event has a direct interpretation.
The approximation follows because each indicator is 1 when a replicate reaches the event and 0 otherwise; their average is the simulated proportion of replicates that reach it. Under independent replications from the chosen simulation reference, this proportion converges to the reference probability as \(B\) increases. This justifies using a simulation tail proportion for the chosen event. It does not guarantee that the event detects unrelated forms of lack of fit.
A broad statistic can be poor for a narrow question. For instance, comparing overall means may conceal excess zeros accompanied by a similar mean, because more zeros and larger positive counts can offset each other. Conversely, counting zeros cannot reveal whether the positive-count tail is too heavy. Different statistics can reveal different features even when they are calculated from the same replicated data.
Practical Selection, Diagnostics, and Limitations
A defensible selection process makes the target and the scale explicit. The following sequence is useful:
Specify a feature such as the number of zeros, frequency above a threshold, largest cluster total, or event rate in a prespecified covariate group.
Use a count, maximum, contrast, or standardized summary whose direction and units represent that feature.
Ask whether cluster size, exposure, or covariate composition makes a raw statistic misleading. Standardize or define a more appropriate contrast when needed.
Calculate the same statistic with the same thresholds, groups, and scale for the observed data and every replicated data set.
Report what feature appears unusual under the selected simulation reference, and inspect complementary diagnostics before drawing broader conclusions.
Useful warning signs include a statistic that changes meaning across clusters, a threshold chosen only after seeing the observed tail, and a result dominated by a single observation when the stated question concerns a general pattern. Also check that the simulation produces values on the same scale as the observed statistic, that the covariate groups are applied consistently, and that the number of replicates is sufficient for the desired numerical resolution. With \(B=99\), for example, the corrected estimate cannot be smaller than \(1/100\); a smaller tail probability cannot be resolved from so few replications.
The main alternatives answer different questions. A graphical comparison of full distributions can expose shape differences that a single statistic misses, but may not summarize a prespecified feature as directly. Residual plots can reveal patterns across fitted values, covariates, or within clusters, while a simulation discrepancy focuses on how a selected summary compares with replicated data. A single omnibus summary may be convenient but can be insensitive to a localized problem; a collection of targeted summaries is more diagnostic, though interpreting many results creates a risk of overemphasizing chance extremes. These approaches complement one another rather than replacing one another.
No discrepancy statistic establishes that a model is correct. A non-extreme simulated value means only that the chosen summary is not unusual in the chosen direction under the specified replication reference. A very extreme value flags a mismatch worth investigating but does not identify its cause. The replication scheme itself matters, as do estimation, dependence within clusters, the number of simulated data sets, and whether the statistic was selected before or after inspecting the results. These checks are diagnostic evidence, not a substitute for substantive model assessment.
Common Pitfall
Correction: A mean is only one discrepancy statistic and can conceal compensating differences. Choose separate, clearly stated statistics for zeros, tail frequency, cluster-level behavior, and covariate-specific fit when those are the diagnostic questions. Interpret each result only for the feature it measures.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In a data set with 50 counts, 15 are zero. Of 299 replicated data sets, 4 have at least 15 zeros. Calculate the corrected upper-tail proportion and state its diagnostic meaning.
- Why might the maximum raw cluster total be a poor discrepancy when clusters have very different sizes?
- A researcher defines a covariate group after observing that it contains an unusually large number of events. What is the concern, and what is a better practice?
- What does a non-extreme simulated discrepancy proportion establish, and what does it not establish?
Solutions. (1) The corrected estimate is \((1+4)/(299+1)=5/300=0.01667\). The observed zero count is relatively high under this reference, suggesting a possible shortfall in reproduced zeros; this is a targeted diagnostic, not a formal conclusion about a specific cause. (2) Larger clusters have more opportunities to accumulate outcomes, so the maximum may reflect cluster size rather than an unusual rate or within-cluster pattern. Use a statistic accounting for size, exposure, or expected totals, or compare suitably standardized cluster summaries. (3) Selecting the group because its outcome looks unusual can make the discrepancy appear more compelling than it would under a prespecified diagnostic. Define scientifically meaningful groups in advance, apply them unchanged to replicates, and treat exploratory selection as exploratory. (4) It establishes only that the chosen statistic is not especially unusual in the selected direction under the specified simulation reference. It does not establish that the model is correct or that other features fit adequately.