Why Diagnostic Reporting Needs More Than a Verdict
A statement such as “the diagnostics looked acceptable” is difficult to evaluate or reproduce. It does not identify which features of the data were examined, what the fitted model was asked to reproduce, or whether the observed pattern was close to the model’s simulated patterns. A useful report makes those elements visible so readers can judge what the checks support and what remains uncertain.
For a generalized linear mixed model (GLMM), diagnostic evidence is especially dependent on the simulation scheme. Replications may hold fitted parameters fixed or account for parameter estimation; they may condition on fitted cluster effects or generate new effects from the estimated random-effects distribution. These choices target different aspects of model adequacy. The report should name the scheme rather than simply say that data were simulated.
What to State About the Simulation
Begin by identifying the fitted model sufficiently for the diagnostic results to be understood: outcome family and link, fixed-effect terms, random-effects structure, and any offset or denominator relevant to the outcome. Then state the assumptions that define the reference simulations. In a typical plug-in parametric replication, the observed covariates, cluster sizes, exposure values, and trial denominators are held fixed. Random effects are generated from the fitted random-effects distribution, and outcomes are generated from the fitted conditional response distribution. The estimated parameters are held at their fitted values.
This procedure assumes that the specified response distribution and link are appropriate conditional on the random effects and covariates, that the random-effects distribution and dependence structure are suitable, and that the simulation preserves the design features being checked. For example, a count model with unequal exposure should use the observed exposure values in every replication. If replicated data do not preserve relevant design features, the comparison may diagnose a mismatch in the simulation setup rather than a defect in the GLMM.
State whether the check is conditional on fitted cluster effects or marginal over simulated effects. A conditional check can focus on response variation after using estimated effects for the observed clusters. A replication that draws new cluster effects examines whether the fitted distribution and model can reproduce cluster-to-cluster patterns. Neither scheme is universally preferable: the report should connect the scheme to the diagnostic question.
Also state the number of replications, the statistics examined, and how reference summaries were formed. If many statistics were inspected, report that fact; a striking result selected from a long, informal search deserves more cautious interpretation than a prespecified check. A simulation envelope is a summary of replicated statistics under the stated scheme. It is not automatically a confidence interval for a model parameter or a guarantee that future observations will fall within its bounds.
Comparing Observed and Replicated Discrepancies
For a diagnostic statistic \(T\), calculate \(T_{\mathrm{obs}}\) from the observed data and \(T^{(b)}\) from each of \(B\) replicated data sets. The statistic should measure the feature of interest—for example, the total number of zeros, variation among cluster-level rates, or concentration of events in a few clusters. Report its observed value with units or a clear definition, alongside the simulated distribution, such as its median and central 95% predictive interval.
When the concern is a discrepancy in one direction, an empirical tail proportion helps describe how unusual the observed statistic is among replications. For an upper-tail check, let \(r\) be the number of replicated statistics at least as large as \(T_{\mathrm{obs}}\). A commonly reported finite-simulation estimate is \((r+1)/(B+1)\). This is a descriptive simulation-based tail measure, not automatically a calibrated hypothesis-test p-value: plug-in parameter estimates, model selection, and inspection of many statistics can affect its interpretation.
The approximate Monte Carlo standard error describes simulation noise in estimating the tail proportion, not uncertainty about the model parameters or the truth of the model. It tends to be largest for probabilities near one half and smaller near zero or one. When very few replications reach the observed value, report the replication count as well as the estimate; an estimate near zero from a finite number of simulations is not evidence that the probability is exactly zero.
Worked Example: Reporting an Excess-Zero Check
Suppose a hypothetical study follows 30 clinics, each with the same number of patients, and records a count outcome. The fitted count GLMM is replicated 1,000 times, holding the observed clinic sizes and covariates fixed while drawing random intercepts and outcomes from the fitted model. Parameters are held at their estimates. The diagnostic statistic is the number of clinic-level observations equal to zero. The observed data contain 96 zeros; the simulations have a median of 74 and a central 95% interval from 61 to 89. Eight of the 1,000 replicated statistics are at least 96.
The observed count is above the simulated interval’s upper endpoint: \(96>89\). The upper-tail estimate, including the finite-simulation adjustment, is \( (8+1)/(1000+1)=9/1001=0.00899 \), or about 0.0090. Using the unadjusted proportion \(\hat{q}=8/1000=0.00800\), its approximate Monte Carlo standard error is \( \sqrt{0.00800(1-0.00800)/1000}=0.00282 \).
A suitable report would say that the observed data had 96 zeros, compared with a replicated median of 74 and a central 95% interval of 61–89; 8 of 1,000 replications had at least 96 zeros. This is evidence that the fitted plug-in model underrepresents zeros under the stated replication scheme. It is not proof of zero inflation: other misspecification, including an inadequate mean structure or random-effects distribution, could also contribute. The Monte Carlo standard error indicates that the estimated tail proportion has appreciable simulation noise relative to its small value.
Worked Reporting Examples
A concise report should preserve the link between the question, statistic, reference simulations, and conclusion. The following examples illustrate different evidence patterns; all data and values are hypothetical.
Worked Example: Cluster-Level Variation Within the Reference Range
In a hypothetical binary-outcome study, 24 clinics each contribute 20 patients. The fitted logistic random-intercept GLMM is replicated 1,000 times with clinic sizes and covariates fixed, and new random intercepts drawn from the fitted distribution. The statistic is the variance of the 24 observed clinic proportions. Its observed value is 0.041 proportion-squared units. The replicated median is 0.028, the central 95% interval is 0.014–0.050, and 178 replications have a statistic at least 0.041.
The observed variance falls within the simulated interval because \(0.014<0.041<0.050\). The upper-tail estimate is \( (178+1)/(1000+1)=179/1001=0.179 \). With \(\hat{q}=178/1000=0.178\), the approximate Monte Carlo standard error is \( \sqrt{0.178(1-0.178)/1000}=0.0121 \).
The evidence does not identify a clear discrepancy in this particular measure of cluster-level variation: the observed value is inside the reference interval, and a substantial fraction of replications are at least as large. A careful conclusion is not that the random-effects structure has been validated. This check assesses one summary of clinic proportions under one simulation scheme; it may not reveal other problems, such as an unusual pattern in a few clinics or a misspecified relationship with a covariate.
Worked Example: A Tail Discrepancy With Few Clusters
A different hypothetical count analysis has only 18 clusters. The prespecified statistic is the share of all observed events contributed by the two clusters with the largest counts. The observed share is 41%. Under 2,000 plug-in replications, with the cluster design fixed and random effects and outcomes simulated, the median share is 30% and the central 95% interval is 22%–39%. Sixty-two replications produce a share of at least 41%.
The observed share exceeds the upper endpoint: \(41\%>39\%\). The upper-tail estimate is \( (62+1)/(2000+1)=63/2001=0.0315 \). The approximate Monte Carlo standard error, using \(\hat{q}=62/2000=0.0310\), is \( \sqrt{0.0310(1-0.0310)/2000}=0.00388 \).
This check suggests that the fitted model may not reproduce the observed concentration of events in a small number of clusters. Because there are only 18 clusters, the estimated random-effects distribution and the comparison itself may be unstable. The report should therefore describe the discrepancy and its design limitation, not present the tail estimate as definitive evidence for a particular alternative model. The model’s predictions for clusters like these should be treated cautiously until the pattern is investigated.
How to Synthesize Checks and Disclose Limitations
Diagnostic reporting is most useful when it distinguishes a specific mismatch from a general judgment. Several checks can point in different directions because they target different features. A zero-count statistic can flag a problem even when cluster-level variance looks plausible. Report each relevant result rather than combining unlike statistics into a single “pass” or “fail” label. If a check was selected after examining the data, or many summaries were inspected, say so and treat an isolated extreme result as exploratory.
State limitations that affect the diagnostic evidence. Common examples include few clusters, sparse categories, a small number of replications, sensitivity to the selected statistic, parameters held fixed during simulation, and conditional checks that do not assess the random-effects distribution. Note whether the checks cover the intended use of the model: a model that describes observed clusters adequately may still predict new clusters poorly. Explain the practical implication in context, such as which estimates, comparisons, or predictions merit caution.
Simulation-based checks complement, rather than replace, other approaches. Residual plots can reveal patterns against fitted values or covariates that a single summary statistic misses, but residuals in GLMMs require careful interpretation. A formal test can focus on a defined null hypothesis, but its assumptions and calibration may not match a complex fitted model. Held-out predictive checks assess performance on data not used for fitting, but may have limited precision when there are few clusters. A report should identify the evidence actually obtained, not imply that one approach answers every diagnostic question.
Define the statistic and the data feature it is intended to assess.
Report the number of replications, what was fixed, what was simulated, and whether fitted parameters or cluster effects were held fixed.
Give the observed statistic, a simulated summary or interval, and a directional tail proportion when relevant; include simulation uncertainty where it helps.
State which feature appears poorly reproduced, what remains unresolved, and how this limits the intended interpretation or prediction.
A common mistake is to write, “The 95% simulation interval contained the observed statistic, so the model fits.” That conclusion is too broad: the interval concerns a particular statistic under a particular simulation scheme, and it may have limited power to reveal other departures. The correction is to report the checked feature and its result precisely, then qualify the scope of the conclusion. Likewise, a small simulation tail proportion is not the probability that the model is wrong.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- A report gives a simulated interval but does not say whether random effects were conditioned on or redrawn. What important interpretation is missing?
- In 499 replications, 24 statistics are at least as large as the observed value. Calculate the adjusted upper-tail estimate.
- An observed statistic lies inside a central 95% simulation interval. What is one justified conclusion, and what is one unjustified conclusion?
- Why should a report identify whether many diagnostic statistics were examined?
Solutions. (1) The reader cannot tell whether the check assesses response variation conditional on fitted effects for observed clusters or also assesses cluster-to-cluster variation under the fitted random-effects distribution. (2) The estimate is \( (24+1)/(499+1)=25/500=0.0500 \). (3) It is justified to say that this statistic is within the stated reference interval under the stated scheme. It is not justified to claim that the entire model has been validated. (4) With many inspected statistics, an extreme result may arise through selection; that makes the evidence more exploratory and calls for appropriately cautious interpretation.