Comparing Observed and Replicated Outcome Distributions

Why Compare Outcome Distributions?

A fitted GLMM can describe the relationship between predictors and a response while still failing to reproduce important features of the responses themselves. For example, a model may predict the overall proportion of binary successes reasonably well but produce too few clusters with unusually high success counts. For count outcomes, it may reproduce the mean count while missing the observed spread or the frequency of large counts. Comparing the observed response distribution with distributions generated under the fitted model gives a direct visual check of this kind of mismatch.

The comparison is between the observed data and replicated datasets generated from the fitted GLMM, not between the data and a single fitted mean. Replicated datasets vary because responses are simulated from the model. A useful plot therefore displays both the observed frequencies and the range of frequencies that occur across many replications.

For example, a binary-response GLMM may specify \(Y_{ij}\mid b_j\sim\operatorname{Bernoulli}(p_{ij})\), with a link relating \(p_{ij}\) to fixed and random effects. A count-response GLMM might specify a Poisson response with a log link. The distributional comparison uses the model’s fitted response law to generate the replicated outcomes; it does not replace that law with a normal approximation. The usual interpretation assumes that the response family and link are appropriate, the fitted model has converged to a usable estimate, and the replication procedure respects the observed design and the model’s dependence structure.

The purpose here is descriptive diagnosis. Replicates are generated under estimated parameters, so the resulting reference distribution is conditional on the fitted model and its estimates. A close visual match does not establish that the model is true, and a mismatch does not by itself identify which part of the model should change.

Frequencies and Replicate Summaries

For a discrete response, first choose categories that make the observed distribution readable. For binary data the categories are 0 and 1. For counts, categories may be exact values or grouped values such as 0, 1, 2, and 3 or more. Use the same categories for the observed and every replicated dataset. If \(C_k\) is the number of observed responses in category \(k\), and \(C_k^{(r)}\) is the number in that category in replicate \(r\), then the observed frequency can be compared with the distribution of the \(R\) replicated frequencies.

The replicate average \(\bar{C}_k^{\,\mathrm{rep}}\) describes the typical number of responses in category \(k\) under the fitted model. A useful plot overlays the observed frequency with the replicate average and displays replicate-to-replicate variation—for instance, by showing the middle 95% of the simulated frequencies. Such a range is a simulation reference range, not a confidence interval for a model parameter.

The reason the replicate average is informative follows directly from simulation. Given the fitted model and the replication scheme, each \(C_k^{(r)}\) is a draw from the model-implied distribution of the category count. If that count has finite expectation, then

Thus the replicate average estimates the model-implied expected category frequency. Increasing \(R\) reduces simulation noise in that average, but it does not remove uncertainty from estimating the GLMM parameters. The simulated distribution of \(C_k^{(r)}\), rather than only its average, is important: a difference of two observations may be ordinary when replicate frequencies vary widely and notable when they vary little.

Count Distributions and Grouped Tails

For counts, a histogram or a table of category frequencies makes the shape visible. Exact categories are useful when counts are small and numerous observations occupy each value. When large counts are rare, grouping the upper tail—for example, into “3 or more”—can prevent a plot from becoming dominated by many nearly empty categories. Grouping should be decided in a way that is interpretable and then applied identically to observed and replicated data.

For a distributional comparison, keep the plotted scale consistent. Frequencies are appropriate when every replicate has the same number of responses as the observed dataset; proportions can make comparisons easier to read and are equivalent up to division by that fixed number. If denominators differ, raw frequencies are not directly comparable without accounting for the different totals. The reference ranges should also be reported or plotted by category: a single overall average can hide which parts of the outcome distribution differ.

Look Beyond the Pooled Distribution

A pooled distribution combines responses across clusters. It can look satisfactory even if the model does not reproduce how outcomes are distributed across clusters. For instance, the total number of binary successes can agree with the model while the observed data contain more clusters with all successes or all failures than the replicates typically do. A second comparison can therefore summarize the response distribution within clusters, using the same cluster summaries for observed and replicated data. This is a different view of the data, not a substitute for checking the pooled response frequencies.

Diagnostics, Alternatives, and Limits

Read the plot for patterns rather than treating one visual feature as a verdict. A repeated excess of high counts, too few zeros, an overall shift in binary proportions, or a mismatch that occurs only for particular clusters can each be a useful clue. Check first that categories, denominators, and the observed design were handled consistently. Also check that the simulation code generated the intended response family and preserved the model’s specified structure. A coding error or an inconsistent grouping rule can create an apparent lack of fit.

Distribution overlays complement other diagnostics. Residual displays can show whether discrepancies relate to fitted values, covariates, or cluster ordering; calibration checks focus on agreement between predicted and observed event rates or counts in relevant groups. Pearson and deviance dispersion summaries compress information into a statistic, whereas an outcome-distribution plot retains the category-by-category pattern. None of these approaches replaces the others: a pooled histogram can miss predictor-specific misfit, and a residual summary may not make a sparse tail discrepancy easy to see.

The replicate ranges also have limitations. A central 95% range is a pointwise description for one category at a time; looking across many categories makes it increasingly likely that at least one observed frequency will fall outside its range by chance. Conversely, ranges can be wide in small or highly variable datasets, making meaningful departures hard to notice. The result depends on the fitted model, the estimates used, the replication scheme, the number of replications, and the chosen categories. Distributional agreement is evidence about the model’s ability to reproduce the displayed features, not a guarantee of correct predictions for new settings.

A common mistake is to compare the observed histogram only with the histogram of fitted means, then conclude that the model fits because the bars look similar. A mean response is not a replicated response distribution: it omits the variation expected from the response model and, in a GLMM, the model’s clustering structure. The correction is to generate full replicated datasets from the fitted GLMM, summarize each replicate using the same categories and design, and compare the observed frequencies with the resulting distribution of frequencies.