Why Discrete Outcomes Need Discrete Summaries
A predictive check asks whether data simulated from a fitted model can reproduce features of the observed data. For a continuous response, it is often useful to compare medians or other quantiles across time or covariate bins. For a categorical response, the central questions are usually different: are the category proportions plausible, and does the model reproduce the observed category frequencies? For a count response, the frequency of zero counts, the upper tail, or selected count quantiles may be more informative than a smooth percentile curve.
The model and the replication scheme matter. Simulations should follow the same observation design as the data: the same subjects or covariate groups, observation occasions, and relevant missing-record structure. The analyst must also choose whether the check is conditional on fitted subject effects or uses new subject effects, as appropriate to the predictive question. The resulting simulated datasets provide a reference distribution for a chosen summary; they do not make that summary a complete description of model fit.
Choosing and Comparing Summaries
For a binary outcome, a natural summary is the number or proportion of records in one category. For a response with \(K\) categories, the full frequency vector \((N_1,\ldots,N_K)\), or its proportions \((N_1/n,\ldots,N_K/n)\), preserves information about which categories are over- or under-represented. For a nonnegative count outcome, useful summaries can include the proportion of zeros, selected frequencies, or empirical quantiles. A mean alone may hide important differences: two models can have the same mean count but very different probabilities of zero or of a large count.
Let \(T_{\mathrm{obs}}\) be a chosen statistic from the observed data and \(T^{(m)}\) the same statistic from simulated replicate \(m\), for \(m=1,\ldots,M\). A central simulation reference interval can be formed from the empirical 2.5th and 97.5th percentiles of the \(M\) replicate statistics. Comparing \(T_{\mathrm{obs}}\) with this interval is a visual or numerical diagnostic. It is not automatically a calibrated 5% hypothesis test: the interval is usually pointwise, and its calibration depends on the simulation procedure and model-fitting context.
For repeated records within subjects, the simulated dataset should preserve the dependence implied by the model. In particular, independently simulating each record from a marginal probability can fail to reproduce within-subject association induced by shared random effects. Simulating complete subjects under the selected conditional or unconditional scheme carries that dependence into the reference distribution. The same principle applies to counts: simulate the record structure, not a detached collection of marginal values, when the check concerns a clustered dataset.
Worked Example: Checking a Binary Response Proportion
Suppose a hypothetical dataset contains one binary response for each of 50 subjects. There are 18 responses in the category “event,” so the observed event proportion is \(18/50=0.360\). The fitted model is used to generate 500 replicate datasets with the same 50 subjects and covariates, using the chosen random-effects simulation scheme.
Suppose the simulated event-count distribution has a median of 15.2 events and an empirical central 95% reference interval from 9 to 22 events. Dividing by 50 gives a median simulated proportion of \(15.2/50=0.304\), and interval limits of \(9/50=0.180\) and \(22/50=0.440\).
The observed proportion, 0.360 or 36.0%, lies inside the simulated interval from 18.0% to 44.0%. Thus this aggregate proportion is not unusual relative to these simulations. It does not establish that the model fits all subjects or covariate patterns; for example, overprediction in one subgroup could be concealed by underprediction in another.
When there are several categories, inspect the vector rather than relying on one chosen category. Because category counts sum to the total sample size, their components are dependent: an increase in one category’s count necessarily reduces the available count for others. Marginal intervals for each category are still useful for locating discrepancies, but they should not be interpreted as a simultaneous joint interval for the entire vector.
Worked Example: Comparing Frequencies Across Three Categories
Consider 60 hypothetical records classified as A, B, or C. The observed counts are 28, 20, and 12. Their proportions are \(28/60=0.467\), \(20/60=0.333\), and \(12/60=0.200\), respectively. Suppose 500 replicate datasets, each with 60 records and the same design, give the following median counts and pointwise 95% reference intervals.
| Category | Observed count | Simulated median | Pointwise 95% interval |
|---|---|---|---|
| A | 28 | 24 | 17 to 32 |
| B | 20 | 21 | 13 to 29 |
| C | 12 | 15 | 8 to 23 |
Each observed count falls inside its corresponding interval: \(17\leq28\leq32\), \(13\leq20\leq29\), and \(8\leq12\leq23\). This check does not flag an obvious marginal frequency discrepancy. The observed A count is four greater than its simulated median, while C is three lower; such differences can be noted, but the three categories cannot vary independently because their counts total 60.
Count Quantiles, Discreteness, and Interpretation
For a count outcome, a quantile can summarize a part of the distribution that the mean misses. To make the calculation reproducible, state the empirical quantile convention. For example, with \(n\) sorted observations, the nearest-rank \(p\) quantile is the observation at rank \(\lceil pn\rceil\). Integer-valued data produce ties and jumps, so quantiles need not vary smoothly across simulations. It is often useful to check several summaries together, such as the zero proportion and an upper quantile, rather than treating a single quantile as decisive.
Worked Example: Checking Median and Upper Count Quantiles
A hypothetical dataset has 80 count records with the following frequencies: 20 zeros, 30 ones, 15 twos, 7 threes, and 8 fours. Under the nearest-rank convention, the median is at rank \(\lceil0.50(80)\rceil=40\). Ranks 21 through 50 are ones, so the observed median is 1. The 90th percentile is at rank \(\lceil0.90(80)\rceil=72\). Ranks 66 through 72 are threes, so the observed 90th percentile is 3.
Suppose 500 simulated datasets of the same design give a central reference interval of 0 to 2 for the median and 2 to 5 for the 90th percentile; their respective simulated medians are 1 and 4. The observed median of 1 and 90th percentile of 3 both lie inside their intervals. The upper quantile is one count below its simulated median, but it is not outside the reference range. This result is a useful diagnostic, not proof that the model reproduces every feature of the count distribution.
A brief justification explains why simulation is particularly useful here. If the outcome is binary with independent probability \(p\) in each of \(n\) records, its event count \(N\) has \(E(N)=np\) and \(\operatorname{Var}(N)=np(1-p)\), so the proportion \(N/n\) has variance \(p(1-p)/n\). But NLME predictive datasets may have varying probabilities across subjects and dependence among records through shared random effects. The simple variance expression then need not describe the statistic’s distribution. Simulating complete datasets under the fitted model automatically reflects the specified heterogeneity and dependence, provided the simulation scheme correctly represents them.
Why Continuous-Response Summaries May Not Apply
A continuous-response VPC commonly displays simulated and observed quantiles over time or covariate bins. The broad idea of comparing summaries remains useful, but some continuous-response operations do not carry over naturally. A category label has no meaningful numerical distance: subtracting “B” from “A” or dividing a category by a reference prediction is not defined in a scientifically meaningful way. For that reason, prediction correction by division, as used for suitable positive continuous measurements, is generally not a sensible transformation of nominal categories.
Count outcomes are numeric, but their support is discrete and often includes zero. Dividing counts by a reference prediction can yield a transformed quantity whose interpretation is unclear, especially when the reference is zero or close to zero. Quantiles can still be calculated for counts, but ties and jumps are expected; smooth curves should not be mistaken for continuous underlying data. Prefer summaries in the original outcome scale unless a carefully justified transformation has a clear interpretation.
The main alternatives are complementary rather than interchangeable. Category proportions or frequencies show which categories contribute to a mismatch. Count quantiles describe selected locations in the count distribution, while a zero proportion or selected frequencies target features such as excess zeros. A continuous-style mean or median alone can miss these patterns. Conversely, a handful of discrete summaries cannot replace inspection of the full model structure or establish that every relevant subgroup is well represented.
Diagnostics, Assumptions, and Common Pitfalls
Before interpreting a check, verify that the simulated datasets use the same outcome definitions and observation design as the observed data. Confirm that category coding, count units, time points, covariates, and missing-record handling are aligned. State whether new random effects are generated or existing subject effects are conditioned upon; those choices answer different predictive questions. If subjects contribute repeated records, retain the subject structure when computing statistics intended to describe the population of subjects or their repeated outcomes.
Look for patterns that a pooled summary can obscure. Calculate proportions or frequencies by relevant covariate or time groups when there is a scientific reason to do so, while recognizing that smaller groups yield noisier summaries. For counts, compare the zero frequency and at least one tail-sensitive feature if those are important to the application. Inspect whether observed summaries repeatedly lie near one side of their simulated distributions across groups, rather than focusing on one isolated discrepancy.
- Wrong reasoning: “The observed category proportion is inside its 95% pointwise simulation interval, so the model is proven correct.” Correction: It means only that this chosen summary is not outside this particular simulated reference range; other features, subgroups, and model assumptions may still be problematic.
- Wrong reasoning: “There are many repeated records, so I can simulate each record independently.” Correction: If the model implies within-subject dependence, independent record-level simulation discards it and can misstate the variability of summaries.
- Wrong reasoning: “A category can be prediction-corrected just like a positive concentration.” Correction: Category labels have no meaningful ratio scale. Compare category proportions or frequencies instead.
- Wrong reasoning: “The mean count matches, so the count model reproduces the data.” Correction: Also inspect features such as zero frequency and upper quantiles when they matter, because the mean does not determine the full count distribution.
Simulation checks also have limitations. A finite number \(M\) of replicates gives an approximate reference distribution, particularly in its tails. Intervals for separate summaries are usually pointwise, not family-wise guarantees. A discrepancy can arise from model misspecification, an unsuitable simulation scheme, or Monte Carlo variability; repeated or systematic patterns deserve more attention than an isolated boundary crossing. These checks diagnose predictive adequacy for chosen summaries, not the truth of the model or its parameters.
Choose an outcome-specific quantity, such as a category proportion, a full frequency vector, a zero proportion, or a count quantile.
Preserve the design and specify how random effects and missing records are handled.
Compute the same statistic in every replicate and assess the observed value against the simulated reference distribution.
Check relevant subgroups and complementary summaries; interpret intervals as diagnostics, not proof of fit.
Check Your Understanding
Answer each question, then compare with the solutions.
- A dataset has 40 binary records, of which 14 are events. What is the observed event proportion?
- Why can marginal simulation intervals for three category counts not be interpreted as an independent joint interval?
- With 50 sorted count observations, what rank defines the nearest-rank 90th percentile?
- Why may independent simulation of repeated records be unsuitable for an NLME predictive check?
- What does it mean if an observed count quantile lies inside its pointwise simulated 95% interval?
Solutions. (1) \(14/40=0.350\), or 35.0%. (2) The category counts sum to the fixed total, so an increase in one category necessarily constrains the others. (3) The rank is \(\lceil0.90(50)\rceil=\lceil45\rceil=45\). (4) It can remove dependence induced by shared subject effects and therefore misrepresent the variability of repeated-record summaries. (5) That chosen quantile is not outside the reference range from those simulations; it does not prove overall model fit.