Predictive Checks for Categorical NLME Models

Why Predictive Checks Must Respect Dependence

A categorical NLME model describes both response probabilities and, through subject-level random effects, how responses from the same individual may be related. A model can reproduce the overall proportion in each category while missing a covariate-specific trend or producing too few subjects with consistently similar responses across visits. Predictive checks help reveal these mismatches by comparing observed data with data the fitted model could plausibly generate.

The key is to simulate the data at the level the model describes. For repeated outcomes, one random effect is drawn for each subject and used across that subject’s visits; the response vector is then generated according to the fitted model. Treating all simulated responses as independent would erase the model’s dependence structure and give misleading reference distributions for diagnostics.

For example, let \(Y_{ij}\) denote the categorical response for subject \(i\) at visit \(j\), \(x_{ij}\) the covariates, and \(b_i\) the subject-specific random effect. A fitted model supplies category probabilities such as \(P(Y_{ij}=k\mid b_i,x_{ij};\hat\theta)\), where \(\hat\theta\) denotes fitted parameters. A population-level replicate first draws \(b_i^{\mathrm{rep}}\) from the fitted random-effects distribution, then generates the subject’s responses using the model and that same draw at all visits. If the model specifies additional dependence beyond \(b_i\), that dependence must also be used when generating the response vector.

The covariates and schedule are held fixed because the check asks whether the model can reproduce outcomes under the observed design. The outcomes are regenerated. Consequently, a predictive check is not a comparison between the observed data and an unrelated sample with different covariate composition or follow-up.

Building a Predictive Reference Distribution

Choose a diagnostic statistic before examining the simulated results. It might measure the discrepancy in category frequencies, the largest calibration error across covariate strata, or the number of subjects with a particular repeated-response pattern. Compute it once for the observed data, then compute exactly the same statistic for each replicate.

The simulated values \(T^{(1)},\ldots,T^{(B)}\) form a model-based reference distribution. For a discrepancy where larger values indicate worse fit, a common Monte Carlo tail estimate is

Here \(I\{\cdot\}\) equals 1 when the stated condition is true and 0 otherwise. The added 1 avoids reporting a zero tail estimate merely because no simulated value exceeded the observed statistic. This quantity summarizes how often the fitted model’s replicates are at least as discrepant as the observed data; it is not automatically a calibrated frequentist hypothesis-test p-value. In particular, simulations using a single fitted parameter estimate do not fully account for parameter-estimation uncertainty.

A plug-in check uses \(\hat\theta\) throughout and is often useful for graphical and diagnostic assessment. If uncertainty from estimating \(\theta\) matters, an outer parametric bootstrap can generate data under the fitted model, refit the model to each generated data set, and repeat the diagnostic procedure. This is more computationally demanding, and the refit must use the same modeling steps as the original fit.

Category Frequencies: An Overall Check

For a multinomial or other categorical response, compare observed category counts with the model’s expected counts under the observed design. For repeated data, a simple Pearson-type discrepancy can summarize the difference, but its reference distribution should come from model-based replicates rather than an automatic independent-observation chi-square approximation.

where \(O_{sk}\) is the observed number of responses in category \(k\) and stratum \(s\), and \(E_{sk}\) is the corresponding fitted expected count. The strata may be omitted for an overall frequency check. Expected counts are computed by summing fitted category probabilities over the observations in each cell. If an expected count is zero, this particular discrepancy is undefined for that cell; the category or partition must be reconsidered, or another discrepancy statistic used.

Covariate-Stratified Calibration

An overall frequency check can conceal systematic errors that cancel across covariate values. A model might predict too many responses in one part of the covariate range and too few in another. To detect this, define strata using clinically meaningful covariates or prespecified ranges, and compare observed category frequencies with replicate distributions within each stratum.

A useful summary is the maximum absolute standardized deviation across strata. For a binary category, let \(O_s\) be the observed number in that category in stratum \(s\), and let \(\bar O_s^{\mathrm{rep}}\) and \(SD_s^{\mathrm{rep}}\) be the mean and standard deviation of its simulated count across replicates. Define

This statistic measures the largest stratum-level deviation relative to the model’s simulated variability. Its reference distribution must also be simulated: the maximum is affected by the number and dependence of the strata. A value such as 2 is not automatically a standard-normal cutoff.

Within-Subject Response Patterns

For repeated categorical outcomes, the arrangement of responses within a subject can be more informative than pooled frequencies. Consider binary responses at three visits. A model might match the total number of responses in category A but still underpredict subjects who show category A at every visit or subjects whose responses alternate. Define a subject-level pattern statistic, such as the number of subjects with the same response at all visits. Compute it on each complete replicate, with each subject’s random effect shared across that subject’s visits.

Simulation Procedure, Diagnostics, and Alternatives

The following sequence keeps the comparison aligned with the fitted model and the question being checked.

Graphical checks are often useful alongside a single statistic. For example, overlay observed category proportions and replicate intervals across covariate strata, or compare the observed number of constant-response subjects with a histogram of replicated values. If a check fails, inspect the model’s fitted probabilities, random-effects distribution, covariate specification, and assumed repeated-response structure. A predictive check identifies a feature the fitted model struggles to reproduce; it does not by itself identify the correct replacement model.

The main alternative is an analytic reference approximation, such as a chi-square distribution for a Pearson statistic. Such approximations rely on conditions that can fail with sparse expected counts, estimated parameters, or repeated responses that are dependent. Simulation offers a reference distribution under the fitted model and observed design, but it is only as credible as that model and the implementation of its simulation. A residual plot may localize errors more directly, while a scalar predictive statistic can summarize a targeted discrepancy; using both is often more informative than relying on either alone.

Assumptions and Common Pitfalls

A predictive check is meaningful only if the simulation matches the model being assessed. It assumes that the fitted model specifies a coherent distribution for the categorical outcomes, random effects, and any within-subject dependence; that covariates and the observation schedule are correctly held fixed; and that the code generates the joint response structure the model claims. The chosen statistic should also correspond to a scientifically relevant feature rather than being selected only because it makes the observed data look unusual.

• Pooling away the feature of interest. Matching overall category totals does not establish covariate-stratified calibration or correct within-subject patterns. Check these features separately when they matter.
• Breaking the dependence structure. Simulating each visit as a fresh independent subject can understate or distort the variability of repeated-pattern statistics. Draw a subject effect once per subject and use it as specified across that subject’s responses.
• Using an automatic chi-square p-value. A Pearson discrepancy can be useful, but its usual reference distribution may be inappropriate for dependent repeated outcomes. Use the empirical distribution from complete replicates for the predictive check.
• Calling the predictive tail area a probability the model is true. It is a comparison between the observed statistic and a fitted-model reference distribution, not the probability that the model is correct.
• Ignoring Monte Carlo error. With a finite number of replicates, a tail estimate is noisy. Increase \(B\) when a more precise tail estimate is needed, especially when few replicates exceed the observed statistic.

The check is limited by its chosen summaries: a model may reproduce the statistics examined and still fail in an unexamined aspect of the data. Plug-in simulations also omit parameter uncertainty unless it is incorporated through an outer procedure. Predictive checks therefore complement model interpretation and other diagnostics; they do not certify a model as universally adequate.