Why Stratify a VPC?
A visual predictive check (VPC) compares summaries of observed outcomes with the corresponding summaries from data simulated under a fitted model. A pooled VPC can look satisfactory even when predictions differ across clinically relevant groups: opposing subgroup patterns may be averaged together, or a large subgroup may dominate the display. A stratified VPC makes the comparison separately within prespecified covariate or demographic groups, such as weight categories or age groups.
Stratification changes the question being asked. Instead of asking whether the model reproduces the overall outcome distribution, we ask whether it reproduces that distribution within each defined subgroup, at comparable times or other design points. The separate panels are diagnostic summaries, not tests of whether a covariate causes an outcome difference. A discrepancy may indicate that the model does not capture subgroup-specific behavior, but it may also reflect sparse observations or variability in the simulated summaries.
Constructing the Panels
Begin by specifying the subgroup definition and the target of the predictive check. For example, if the target is concentration over time by baseline weight group, define the weight cutpoint before inspecting the VPC patterns. Use covariate values and sampling designs appropriate to the predictive-check target when generating replicates. For a check conditional on the observed design, simulations retain the relevant subjects' covariates and observation times while generating new outcomes under the fitted model.
Within each stratum, divide observations into time bins using a common binning rule where feasible. For each stratum-bin combination, calculate the selected observed quantiles, often the median and lower and upper quantiles. Calculate the same summaries in each simulated replicate. The distribution of replicate summaries then describes how much those summaries vary under the model and the simulated design.
Here \(g\) indexes the stratum, \(b\) the time bin, \(p\) the quantile (for example, \(p=0.5\) for the median), and \(r\) the replicate. \(Q^{\mathrm{obs}}_{g,b}(p)\) is the observed quantile; \(Q^{(r)}_{g,b}(p)\) is the same quantile calculated from replicate \(r\). A central simulation envelope for the quantile can be formed from the chosen lower and upper percentiles of the \(R\) replicate quantiles. Plot the observed quantile against that envelope in the corresponding panel. State the number of subjects and observations contributing to each panel or bin, since the stratum label alone does not reveal how much data support the displayed summary.
The envelope is not the same thing as an interval for an individual outcome. It describes variation in a summary statistic across simulated data sets. Nor does it automatically include uncertainty in estimated model parameters: that is included only if the simulation procedure propagates parameter uncertainty. With a finite number of replicates, the estimated envelope itself also has Monte Carlo error.
Worked Example: Comparing Median Trends by Weight Group
Suppose a hypothetical study has 60 subjects in each of two prespecified baseline-weight groups: below 70 kg and at least 70 kg. In a 0–2 hour bin, the observed concentration medians are 8.2 mg/L and 11.0 mg/L, respectively. In a 2–6 hour bin, they are 3.1 mg/L and 4.0 mg/L. Assume the same time bins are used in every replicate, and that simulations retain each subject's baseline weight and sampling times.
For the lower-weight group in the 0–2 hour bin, suppose the 10th and 90th percentiles across simulated replicate medians are 7.1 and 9.4 mg/L. The observed median, 8.2 mg/L, lies between these values. In the higher-weight group, suppose the corresponding envelope is 9.6 to 12.5 mg/L; the observed median of 11.0 mg/L also lies within it.
At 2–6 hours, suppose the lower-weight group's median envelope is 2.6 to 3.8 mg/L and the higher-weight group's is 3.2 to 4.8 mg/L. The observed medians, 3.1 and 4.0 mg/L, again lie within their respective envelopes. These figures show no obvious median discrepancy in these bins, but they do not establish that the model is correct. The check covers only the displayed summaries and bins; other quantiles or time regions may show different patterns.
The higher medians in the higher-weight group are an observed descriptive contrast, not evidence that weight causes higher concentrations. The purpose here is narrower: to assess whether the model's simulated distributions reproduce the observed summaries within each defined group.
Why Pooled Summaries Can Hide a Pattern
A pooled distribution combines subgroup distributions using weights determined by the subgroup proportions in the bin. Let \(F_g(y)\) denote the outcome distribution in stratum \(g\), and let \(w_g\) be that stratum's fraction of the pooled observations. The pooled distribution is the weighted mixture
The pooled quantile is calculated from this mixture; it is generally not the weighted average of the subgroup quantiles. Because subgroup proportions can differ across time bins, a pooled trend can change when the composition changes, even if the within-stratum distributions do not. Conversely, distinct within-stratum patterns can be obscured in the pooled summaries.
A simple discrete example makes the point. Suppose every outcome in group A is 0 and every outcome in group B is 100. If group A contributes 90% of observations, the pooled distribution has \(F_{\mathrm{pooled}}(0)=0.9\), so its median is 0. If group A contributes only 40%, then \(F_{\mathrm{pooled}}(0)=0.4\) and the pooled median is 100. The two subgroup distributions have not changed; only their relative sizes changed. Separate panels preserve the information that the group medians are 0 and 100 in both cases.
Accounting for Subgroup Size and Simulation Uncertainty
A panel can appear precise while being based on few observations, especially when many subjects contribute measurements to other bins but not to that panel's particular bin. Report both the number of subjects and the number of observations per relevant stratum and bin where possible. Repeated observations from one subject are not equivalent to observations from different subjects; the design and the simulated replicates should preserve the dependence relevant to the predictive check.
Simulation envelopes help show how variable a subgroup quantile is under the fitted model and design. A wider envelope in a small subgroup is often expected: fewer observations make each replicate's empirical quantile more variable. An observed quantile inside an envelope is not proof of adequate fit, and one outside it is not, by itself, proof of misspecification. Look for the size, direction, persistence, and location of departures, and consider whether the subgroup has enough data to support the displayed summary.
Worked Example: Reading a Simulation Envelope
In a hypothetical high-age stratum, a particular time bin contains 24 observed subjects. The observed median concentration is 12.4 mg/L. Across 1,000 simulated replicates, sort the 1,000 replicate medians from smallest to largest. Using the nearest-rank convention, the 5th and 95th percentiles are the 50th and 950th ordered values. Suppose these are 10.8 and 13.7 mg/L.
Thus the central 90% simulation envelope for the median is 10.8 to 13.7 mg/L, and the observed 12.4 mg/L falls within it. This indicates that the observed median is compatible with the range of replicate medians shown for this bin under the specified model and simulation design. It does not mean there is a 90% probability that the model is correct, nor is it a confidence interval for an unknown population median.
If the observed median instead were 14.2 mg/L, it would exceed the upper envelope by \(14.2-13.7=0.5\) mg/L. That would be a visible discrepancy to investigate, not an automatic rejection rule. Check the neighboring bins and other quantiles, the stratum's data support, and whether the simulation procedure represents the intended predictive target.
Diagnostics, Alternatives, and Limitations
Before interpreting a subgroup contrast, verify that the stratum assignment is applied identically to observed and simulated data, that units and time bins agree, and that the covariate used for grouping is defined at the intended time (for example, baseline rather than a post-baseline value). Inspect sample counts and the number of subjects with observations in each bin. A coding error in group assignment or a mismatch in the analysis data and simulation design can create apparent departures unrelated to model adequacy.
If some strata are too sparse, do not silently merge them after seeing the plotted results. Options include reporting the sparse panels with clear counts and cautious interpretation, using fewer prespecified strata that remain scientifically meaningful, or using wider time bins when this is compatible with the predictive target. Each choice trades detail for more stable summaries. There is no universal minimum subgroup size: the adequacy depends on the quantile, within-subject sampling, and the intended interpretation.
| Approach | What it shows | Main limitation |
|---|---|---|
| Pooled VPC | Overall predictive behavior across the included population | Can conceal subgroup departures or reflect changing subgroup composition |
| Stratified VPC | Predictive behavior separately within defined groups | Smaller groups and bins can produce unstable quantiles and wide envelopes |
| Separate model fits by subgroup | Group-specific fitted behavior under distinct fits | Changes the modeling question and may reduce data available per fit; it is not simply a stratified check of one fitted model |
A stratified VPC remains a graphical diagnostic. It does not identify the cause of a mismatch, and it only evaluates the chosen summaries at the chosen bins and strata. Results can be sensitive to cutpoints, sparse sampling, extreme observations, and the number of simulations. Prespecifying groups limits opportunistic searches, but it does not eliminate these limitations.
Worked Example: Recognizing a Sparse-Panel Warning
A hypothetical lower-weight stratum has three observed subjects in one time bin, with concentrations 8, 10, and 22 mg/L. The observed median is 10 mg/L, because the ordered middle value is 10. Across simulations, suppose the 5th and 95th percentiles of the replicate medians are 6.1 and 24.5 mg/L.
The observed median lies within this wide 6.1–24.5 mg/L envelope. The correct reading is that the median is highly variable under this sparse design and that the plot gives limited evidence about the subgroup's central tendency in this bin. It would be misleading to call the model well validated solely because 10 mg/L lies inside the envelope. The three observations also provide little basis for stable lower- and upper-tail summaries.
A useful next step is to check whether a wider, scientifically defensible time bin was prespecified or can be justified without selecting it to obtain a preferred result. If not, retain the sparse-bin result but explicitly report its count and uncertainty, and avoid making a strong subgroup conclusion.
Practical Sequence
Choose scientifically interpretable covariate groups before examining subgroup VPC patterns, and state whether the check is conditional on the observed design.
Use the same strata, bins, and quantiles for observed data and every replicate; record subject and observation counts for each panel or bin.
Read departures in context, distinguishing predictive variability from finite-simulation error and any parameter uncertainty that was not propagated.
Describe which subgroup summaries and bins are supported, flag sparse regions, and avoid treating a visual envelope as a formal pass/fail test.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Why can a pooled VPC hide a mismatch in one covariate subgroup?
- A stratum-bin has 18 observed subjects. Across 200 replicate medians, the 10th and 90th percentile values are 4.2 and 6.8 mg/L. The observed median is 7.1 mg/L. What does this show, and what does it not establish?
- Why should subgroup definitions generally be chosen before inspecting the VPC?
- A panel has only three observations, and its observed median lies inside a very wide simulation envelope. Give the most defensible interpretation.
Solutions. (1) The pooled distribution is a mixture whose weights depend on subgroup proportions. A large group can dominate, and opposing or distinct subgroup patterns can be obscured. (2) The observed median exceeds the displayed 90% simulation envelope's upper endpoint by \(7.1-6.8=0.3\) mg/L, a discrepancy worth investigating. It does not by itself prove misspecification or provide a formal test. (3) Prespecification reduces the risk of selecting groups because they produce a preferred pattern and supports an interpretable comparison. (4) The bin provides weak information about the subgroup median; the wide envelope indicates substantial variability under the simulated design. Being inside the envelope alone is not evidence that the model has been validated.