Why Stratify a Visual Predictive Check?
A pooled visual predictive check (VPC) can look adequate even when the model predicts one clinically relevant group too high and another too low. Stratification asks a more specific question: does the model reproduce the observed distribution within each prespecified group? Examples include dose levels or renal-function categories. The comparison is made separately for each group, rather than allowing differences between groups to be averaged together.
A stratified VPC does not change the basic predictive-check target or the simulation procedure. It applies the same comparison within defined subsets of the data. The earlier material on visual predictive checks and prediction correction establishes how to choose the predictive target, generate replicate data, and interpret reference envelopes. Here the key additional requirements are to apply the group definition consistently and to judge whether each stratum contains enough observations for the quantiles being plotted.
Define and Apply the Strata Consistently
Let \(g\) denote a group, \(k\) a time bin, and \(b\) a simulation replicate. For example, \(g\) might identify a low-, medium-, or high-dose category, while \(k\) identifies a time interval. Write the observed concentrations in that stratum and bin as \(Y_{gk}\). The observed \(p\)th quantile is \(\widehat{Q}_{gk}(p)\). In replicate \(b\), calculate the corresponding quantile \(Q^{(b)}_{gk}(p)\) from its simulated concentrations.
The group rule must be the same for the observed and simulated records. If the check conditions on the groups in the observed study design, retain each observation’s observed group when forming its simulated counterpart. If the predictive target instead simulates group membership, apply the same prespecified classification rule to the simulated covariate values. In either case, do not assign groups by looking at the observed concentrations: doing so makes the comparison outcome-dependent.
Group boundaries should have a scientific or design rationale and be fixed before inspecting the VPC results. Categories may be useful for a dose-level comparison or a clinically defined renal-function category. Arbitrary cut points can create groups that are difficult to interpret, and repeatedly changing the cut points until a plot looks satisfactory undermines the diagnostic value of the check.
Within each stratum, time bins must also yield meaningful summaries. A common set of time bins aids visual comparison across groups, but may leave some group-by-time cells nearly empty. Bins can be adapted to the sampling pattern when justified; the resulting choices and counts should be reported. The general principles of bin selection and sparse-sampling VPCs still apply within each stratum.
Constructing the Group-Specific Quantile Comparisons
For each group and time bin, calculate the selected observed quantiles, such as the 10th, 50th, and 90th percentiles. Calculate those same quantiles for every simulated replicate using the same group and bin definitions. Across replicates, summarize each simulated quantile by its median and a chosen reference interval. Plot the observed quantile against that simulated summary. Repeat the procedure separately for every stratum.
The simulated reference interval describes the variation in the replicate quantiles under the specified predictive-check procedure. It is not automatically a confidence interval for a model parameter, and it is not a guarantee that a well-specified model will contain the observed quantile in every group and bin. The interpretation also depends on the predictive target and on whether parameter uncertainty is included in the simulations.
A practical display uses one panel per stratum, with observed and simulated quantile curves shown over time. Use consistent axes when comparing the magnitude of concentrations across groups; separate axes can make patterns easier to see when scales differ greatly, but may hide those magnitude differences. Label each panel with the group definition and show the number of observations, and preferably the number of individuals, contributing to each bin.
Worked Example: Reading a Stratified Quantile Comparison
Suppose a hypothetical study has a prespecified low-dose group. In one time bin, its 10 observed concentrations, in ascending order, are 1.0, 1.2, 1.4, 1.5, 1.8, 2.0, 2.2, 2.5, 3.0, and 3.4 mg/L. For this example, use the nearest-rank sample quantile: the \(p\)th quantile is the observation at rank \(\lceil np\rceil\).
For the median, \(n=10\) and \(\lceil 10(0.50)\rceil=5\), so \(\widehat{Q}_{gk}(0.50)=1.8\) mg/L. For the 90th percentile, \(\lceil 10(0.90)\rceil=9\), so \(\widehat{Q}_{gk}(0.90)=3.0\) mg/L.
Now suppose 200 replicate data sets are generated using the defined predictive target, and the 90th-percentile replicate values have a median of 3.6 mg/L and a 95% reference interval from 3.1 to 4.2 mg/L. The observed 90th percentile, 3.0 mg/L, is below that interval. This is a signal to investigate possible underprediction of the upper tail in this group and bin; it does not, by itself, establish a cause. In contrast, if the median replicate 50th percentile is 1.9 mg/L and its reference interval is 1.5 to 2.4 mg/L, the observed median of 1.8 mg/L is within that interval.
These results are interpretable only with the counts in view. Ten observations support a visible empirical distribution, but still provide limited resolution for a tail quantile. The apparent 90th-percentile discrepancy should therefore be considered alongside its simulation variability, the sampling design, and neighboring bins.
Does Each Stratum Have Enough Observations?
A stratum can have many observations overall but very few in a particular time bin. Quantile precision depends on the count in the specific group-by-bin cell, not just on the total study size. This is especially important for tail quantiles: a 5th percentile estimated from a small cell may be determined almost entirely by the smallest one or two observations.
A simple order-statistic argument makes the issue concrete. Suppose \(n\) observations are independent draws from a continuous distribution with true \(p\)th quantile \(q_p\). Each observation has probability \(p\) of falling below \(q_p\). Thus the number falling below \(q_p\) has a binomial distribution with mean \(np\). In particular, the probability that none fall below \(q_p\) is \((1-p)^n\).
For an upper-tail quantile, the corresponding concern is having too few observations above it: the number above the true \(p\)th quantile has mean \(n(1-p)\), and the probability of none above it is \(p^n\). These calculations are not a universal pass/fail rule. They show why tail summaries become coarse and unstable when \(n\) is small.
Worked Example: A Sparse Stratum and a 5th Percentile
Consider a hypothetical group-by-time cell with 12 observations. If the target is the 5th percentile, the expected number of observations below the true percentile is \(np=12(0.05)=0.6\). The probability that none fall below it is
Thus, even when the underlying distribution is the one assumed by the model, this cell has about a 54.0% probability of containing no observations below the true 5th percentile. A plotted empirical 5th percentile in such a cell cannot be treated as a finely resolved estimate of the lower tail.
For comparison, a 50th percentile has an expected \(12(0.50)=6\) observations below the true median, so it is generally less dominated by a single extreme order statistic. It is still uncertain with only 12 observations. The conclusion is not that 12 is always too few; it is that the plotted quantile, the cell count, and the intended interpretation must be considered together.
Repeated observations require additional care. The binomial calculation assumes independent draws; concentrations from the same individual are generally not independent. A cell containing 20 records from 5 individuals does not provide the same information as 20 independent individuals. The simulation should preserve the relevant subject-level sampling structure, and the VPC should report both record counts and the number of contributing individuals when feasible.
Warning signs include quantile curves that jump abruptly between neighboring bins, extreme quantiles that equal a cell minimum or maximum, large changes after a small number of observations are removed, and bins with very few contributing individuals. These patterns may reflect a genuinely unusual data feature, but they may also indicate inadequate resolution. Show the counts rather than smoothing away the evidence of sparsity.
Why Pooling Can Hide Group-Specific Misfit
A pooled VPC combines observations from different strata. If the model overpredicts in one group and underpredicts in another, these discrepancies can partially cancel in the combined distribution. The pooled check can therefore look satisfactory while the model’s conditional predictions are poor. Conversely, a pooled discrepancy may be driven mainly by a change in group composition rather than a within-group predictive failure.
Worked Example: A Pooled Median Masks Opposite Group Errors
Consider a simplified hypothetical single-bin example with 20 observations in each of two groups. The model’s predictive median is 2 mg/L for the low-dose group and 8 mg/L for the high-dose group. Suppose the observed medians are 3 mg/L and 7 mg/L, respectively. Within the low-dose group, the observed median is 1 mg/L above the model median; within the high-dose group, it is 1 mg/L below.
If each group contributes the same number of observations and the pooled distribution is summarized by its median, a simplified representation gives pooled model values of 20 observations at 2 and 20 at 8. The two middle values are 2 and 8, so their average is \((2+8)/2=5\) mg/L. The corresponding pooled observed values, 20 at 3 and 20 at 7, also have pooled median \((3+7)/2=5\) mg/L.
The pooled medians agree at 5 mg/L, but neither group’s observed median agrees with its model median. Real data are not usually this simplified, but the example demonstrates the principle: pooled agreement does not establish group-specific agreement. Stratified panels expose the opposite within-group differences.
Choosing a Useful Display and Interpreting Its Limits
A stratified VPC is most useful when the groups address a prespecified scientific or design question and have enough data to support the summaries. It complements, rather than replaces, the pooled check. A pooled view summarizes overall predictive behavior; stratified views reveal whether that behavior differs across the selected categories. Prediction correction under dose variation can address a related scaling issue, but it does not make group-specific diagnostics unnecessary when the scientific question concerns dose groups.
Interpret any unusual pattern as a diagnostic signal, not a standalone test or proof of a mechanism. A group-specific mismatch can arise from model structure, sampling variability, sparse cells, misspecified residual variability, or features of the predictive target. The VPC alone does not identify which explanation is correct. Follow up using the diagnostic workflow, the group counts, and the study design.
A stratified display also has limits. Creating many categories divides the data into smaller cells and increases the number of comparisons, making chance deviations and unstable quantiles more likely. Categories can conceal trends within each group, while arbitrary cut points can make results sensitive to small changes in classification. Continuous-covariate displays address a different question and should not be imitated by repeatedly subdividing a continuous variable into many sparse categories.
Before interpreting the plot, check that the strata were defined without reference to outcomes, the same rule was applied to observed and simulated data, and each plotted cell has adequate record and subject counts for the selected quantiles. If a tail quantile is not supported, omit it or label it as unreliable rather than presenting it with false precision.
Check Your Understanding
Answer each question using the definitions and reasoning above.
- A study defines renal-function categories before analysis. What must be done to use those categories in a stratified VPC?
- A cell contains 10 observations. Under the independent continuous-sampling argument, what is the probability that no observations fall below the true 10th percentile?
- Why can a pooled VPC look acceptable when a stratified VPC shows a discrepancy in two groups?
- A bin contains 24 concentration records from 4 individuals. Why is it misleading to treat this as 24 independent observations when judging quantile information?
- In one stratum, the observed 90th percentile lies just outside the simulated reference interval, but the bin has only 8 observations. State a cautious interpretation.
Solutions. (1) Apply the same prespecified classification rule to the observed and simulated data according to the predictive target, and calculate summaries separately within each group and time bin. (2) The probability is \((1-0.10)^{10}=0.9^{10}\approx 0.3487\), or 34.9%. (3) Overprediction in one group and underprediction in another can cancel in the pooled distribution, as can be seen in the opposite group-specific median differences in the worked example. (4) Records from the same individual are dependent, so 24 records do not represent 24 independent sources of information; report the four contributing individuals and preserve subject-level structure in simulation. (5) The observation is a signal worth investigating, but the empirical tail quantile is poorly resolved with only 8 records. Check counts, contributing individuals, neighboring bins, and the predictive setup before treating it as evidence of systematic misfit.