Why Stratify a Predictive Check?
A predictive check asks whether a fitted pharmacodynamic (PD) model can reproduce features of the observed responses. A pooled check can look satisfactory even when the model describes one clinically relevant part of the data poorly. For example, predictions may be too high at a low dose and too low at a high dose, with the two errors offsetting in an overall average. Stratifying the check helps reveal such local mismatches.
A stratum is a defined subset of observations, such as measurements after a particular dose, measurements from participants with low baseline response, or measurements from a prespecified clinical subgroup. The purpose is not to make every small subset fit perfectly. It is to ask whether the model reproduces relevant features of the data across groups where different predictive behavior is plausible.
Constructing the Check
Let \(Y_{ij}\) denote the PD response for subject \(i\) at observation \(j\). Choose a stratification variable and define strata \(s=1,\ldots,S\). Within a stratum, and often within a time bin as well, select a summary such as a mean, median, or response quantile. For example, a check may compare the observed median response at a given time in each dose group with simulated medians for that same dose group and time.
Generate \(B\) simulated data sets under the fitted model. Each simulation should reflect the model’s relevant variability, including between-subject variation and residual error, and should retain the observed design features needed for the comparison, such as dose assignments and observation times. If the check conditions on covariates, retain those covariate values as appropriate. Calculate the same summary in each simulated data set and stratum as in the observed data.
Here \(T_{\mathrm{obs},s}\) is the summary from the observed data in stratum \(s\), and \(T_{b,s}\) is the corresponding summary from simulation \(b\). An empirical predictive interval can be formed from the simulated summaries. For a central 90% interval, use the 5th and 95th percentiles:
The observed summary falling outside this interval is a visual warning that the model rarely reproduces that feature under the specified simulations. It is not a 5% significance test, and the interval is not a confidence interval for a population parameter. With parameter values fixed at their estimates, it describes simulated variability conditional on those estimates. If uncertainty in parameter estimates is important, it can also be represented in the simulations; the resulting predictive reference then includes that additional source of variation.
Why Pooled Summaries Can Conceal Misfit
For a mean, the pooled value is a weighted average of the stratum-specific means. When strata have equal sizes, errors in opposite directions can cancel exactly. The same intuition applies to other summaries, though pooled medians and quantiles are not generally weighted averages of stratum-specific medians and quantiles.
This gives a simple justification for stratification: agreement of a pooled summary constrains the combined data but does not require agreement in each component. Comparing the component summaries tests predictive adequacy at a finer, scientifically interpretable level.
Worked Example: Opposing Dose-Specific Errors
Hypothetical data. At a prespecified post-dose time, 40 subjects receive a low dose and 40 receive a high dose. The observed mean PD responses are 8.0 units in the low-dose group and 6.0 units in the high-dose group. Across 1,000 simulations, the central 90% predictive intervals for the corresponding group means are 9.2 to 10.8 units and 3.2 to 4.8 units, respectively.
For the low-dose stratum, the observed mean is \(8.0\) units, below the interval’s lower endpoint of \(9.2\) units. For the high-dose stratum, the observed mean is \(6.0\) units, above the interval’s upper endpoint of \(4.8\) units. Thus the model tends to predict too much response at low dose and too little at high dose, at this time point.
Because the group sizes are equal, the pooled observed mean is \( (40\times 8.0+40\times 6.0)/80=7.0 \) units. The average of the two simulated stratum-specific mean centers, 10.0 and 4.0 units, is also \(7.0\) units. A pooled mean comparison could therefore appear well aligned even though both dose-specific comparisons show a mismatch. This arithmetic demonstrates why the stratified displays add information rather than merely repeating the pooled check.
The result is a diagnostic signal, not proof that dose is the cause of the error. Other features correlated with dose, or an unsuitable response summary, could contribute. The next step is to examine the design and model predictions in context.
Choosing and Implementing Strata
Strata should be motivated by the scientific question and defined clearly enough that the same comparison is made in the observed and simulated data. Dose groups and prespecified clinical subgroups are often straightforward. Baseline response can be used to check whether predictions differ in performance across the starting range, but its role in the simulation requires care.
If baseline response is treated as a covariate on which follow-up predictions are conditioned, use the observed baseline values consistently when generating predictions and assign subjects to strata using those observed values. If baseline is itself an outcome generated by the model, the diagnostic may instead evaluate the joint baseline-and-follow-up behavior. In that case, state how strata are defined in each replicate; silently using observed baseline to classify the observed data but simulated baseline to classify replicates changes the comparison.
For a continuous variable such as baseline response, choose cut points before inspecting the diagnostic when possible. Clinically meaningful thresholds or prespecified quantiles can be useful, but each choice answers a different question. Avoid making many narrow bins simply because a plot looks more informative: small strata produce noisy observed summaries and unstable predictive intervals.
Worked Example: Checking Predictions Across Baseline Groups
Hypothetical data. A model predicts follow-up PD response conditional on observed baseline response. Investigators prespecify low and high baseline groups, each containing 30 subjects. At a follow-up visit, the observed median responses are 12.0 units in the low-baseline group and 18.0 units in the high-baseline group. In 1,000 simulated data sets, the central 90% intervals for the group medians are 11.0 to 15.0 units and 16.0 to 20.0 units.
The low-baseline median, 12.0 units, lies within its predictive interval because \(11.0\leq12.0\leq15.0\). The high-baseline median, 18.0 units, also lies within its interval because \(16.0\leq18.0\leq20.0\). For this summary and visit, the observed medians are compatible with the simulated predictive distributions in both groups.
This check is conditional on the observed baseline values because the model uses baseline as a covariate. The same observed group membership and baseline values must therefore be used in the simulations. If the investigators instead simulated baseline and then formed groups from each simulated baseline, they would be checking a different feature: the model’s ability to reproduce the baseline distribution as well as follow-up response.
The conclusion is limited to the selected medians and visit. It does not establish that every part of each group’s response distribution is reproduced, nor does it show that the model is adequate at other times.
Reading the Pattern, Not Just the Interval
A useful stratified display places the observed summaries alongside their simulated reference intervals for each stratum, and, for dynamic responses, repeats the comparison across time bins. Look for coherent patterns: repeated observations beyond an interval, discrepancies in the same direction across adjacent times, or a discrepancy concentrated in one dose or subgroup. Such patterns are generally more informative than an isolated excursion.
The uncertainty in each observed summary depends on how much information is available in its stratum. A wide predictive interval may reflect substantial modeled variability, while a noisy observed summary may result from few subjects or observations. Report stratum sizes and inspect the underlying data rather than treating every out-of-interval point as equally persuasive. Where relevant, compare more than one summary, such as a median and an upper quantile, to see whether the discrepancy is in the center or in the distribution’s tails.
For Monte Carlo checks, the number of simulations matters. With too few replicates, estimated predictive quantiles can be unstable. Increasing \(B\) reduces simulation error in the reference interval, but it cannot compensate for sparse observed strata, a poorly chosen summary, or a flawed model. If parameter uncertainty is omitted, be explicit that the check is conditional on fitted parameter estimates.
Worked Example: A Subgroup-Specific Upper-Tail Check
Hypothetical data. A clinical subgroup contains 24 subjects. At a selected visit, its observed 90th-percentile PD response is 31.0 units. The 5th and 95th percentiles of the 90th-percentile summaries from 1,000 model simulations are 24.0 and 29.0 units.
The observed upper-tail summary exceeds the simulated upper endpoint: \(31.0>29.0\) units. The model’s simulated subgroup 90th percentile therefore does not reach the observed value within this central 90% predictive interval. This suggests that the model may underrepresent high responses in this subgroup, although a small subgroup and an upper-tail statistic can make the observed summary particularly variable.
A careful diagnostic report would state the subgroup definition, its size of 24 subjects, the visit, the chosen 90th-percentile summary, and the simulated interval of 24.0 to 29.0 units. It would not conclude that the model fails for every subgroup member or that a particular structural change is required. The check identifies where further model assessment is warranted.
Alternatives, Diagnostics, and Limitations
A pooled predictive check is simpler and often more stable, but it can conceal opposing subgroup errors. A stratified check sacrifices some precision to show where predictive behavior differs. Residual plots against dose, baseline, or covariates provide a complementary view: they display deviations relative to predictions, whereas a stratified predictive check asks whether simulated data reproduce a selected distributional summary. Observed-predicted plots and individual fit displays answer related but distinct questions. Using these views together can help determine whether a signal is broad or localized.
Stratification does not replace careful model specification. It can also create misleading impressions if cut points are selected after inspecting the results, if groups are too small, or if many strata and time bins are examined without acknowledging the increased chance of seeing an unusual discrepancy somewhere. Predictive intervals at separate times or strata are generally pointwise, not simultaneous: they do not guarantee that all summaries will fall inside their intervals at once.
A key limitation is that a predictive mismatch does not uniquely identify its source. It may reflect model structure, covariate relationships, variability assumptions, or design features not represented appropriately in the simulations. Conversely, summaries can agree while other features of the response distribution remain poorly reproduced. Check multiple scientifically relevant features, retain the context of the model and design, and treat the display as a diagnostic rather than an automatic model-selection rule.
Practical Sequence
Specify the clinical grouping, cut points, response summary, and time bins before interpreting the results.
Simulate the fitted model while retaining the relevant dosing, sampling, and covariate structure; define stratum membership consistently.
Construct empirical predictive intervals from replicate summaries and display observed values with stratum sizes.
Look for coherent, clinically meaningful departures and use complementary diagnostics to investigate them.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- In a dose-stratified check, why can a pooled mean agree with the model even when the dose-specific means do not?
- For a central 90% predictive interval formed from simulated medians, what does it mean if the observed median falls outside the interval?
- A model conditions follow-up predictions on observed baseline response. How should baseline groups be handled in the simulations?
- Name two limitations that should be considered when interpreting many strata and time bins.
Solutions. (1) With equal group sizes, positive and negative errors can cancel in the pooled mean; with unequal sizes, the pooled mean is a weighted combination and can still conceal a subgroup mismatch. (2) The observed summary is relatively unusual under the specified simulated predictive distribution; this is a diagnostic warning, not a formal 5% test or proof of a cause. (3) Retain the observed baseline values and use the same group assignments when simulating conditional follow-up predictions. (4) Small strata can yield unstable summaries, and many pointwise comparisons increase the chance of at least one unusual-looking result; intervals also do not provide simultaneous coverage across all comparisons.