What a Predictive Check Evaluates
A QSP model can predict a clinical outcome distribution, not just a single expected value. For example, it may predict the range of response rates across trials, the distribution of a biomarker at a given visit, or the pattern of mean outcomes over several visits. A predictive check asks whether a clinical dataset is compatible with these simulated distributions for summaries chosen in advance.
The comparison is meaningful only when the observed data and predictions refer to the same clinical question. The endpoint definition, population, treatment, visit times, follow-up, and analysis rule should align. A simulated trial should also reflect the relevant trial design, including sample size and, where applicable, the handling of missing outcomes or censoring. Otherwise, differences may reflect mismatched data processing or design rather than a failure of the model’s clinical predictions.
The predictive distribution should include the sources of uncertainty relevant to the prediction, such as between-patient variability, residual variability, and uncertain inputs or parameters when those are propagated. It should also include the sampling variability of a trial of the planned size. Thus, the spread of simulated trial summaries is generally wider than uncertainty about a model’s expected outcome alone. A narrow band around a predicted mean is not a substitute for a predictive distribution of trial outcomes.
Choose Summaries Before Comparing
A predictive check is most informative when its summaries target clinically important features and are specified before inspecting the observed-versus-predicted comparison. A small, interpretable collection can include:
- Response rates: the proportion meeting a prespecified response definition at a specified time.
- Quantiles: such as the median and upper or lower quantiles of a continuous endpoint, calculated using the same rule for observed and simulated data.
- Longitudinal summaries: such as the mean or median at each prespecified visit, or a prespecified change from baseline.
- Subgroup summaries: when subgroup predictions are part of the intended clinical question, using defined subgroups and adequate sample sizes.
A summary is a targeted view, not a complete description of a distribution. Matching the mean does not ensure that the tails match; matching an overall response rate does not ensure that responses occur in the right subgroups or at the right times. Conversely, an apparent mismatch in one summary does not by itself identify why the model missed.
Worked Example: Checking a Response Rate
Worked Example: Checking a Response Rate
Suppose a hypothetical study observes 68 responders among 100 patients at a prespecified visit. Before examining this result, the analysis plan specifies the response rate as the summary. The QSP predictive simulation produces 2,000 replicated trials, each with 100 patients, using the same response definition and visit. Across those replicates, the mean response rate is 0.55, the standard deviation is 0.05, the central 95% predictive interval is 0.45 to 0.65, and 18 replicate response rates are at least as large as the observed rate.
The observed rate is
It exceeds the upper endpoint of the simulated central 95% predictive interval, 0.65. Its standardized difference from the simulated mean is
This standardized difference is a descriptive measure, not automatically a normally distributed test statistic. The simulation-based upper-tail proportion is \(18/2000=0.009\). Therefore, the observed response rate is unusually high relative to these simulated trials: only 0.9% had a rate at least this high. This is evidence of a discrepancy for the prespecified response-rate summary. It does not establish that the QSP model is invalid in every respect or identify the cause of the discrepancy.
Quantiles Reveal More Than the Center
For a continuous outcome, apply the same quantile definition to the observed data and to every simulated trial. Compare the observed median, for example, with the distribution of simulated trial medians. Do not compare an observed sample median only with a model’s population median and then ignore sampling variation. The simulated distribution of medians accounts for the fact that a finite trial’s median varies even if the underlying predictive distribution is unchanged.
Checking more than one quantile can distinguish a general location shift from a tail mismatch. If the observed median and upper quantile are both higher than expected by similar amounts, a location discrepancy is plausible. If the median is compatible but the upper quantile is consistently too low, the model may underpredict high outcomes or the spread of the distribution. These patterns guide investigation; they do not uniquely diagnose a biological mechanism.
Worked Example: Checking the Median and Upper Tail
In a hypothetical biomarker study, the observed value at a specified visit has a median of 8.4 mg/L and a 90th percentile of 14.0 mg/L. In 1,000 matched simulated trials, the median of the trial medians is 7.5 mg/L, with a central 95% predictive interval of 6.7 to 8.3 mg/L. The median of the trial-specific 90th percentiles is 12.8 mg/L, with a predictive interval of 11.5 to 14.1 mg/L.
The observed median is \(8.4\) mg/L, which is just above the simulated interval’s upper endpoint of \(8.3\) mg/L. This suggests that the observed distribution is shifted upward around its center relative to the simulated trial distributions. The observed 90th percentile, \(14.0\) mg/L, lies within its predictive interval of 11.5 to 14.1 mg/L, near the upper edge. The evidence for a discrepancy is therefore clearer for the median than for the upper-tail summary.
It would be inaccurate to conclude that the model fits the entire distribution because the 90th percentile is inside its interval. Nor does the median discrepancy prove that the model underpredicts every patient’s biomarker value. The check identifies which prespecified summaries appear discrepant; finer diagnosis requires examining the relevant data, design, and model predictions.
Longitudinal Checks Preserve the Pattern Over Time
For repeated outcomes, calculate the chosen summary separately at each prespecified visit in the observed dataset and in every simulated dataset. This produces a distribution of trial-level trajectories. A useful display shows the observed values alongside the predictive median and predictive interval at each visit. The visit-by-visit comparison helps identify persistent directional error, a mismatch that grows over time, or a prediction that is too variable.
Intervals at separate visits are marginal intervals: each describes the predictive distribution at one visit. They are not automatically a simultaneous band for the whole trajectory. A trajectory that falls outside one marginal interval at one visit is not, on its own, a formal test of the complete time course. If the intended question concerns a whole trajectory, specify a trajectory-level summary in advance, such as the maximum absolute standardized deviation across visits, and calculate that summary for each replicated trial as well as the observed trial.
Worked Example: Detecting a Persistent Longitudinal Difference
Consider hypothetical mean biomarker values, in mg/L, at baseline and two follow-up visits. The observed trial means are 10.0, 8.2, and 7.5. The simulated predictive medians and central 95% predictive intervals are:
| Visit | Observed mean (mg/L) | Predictive median (mg/L) | 95% predictive interval (mg/L) | Predictive SD (mg/L) |
|---|---|---|---|---|
| Baseline | 10.0 | 10.0 | 9.4 to 10.6 | 0.30 |
| Week 4 | 8.2 | 9.2 | 8.6 to 9.8 | 0.30 |
| Week 8 | 7.5 | 8.5 | 7.9 to 9.1 | 0.30 |
At baseline, the observed mean matches the predictive median. At Week 4 and Week 8, the observed means are below the lower endpoints of the predictive intervals. Relative to the predictive medians, the standardized differences are
Both follow-up summaries are lower than predicted by 1.0 mg/L, while baseline agrees. This pattern is more informative than a single endpoint comparison: it suggests that the predicted change over time is systematically too small for these trial data. It does not establish which model component or clinical process explains the difference. The simulation must represent the same visit schedule and analysis as the observed trial for this interpretation to be meaningful.
Interpreting Simulation-Based Comparisons
For a scalar summary \(T\), a simple descriptive measure is the standardized predictive difference,
This reports how far the observed summary lies from the simulated mean in units of the simulated standard deviation. It is useful for describing scale and direction, but it is not a formal test unless additional distributional assumptions and a testing procedure are justified. For skewed or discrete summaries, a standardized difference can be especially misleading; predictive quantiles or simulation-based tail proportions are often easier to interpret.
A simulation-based upper-tail proportion can be estimated as the number of replicate summaries at least as large as the observed summary, divided by \(B\). A lower-tail proportion is defined analogously, and a two-sided comparison must specify how extremeness is measured. Such proportions describe the observed summary’s position in the simulated distribution. They are not the probability that the model is true, and they are not automatically calibrated frequentist p-values. The particular predictive framework, including how parameters were estimated and whether the observed data informed them, matters to interpretation.
The justification for these checks is direct: if the model and trial-generation assumptions adequately represent the clinical setting, replicated trials generated under those assumptions should reproduce the features targeted by the prespecified summaries, within their expected predictive variation. A persistent mismatch flags a feature that the model-based predictive distribution does not reproduce. It does not, by itself, distinguish model misspecification from a data, design, or implementation mismatch.
Diagnostics, Alternatives, and Limitations
When a discrepancy appears, first check comparability and computation: confirm endpoint coding, units, visit windows, denominator, analysis population, missing-data handling, and trial sample size. Then inspect whether the simulated replicates include the uncertainty sources and design features relevant to the check. Monte Carlo instability can make estimated predictive quantiles or tail proportions unreliable, so the number of replicates should be sufficient for the precision needed, especially for tail summaries.
Prespecification matters because trying many summaries and reporting only the most extreme comparison exaggerates the apparent evidence of failure. A planned set of summaries can still be reviewed together, but the interpretation should acknowledge that multiple opportunities to find a discrepancy were examined. Sparse subgroups and rare events can also produce unstable summaries; in those settings, report the counts and uncertainty, and avoid treating a noisy tail estimate as a precise finding.
A point prediction, such as the predicted mean response rate, is simpler to communicate but hides trial-to-trial variation. A predictive distribution gives the range expected across replicated trials, at the cost of more modeling assumptions and computation. A single global fit metric can summarize performance compactly but may conceal opposing errors in different parts of the distribution or at different visits. Predictive checks are therefore complementary: they localize mismatches in interpretable summaries, while neither a collection of passing checks nor a good aggregate score proves that all clinically relevant predictions are adequate.
A Practical Sequence
Specify the clinical population, endpoint, treatment, visits, and analysis rule to be checked.
Select response rates, quantiles, longitudinal summaries, or other clinically relevant features before comparing observed and simulated results.
Apply the relevant design and uncertainty assumptions, and calculate each summary using the same rule for every replicate.
Report observed summaries, predictive intervals or distributions, and any descriptive standardized differences or simulation-based tail proportions. Investigate mismatches without treating them as automatic proof of a specific cause.
Check Your Understanding
Answer each question using the predictive-check framework.
- A clinical trial has 80 patients, but each simulated replicate has 500. Why is that a poor direct comparison for the trial’s observed response rate?
- An observed median lies inside its 95% predictive interval, but the observed 90th percentile lies above its interval. What feature might warrant investigation?
- What does a simulation-based upper-tail proportion of 0.02 mean, and what does it not mean?
- At one visit, an observed mean lies outside its marginal predictive interval. Does this alone establish that the full longitudinal trajectory is inconsistent with the model?
- Why can agreement in a predicted mean fail to demonstrate agreement in the outcome distribution?
Solutions: 1. Trial-level response-rate variability depends on sample size; replicates of 500 patients have a different sampling distribution and are not matched to the observed trial. 2. The upper tail may be too high or too variable in the observed data relative to the prediction, even though the center is compatible. 3. About 2% of simulated replicate summaries were at least as large as the observed summary under the specified simulation; it is not the probability that the model is true or automatically a calibrated formal-test p-value. 4. No. A marginal interval concerns one visit; a trajectory-level conclusion requires a prespecified comparison of the pattern across visits. 5. Different distributions can share the same mean while differing in spread, tails, response probabilities, or time course.