Cross-Model Posterior Predictive Checks

Why Check an Integrated Prediction at Its Interfaces?

An integrated model can produce a plausible final trial prediction even when one of the links feeding it is not describing its data well. For example, a population PK component may generate an implausible distribution of exposure, while a downstream exposure-response component compensates with a different response relationship. Checking only the final outcome can conceal that mismatch. Cross-model posterior predictive checks examine predictions at selected interfaces as well as at the end of the chain.

The aim is not to prove that a model is true. It is to ask whether data generated under the fitted, integrated model resemble the observations in features that matter for its intended use. The earlier material on calibrating integrated predictions with external data addresses calibration; here the emphasis is on designing and interpreting these predictive comparisons across linked components.

What Must Be Replicated?

Let \(D\) denote the data used to fit the integrated model, \(c\) the relevant design and covariates, and \(\theta\) the collection of model parameters and latent quantities. A posterior predictive replicate is generated by first drawing \(\theta\) from its posterior distribution given \(D\), then generating new data under the model and design:

In a linked model, \(p(Y^{\mathrm{rep}}\mid\theta,c)\) represents the full generative chain needed for the check. A replicate may include an exposure quantity generated by the upstream component and a response generated conditional on that exposure by a downstream component. Parameters and shared latent quantities must be drawn jointly as specified by the fitted model; replacing them with separate marginal draws can destroy their dependence. This is particularly important when the observed interface contains uncertainty that was carried across the handoff.

The comparison is meaningful only when the observed and replicated summaries refer to the same target. If observed exposure is estimated from sparse concentration samples, for example, comparing it directly with a perfectly observed model-generated exposure metric omits the observation and estimation process. Instead, simulate data through the corresponding process or define the check on a genuinely comparable latent quantity. Likewise, use the same dosing regimen, covariate distribution, follow-up, endpoint definition, and sampling scheme as the observations being checked.

Choose Summaries That Target the Handoff

A check starts with a prespecified summary \(T\), calculated in the same way for the observations and for each replicate. Useful summaries are tied to a decision-relevant feature of the interface rather than chosen solely because they are easy to calculate. Examples include an exposure quantile, the proportion responding in each of a few exposure strata, or a treatment-control difference in a simulated trial.

For observed data \(Y\), compare \(T(Y)\) with the empirical distribution of \(T(Y^{\mathrm{rep}})\). A predictive interval places the observed summary in context, while a tail area summarizes the fraction of replications at least as extreme in a specified direction. For an upper-tail check based on \(M\) replicates, a common Monte Carlo estimate is:

Here \(I\{\cdot\}\) equals 1 when the condition holds and 0 otherwise. This tail area is a descriptive posterior predictive quantity, not a conventional frequentist p-value: it is not the probability that the model is true, nor generally a uniformly distributed test statistic under a null hypothesis. Its value depends on the chosen summary, direction, model, and data used to form the posterior.

Worked Examples Across the Chain

Why the Predictive Distribution Has This Form

The posterior predictive distribution averages predictions over uncertainty about model quantities rather than treating one fitted parameter estimate as known. To see how this works in computation, draw \(M\) parameter sets \(\theta^{(m)}\) from \(p(\theta\mid D)\). For each draw, simulate \(Y^{\mathrm{rep},m}\) from \(p(Y^{\mathrm{rep}}\mid\theta^{(m)},c)\), then calculate \(T(Y^{\mathrm{rep},m})\). The empirical distribution of those \(M\) summaries approximates the integral defining the predictive distribution.

The convergence is a Monte Carlo approximation as the number of simulated replicates increases, assuming the draws correctly represent the posterior predictive distribution. It does not remove uncertainty or misspecification in the model itself. If only a point estimate \(\hat{\theta}\) is used, the simulation instead approximates \(p(Y^{\mathrm{rep}}\mid\hat{\theta},c)\); it omits posterior parameter uncertainty and is not the full posterior predictive check.

Design, Diagnostics, and Common Pitfalls

A useful check makes its target and simulation recipe explicit. Before interpreting results, verify that each replicate passes through the components needed to produce the checked quantity and that the same summary is computed on observed and replicated data. Then examine whether the result is robust to Monte Carlo noise, and whether several prespecified summaries tell a coherent story. A tail area near 0 or 1 can flag tension for a chosen summary, but is not a stand-alone decision rule.

• Omitting uncertainty at a handoff. Passing a single estimated exposure value downstream can make simulated responses too concentrated. Generate or propagate the exposure uncertainty required by the integrated model.
• Comparing unlike quantities. A model-predicted latent AUC and a noisy estimate derived from sparse samples are not automatically comparable. Include the relevant observation process or choose a shared estimand.
• Simulating the wrong design. A check can appear to fail because replicate datasets use a different dose, covariate mix, follow-up, or endpoint definition. Match the design that produced the observations.
• Relying on a single aggregate. A plausible trial-level response rate can coexist with a poor exposure distribution or an inconsistent response-by-exposure pattern. Check multiple interfaces that matter to the intended use.
• Overinterpreting an in-sample check. When the same observations informed the posterior, the model may adapt to them, making some checks appear reassuring. For claims about prediction in new data, evaluate genuinely held-out or external observations under their design.

A common mistake is to say, “The posterior predictive p-value is 0.22, so there is a 22% probability the model is correct.” The calculation instead says that, under this fitted model and design, 22% of simulated summaries were at least as large as the observed summary in the specified direction. It neither assigns probability to model truth nor evaluates summaries that were not checked.

How This Check Differs From Alternatives

Cross-model posterior predictive checks provide a model-based comparison of observed summaries with replicated summaries, including parameter uncertainty represented by the posterior. A plug-in simulation based on one parameter estimate is simpler, but does not include that uncertainty. A comparison with held-out data asks a more direct question about prediction beyond the fitting data, but requires appropriate data not used to fit the model. Neither approach eliminates the need to choose relevant summaries and align the observation process.

A predictive interval or tail area is also not a formal hypothesis test with a universal rejection threshold. Multiple summaries can be informative, but selecting a striking one after seeing many results increases the chance of emphasizing an accidental discrepancy. Prespecify decision-relevant summaries where possible, report what was checked, and interpret each comparison as evidence about a specified feature of the integrated model—not a certification of the entire evidence chain.