Calibration, Model Adequacy, and Model Comparison

Three Questions That Must Not Be Confused

A residual plot, a simulation diagnostic, and a predictive score may all be called “model checks,” but they answer different questions. In an SDE analysis, first ask whether the inference algorithm returns the intended distribution, then whether the fitted generative model can reproduce important features of the data, and finally whether one candidate gives better predictions than another. Passing one check does not establish the others.

For example, an algorithm can accurately sample from the posterior distribution of a misspecified SDE. Its computation is then calibrated to the assumed model, even though that model may fail to describe the observed concentration trajectories. Conversely, an SDE may be scientifically adequate while a coding error or poorly mixing sampler produces unreliable posterior estimates. Predictive comparison is relative: one candidate may forecast better than another even though both have important shortcomings.

Simulation-Based Calibration of an Inference Algorithm

Simulation-based calibration (SBC) checks computation under a known generative model. Its basic procedure is: draw parameters from the prior, simulate data conditional on those parameters, run the inference algorithm, and compare the generating parameter with draws from the resulting posterior. This is a check of the algorithm relative to the specified prior and likelihood; it is not a test that the specified SDE is a realistic account of a clinical population.

Here is the basic rank result. Suppose the parameter is scalar and continuous, the prior is proper, the data are generated from the prior predictive distribution, and the algorithm returns independent exact draws from the posterior. Conditional on the simulated data, the generating parameter and the posterior draws have the same distribution. Their rank is therefore uniform, provided ties have probability zero.

Proof. For any fixed data value \(Y=y\), Bayes’ rule says that the conditional distribution of \(\theta_0\) given \(Y=y\) is the posterior. The draws \(\theta_1,\ldots,\theta_M\) are also independent draws from that posterior. Thus, conditional on \(Y=y\), the \(M+1\) values are independent and identically distributed. With a continuous distribution, they have no ties with probability one. By exchangeability, each of the \(M+1\) values is equally likely to occupy any specified rank. In particular, \(\theta_0\)'s rank is each of \(0,\ldots,M\) with probability \(1/(M+1)\). This conditional distribution does not depend on \(y\), so the same uniform result holds after averaging over \(Y\). \(\square\)

Model Adequacy: Can the SDE Reproduce Relevant Data?

After computation is checked, assess whether the fitted model can reproduce features of the observations that matter scientifically. A posterior predictive distribution for a replicated data set \(Y^{\mathrm{rep}}\), conditional on observed data \(y\), is defined by first drawing \(\theta\) from the posterior given \(y\), then drawing \(Y^{\mathrm{rep}}\) from the model conditional on \(\theta\). In a pharmacokinetic SDE, this procedure may include simulating latent paths and the observation process. The replicated data must be generated using the same dosing, sampling times, and observation model as the data being checked.

A useful check chooses a discrepancy or summary \(T(Y)\) before interpreting the results. Examples include the number of unexpectedly large one-step innovations, the maximum concentration, a late-time concentration summary, or the variability among subjects. Compare \(T(y)\) with its distribution across replicated data sets. The choice should reflect the intended scientific use: reproducing the mean concentration profile alone may be insufficient if peak exposure or between-subject variability is central.

Posterior predictive checks are descriptive checks of compatibility, not automatic hypothesis tests with a universal pass/fail threshold. Because the observed data inform the posterior used to generate the replications, posterior predictive tail areas generally should not be interpreted as uniformly distributed \(p\)-values under a correct model. A useful report states the discrepancy, how it was selected, the observed value, and the range or distribution from replicated data.

Predictive Model Comparison

Model adequacy asks whether a candidate can reproduce relevant data. Predictive comparison asks which of two or more candidates predicts a defined target better. To avoid rewarding a model for fitting the same observations on which it was judged, evaluate predictions on held-out observations or use a justified cross-validation procedure. Candidates must be evaluated on the same cases, outcomes, and conditioning information. For a held-out observation \(y_i\), a common score is the log predictive density \(\log q(y_i)\), where \(q\) is the candidate’s predictive density given the training data and the available predictors.

Proof. By definition, \(D_{\mathrm{KL}}(p\|q)=\int p(y)\log[p(y)/q(y)]\,dy\), which equals the difference of expectations in the statement. To show nonnegativity, use \(\log x\leq x-1\) for \(x>0\), with \(x=q(y)/p(y)\) wherever \(p(y)>0\). Then \(\int p(y)\log[q(y)/p(y)]\,dy\leq\int p(y)[q(y)/p(y)-1]\,dy\leq 0\), because \(\int q(y)\,dy=1\) and \(\int_{\{p>0\}}q(y)\,dy\leq1\). Negating gives \(D_{\mathrm{KL}}(p\|q)\geq0\). Equality holds when \(p=q\) almost everywhere. \(\square\)

Diagnostics, Alternatives, and Limits

The three questions call for different diagnostics. For algorithm calibration, inspect repeated SBC ranks and also monitor sampler-specific diagnostics such as effective sample sizes and convergence across chains. For model adequacy, examine posterior predictive distributions of prespecified summaries and relevant residual patterns. For predictive comparison, report paired differences in held-out scores and their uncertainty; do not compare scores computed on different targets or different held-out cases.

Common alternatives answer related but distinct questions. Information criteria and likelihood-based comparisons can be useful when their assumptions and target are appropriate, but they are not substitutes for checking predictions on the intended task. In particular, a criterion aimed at in-sample fit or an approximation to predictive performance need not reveal a failure in a clinically important tail or dose-time region.

Common mistake: “The SBC ranks look uniform, so the SDE is adequate.” This conclusion is invalid. Uniform ranks support the claim that the algorithm behaves correctly on simulations from the assumed model and prior; they do not test whether that model resembles observed clinical data. The correction is to report the algorithm check separately, then examine model adequacy using data-relevant replicated summaries, and compare predictions on a clearly specified target.

A second limitation is scope. SBC conclusions depend on the prior-predictive simulations covering the parameter regions of interest; a narrow or unrealistic prior may fail to expose errors elsewhere. Predictive scores depend on the test population and prediction horizon: better prediction at observed sampling times does not guarantee reliable extrapolation to a new dose or schedule. Adequacy is therefore purpose-dependent, and comparison is relative rather than an assertion that the winning candidate is true.