End-to-End Validation and Comparison Workshop

Why Validate the Whole Analysis?

A PK SDE analysis is a chain of dependent choices: a stochastic model generates latent concentration paths, an observation model connects those paths to measurements, and a computational method approximates the posterior and predictive distributions. A plausible parameter estimate alone does not establish that the chain works. The fitted model may be misspecified, the posterior computation may be unreliable, or predictions may be much less certain than a point estimate suggests.

An end-to-end validation study makes these issues visible by generating data under controlled conditions and then repeating the intended analysis. It combines the earlier material on calibration and model comparison with residual diagnostics and predictive distributions. The goal is not to certify a model once and for all. It is to find out how the proposed analysis behaves under specified conditions, and to state what the evidence does and does not support.

A Simulation and Evaluation Plan

Begin by specifying the data-generating process, including the dose history, observation times, subject variability, stochastic path variation, and measurement error. Generate complete synthetic trials from known parameter values or from a declared parameter distribution. Then fit each candidate model using the same observation data and a documented computational procedure. Reserve observations or trials for predictive evaluation when the comparison requires a genuine out-of-sample check.

A useful design varies features that may challenge the analysis: sample size, observation timing, dose schedule, between-subject variability, diffusion magnitude, measurement error, and proximity to parameter boundaries. Include both scenarios in which a candidate is correctly specified and scenarios in which it is not. This separates the question “does the computation recover the intended answer under its own assumptions?” from “does the model describe data generated by a different process?”

For a transparent illustration, suppose the latent concentration follows the geometric Brownian PK model

where \(k>0\), \(\sigma\geq 0\), and \(W_t\) is standard Brownian motion. Measurements may be made on the log scale as \(Y_j=\log C_{t_j}+\epsilon_j\), with independent \(\epsilon_j\sim N(0,\tau^2)\). The model keeps concentration positive, and \(\sigma=0\) gives the deterministic exponential-disposition case. These assumptions define a simulation example, not a claim that this model is appropriate for every PK setting.

Posterior Calibration and Coverage

Simulation-based calibration (SBC) checks whether posterior computation behaves as expected when the model and prior used for fitting are also the data-generating model. In each replicate, draw parameters from the prior, simulate data conditional on them, and draw from the posterior conditional on those data. Compare the generating value with the posterior draws. This assesses computation under the specified model; it is not, by itself, evidence that the model adequately describes real data.

The reason for the uniform ranks is exchangeability. In one replicate, first draw \(\theta\) from its prior and data \(Y\) from the model conditional on \(\theta\). Given \(Y\), draw \(S\) independent values from the exact posterior. The generating \(\theta\), conditional on \(Y\), has that same posterior distribution as each of the \(S\) draws. Thus the \(S+1\) values are exchangeable conditional on \(Y\). For a continuous parameter, ties have probability zero, and each value is equally likely to occupy any of the \(S+1\) ordered positions. The generating value’s rank is therefore uniform. With discrete parameters or ties, randomizing the order within tied values gives the corresponding uniform-rank result. This argument requires proper prior-predictive simulation, the same model for generation and fitting, and exact independent posterior draws; approximate, dependent computation can distort ranks.

Coverage complements ranks. For example, a nominal 90% posterior interval should contain the generating parameter in about 90% of repeated prior-predictive simulations if the model and posterior computation are correct. In a finite simulation study, observed coverage has sampling variability, so report the number of replicates and uncertainty around the estimated coverage. Also check joint calibration when dependence among parameters matters: marginally calibrated parameters can still have a miscalibrated joint posterior.

Model Comparison and Predictive Uncertainty

Compare candidates using predictions evaluated on the same held-out observations or trials. A log predictive density score rewards assigning high probability to outcomes that were not used to fit the model. For held-out observations \(y_1,\ldots,y_m\), one possible summary is the sum of their log predictive densities. Scores are comparable only when candidates predict the same outcomes on the same scale and with the same conditioning information. If a model has a higher score, that is evidence for better prediction on this evaluation set; it is not proof that the model is true.

A prediction averages over posterior uncertainty in parameters and latent states, and may also include new measurement error. These sources should be distinguished in reporting. For a predictive variable \(Y\) and parameter \(\theta\), with finite second moments, the law of total variance is

To verify it, write \(m(\theta)=E[Y\mid\theta,y]\). Add and subtract \(m(\theta)\) inside the squared deviation from the overall mean \(\mu=E[Y\mid y]\). After expanding, the cross term has conditional expectation zero because \(E[Y-m(\theta)\mid\theta,y]=0\). The remaining terms are \(E[\operatorname{Var}(Y\mid\theta,y)\mid y]\) and \(E[(m(\theta)-\mu)^2\mid y]\), which are the two terms displayed. The first represents average conditional predictive variation; the second represents variation in conditional means across posterior parameter values.

Computational Reliability and Failure Modes

A validation report should make it possible to distinguish model behavior from algorithm behavior. For posterior simulation, inspect trace plots, effective sample sizes, between-chain agreement such as \(\hat R\), and any sampler-specific warnings. Compare independent runs and, where feasible, known-parameter simulation results. For discretized SDE likelihoods or path simulations, assess sensitivity to the time step or approximation method. A predictive score can be misleading if it reflects numerical error rather than model fit.

A frequent mistake is to read a uniform SBC rank histogram as proof that the model is adequate. Uniformity is expected when data are generated from the fitted model and the posterior is computed correctly; it does not test whether real data follow that model. Conversely, poor SBC ranks can point to computation or implementation problems even when the model family is scientifically reasonable. Keep calibration, model adequacy, predictive performance, and algorithm reliability as separate conclusions.

Other warning signs include comparing scores calculated on different observations, using a single small validation set to claim a definitive winner, ignoring autocorrelation in posterior draws, and reporting narrow predictive intervals that condition on a fixed parameter estimate. If a candidate wins only in one simulation scenario, state that limitation. Results also depend on the parameter ranges, sampling schedules, and dose regimens included in the simulation design; extrapolating beyond those conditions needs separate evidence.