Match the Evidence to the Intended Use
A model can reproduce several features of the data used to fit it and still be unreliable for a particular prediction task. Conversely, a model may predict a relevant outcome adequately even when a diagnostic displays some visible discrepancies. Reviewing predictive evidence therefore starts with a specific question: what decision or prediction is the model intended to support, and what evidence would reveal failure for that use?
A predictive check compares observed data, or summaries of them, with corresponding data simulated from a model. A bootstrap analysis resamples data and refits the model to examine how estimates vary under resampling. External validation evaluates predictions using data that were not used to fit or select the model. These approaches answer related but distinct questions; one cannot substitute automatically for another. The earlier material on goodness-of-fit diagnostic interpretation concerns checks against fitted data, while this review focuses on what predictive and validation evidence supports.
The purpose and construction of an interval matter. A prediction interval for an individual concentration should reflect the variability relevant to individual observations, such as residual variability and, when appropriate, parameter uncertainty. An interval for a simulated median instead describes variation in that median across replicated datasets. Neither is the same as a confidence interval for an estimated parameter. A reviewer should identify exactly what is being simulated and what the plotted interval summarizes before judging whether observed values are covered.
Check the Simulation Design Before the Plot
A predictive check is informative only if the simulation represents the data-generating and observation process relevant to the question. For a pharmacokinetic check, this can include the dose and dosing history, sampling times, number of subjects, covariate distribution, residual error, and any censoring or missing-data process that affects the recorded concentrations. If the check is intended to assess a particular use, its simulated design should also resemble that use closely enough to make the comparison relevant.
Visual predictive checks commonly divide observations into time or other prespecified bins. In each bin, the analyst may compare observed concentration quantiles with quantiles from simulated datasets. For example, each simulated dataset produces a median in a given time bin; the distribution of those medians across simulations gives a prediction interval for the observed median. The reviewer should check bin definitions, the number and distribution of observations in each bin, and whether the same summary is calculated for observed and simulated data. Sparse bins can produce unstable quantiles, and broad bins can hide time-dependent patterns.
A useful way to understand a prediction interval for a summary is to let \(T_m\) denote the summary calculated from simulated dataset \(m\), for \(m=1,\ldots,M\). The empirical 90% prediction interval can be formed from the 5th and 95th percentiles of \(T_1,\ldots,T_M\). If the model and simulation design correctly represent the process, the observed summary \(T_{\mathrm{obs}}\) should not systematically fall outside such intervals across repeated comparable datasets. That expectation is not a guarantee that every observed summary will fall inside: even a correctly specified model produces some values beyond a 90% interval.
This construction also clarifies a common source of overinterpretation: one observed value inside an interval does not prove that the model is correct, and one value outside does not by itself prove that it is inadequate. Review the number, magnitude, and pattern of departures, as well as whether the simulations included the intended sources of variation. Monte Carlo error also matters: with few simulations, the estimated interval endpoints themselves may be unstable.
Worked Example: Reviewing a Visual Predictive Check
In a hypothetical study, 120 patients contribute concentration samples at scheduled times after dosing. A reviewer sees a visual predictive check based on 500 simulated datasets, each with the same number of patients and the same scheduled sampling times. The plot shows observed median concentrations and 90% prediction intervals for the simulated median in each time bin. The 6-hour bin has an observed median of 4.6 mg/L and a simulated interval of 3.8–5.2 mg/L. The 12-hour bin has an observed median of 3.1 mg/L and an interval of 2.8–3.7 mg/L. At 24 hours, the observed median is 2.4 mg/L, above an interval of 1.5–2.2 mg/L.
Check the comparison. The observed and simulated quantities are medians by time bin, so the plotted intervals concern replicated medians, not individual patient concentrations. The 6-hour and 12-hour medians lie within their respective intervals; the 24-hour median does not.
Review the design. Matching the number of patients and scheduled sampling times supports a like-for-like comparison, provided the simulated datasets also represent the relevant dosing histories, error, and observation rules. The reviewer should check these details and the number of observations in the 24-hour bin before interpreting its departure.
Interpret the pattern. The single high 24-hour median is a signal to investigate late-time predictions, not a stand-alone verdict. If nearby late-time bins also show observed medians above their intervals, that repeated pattern would be more concerning than an isolated departure. The evidence as stated does not establish why the discrepancy occurs or whether the model is suitable for a particular dose decision.
Bootstrap Results: Uncertainty, Not Independent Validation
In a subject-level bootstrap, each replicate samples subjects with replacement from the original analysis set and refits the model. If a subject contributes repeated concentrations, those records are ordinarily resampled together so that within-subject structure is retained. The choice of resampling unit should reflect how the data were collected and the target of inference. Resampling individual records independently when observations are clustered within patients can distort the variation relevant to patient-level predictions.
A bootstrap distribution can show the spread, skewness, or instability of an estimate under resampling. For example, a percentile interval uses the chosen lower and upper quantiles of the bootstrap estimates. But bootstrap resampling of the fitting dataset is not external validation: it does not provide an independent test of predictions in a new population or setting. Nor does a narrow bootstrap interval alone establish that the model’s structural assumptions are appropriate. Review the resampling scheme, refitting procedure, failed runs, and whether the reported uncertainty corresponds to the model quantity needed for the intended use.
Worked Example: Interpreting a Bootstrap Interval for Clearance
A hypothetical population pharmacokinetic analysis estimates typical clearance as 5.1 L/h from 120 patients. The team performs 200 bootstrap resamples, sampling patients with replacement and retaining all of each selected patient’s records. The model converges successfully in 184 replicates. Among those successful replicates, the 2.5th and 97.5th percentiles of typical clearance are 4.3 and 5.9 L/h.
Report what was calculated. The percentile interval from the successful replicates is 4.3–5.9 L/h. It describes the central 95% of the estimated bootstrap distribution among those 184 successful refits; it is not a 95% probability statement that the fixed true clearance lies in this particular interval.
Check the failures. The failed fraction is \(16/200=0.080\), or 8.0%. That is material information, not a detail to omit. The reviewer should establish whether failures occurred randomly or concentrated in resamples with particular data features or extreme estimates. If failure is related to the parameter values, the interval based only on successful fits may underrepresent uncertainty.
Assess relevance. This analysis informs uncertainty in the clearance estimate under the specified resampling scheme. It does not show that simulated concentrations match observed concentrations, and it does not demonstrate predictive performance in an independent dataset. For a dosing decision, the reviewer would also ask whether the clearance uncertainty is propagated into the decision-relevant prediction.
External Validation and Observed-Data Coverage
External validation is strongest when the model, prediction target, and evaluation criteria are specified before examining the validation outcomes, and the data are genuinely independent of model fitting and model selection. The reviewer should check that the external data are relevant to the intended use: a different sampling schedule, measurement method, dose range, or patient mix may limit the comparison or require a clear explanation. If model choices were adjusted after inspecting the external results, the dataset is no longer a fully independent evaluation for those choices.
Observed-data coverage is the fraction of observations that fall within their corresponding prediction intervals. If 72 of 80 independent patient-level observations fall within nominal 90% prediction intervals, the observed coverage is \(72/80=0.900\), or 90%. That agreement is encouraging but not conclusive. Coverage alone can conceal systematic bias: predictions may be shifted high or low while producing a similar number inside the intervals. It also does not show whether interval widths are useful for the intended decision.
The unit of counting must be clear. With repeated measurements from each patient, treating every measurement as independent can make the apparent amount of evidence misleading. Report the number of patients and observations, the coverage definition, and any repeated-measure structure. Stratifying coverage by a prespecified feature of the design—such as dosing regimen or sampling period—can reveal a pattern hidden by the pooled result. Such a pattern is a prompt for review, not by itself a general claim about a subgroup.
Worked Example: Reading Coverage in an External Dataset
A hypothetical model predicts one trough concentration for each of 80 patients in an independent dataset. Before examining these data, the team specifies 90% prediction intervals for individual trough concentrations. Overall, 72 observations fall inside their intervals. Coverage is also reported by dosing regimen: 38 of 40 observations in the standard-regimen group and 34 of 40 in the reduced-regimen group are covered.
Calculate coverage. Overall coverage is \(72/80=0.900\), or 90%. For the standard regimen it is \(38/40=0.950\), or 95%; for the reduced regimen it is \(34/40=0.850\), or 85%. The subgroup counts sum correctly: \(38+34=72\) covered observations and \(40+40=80\) total observations.
Look beyond the pooled percentage. Overall coverage matches the nominal 90%, but the reduced-regimen group has lower observed coverage. The reviewer should examine the locations of its six uncovered observations, the observed and predicted concentrations, and whether interval construction and data handling were applied consistently. This pattern warrants investigation; these small counts alone do not establish a definitive difference between regimens.
Check what the interval includes. Because the target is an individual trough concentration, the reviewer should verify that the predictive distribution represents individual-level variability, not just uncertainty in the mean prediction. The observed coverage is evidence about prediction performance for this dataset and target; it does not automatically validate predictions at other times, doses, or populations.
Compare the Evidence and Diagnose Failure Modes
A strong review asks whether the method answers the actual question, whether its assumptions and design are defensible, and whether the reported results reveal relevant failure patterns. The following comparison helps keep the evidential roles distinct.
| Evidence | Main question | Key review checks | Important limitation |
|---|---|---|---|
| Predictive check | Are observed data or summaries compatible with model-simulated replicates under this design? | Simulation design, matched summaries, bins, number of simulations, patterns of departure | Often uses the fitting data; compatibility is not independent validation |
| Bootstrap | How do estimates or predictions vary under the stated resampling and refitting scheme? | Resampling unit, failed fits, successful replicate count, interval construction | Resampling uncertainty does not establish predictive adequacy in new data |
| External validation | How well do prespecified predictions perform in independent relevant data? | Independence, relevance of population and design, prediction target, coverage and patterns | Evidence is limited to the evaluated data and prediction task |
Common warning signs include simulation designs that omit key features of the observed study, unequal or sparse bins, intervals whose target is not stated, repeated observations counted as though independent, and summaries that report pooled coverage without revealing important patterns. Also check whether the number of simulated datasets is adequate for the displayed quantiles and whether the analysis reports unsuccessful bootstrap refits. A plot can appear reassuring while answering a different question from the one a decision requires.
Common mistake: “The model is validated because 90% of external observations fell inside 90% prediction intervals.” The correction is that this is one useful coverage result, not a complete validation claim. Review whether the data are independent and relevant, whether the intervals represent individual outcomes, whether coverage is patterned by time or design, and whether predictions are systematically high or low. Even exact nominal coverage in one dataset does not prove adequate performance for every intended use.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A visual predictive check displays an interval for simulated medians by time bin. What quantity does that interval describe, and what would be a mismatch if the observed points were individual concentrations?
- In a bootstrap with 250 resamples, 225 fits converge. What proportion converged, and what should a reviewer ask about the other fits?
- In an independent dataset, 45 of 50 observations are inside nominal 90% prediction intervals. Calculate coverage and state one reason this result is not sufficient by itself.
- Why does a bootstrap based on the fitting dataset not constitute external validation?
Solutions. (1) It describes the simulated distribution of the median across replicated datasets in each bin; comparing individual observed concentrations to it mismatches the statistic and interval target. (2) \(225/250=0.900\), or 90%, converged. Review the causes and pattern of the 25 failures and whether excluding them biases the reported bootstrap distribution. (3) \(45/50=0.900\), or 90% coverage. Coverage alone can conceal directional bias, and its relevance depends on independent, appropriate data and correctly constructed intervals. (4) The bootstrap resamples the fitting data to examine variation under a resampling scheme; it does not evaluate predictions in a separate, independent dataset.