Deciding Whether an Individual Forecast Is Trustworthy

Trustworthiness Is a Decision-Specific Judgment

An individual forecast can be numerically precise yet unhelpful, or uncertain yet adequate for a low-stakes decision. Its trustworthiness therefore cannot be judged by the width of an interval alone. The relevant question is whether the forecast, given its data and assumptions, is dependable enough for the intended decision: for example, deciding whether a concentration is likely to fall outside a specified range or whether another measurement is needed.

Let \(y_i\) denote the observed concentrations for individual \(i\), \(D_i\) the recorded dosing and sampling history, and \(M\) the population PK model, including its structural, variability, and residual-error assumptions. Let \(\theta_i\) denote the individual parameters. A Bayesian forecast uses the posterior distribution \(p(\theta_i\mid y_i,D_i,M)\) together with the model to predict a future concentration or other quantity of interest. The assessment in this tutorial concerns whether that forecast is sufficiently supported for its stated use; it does not certify that the model is true.

A useful assessment keeps several questions distinct. Are the inputs accurate? Is there enough informative data to update the prior? How uncertain is the quantity that drives the decision? Do the observations agree reasonably with the prior and the model’s predictive implications? Are there clinical or design changes that make the historical model inappropriate for the forecast period? A reassuring answer to one question cannot compensate automatically for a serious problem in another.

What the Forecast Uncertainty Contains

A future concentration is uncertain for more than one reason. The individual parameters are uncertain after observing the available data; even if those parameters were known, residual variability can make an individual concentration differ from its model expectation. Uncertainty in dose amounts, dose times, adherence, or future dosing can contribute as well. The posterior predictive distribution integrates over these uncertainties that are represented in the model.

Here \(D_{\mathrm{future}}\) is the assumed future dose history. If that history is uncertain, a fuller prediction also averages over its plausible values. For a forecast conditional on a fixed future dose plan, the displayed distribution does not account for uncertainty that the plan will actually be followed. That distinction matters: a precise conditional forecast is not a guarantee about an uncertain real-world dosing course.

The law of total variance makes the two main sources of predictive spread explicit. Conditional on the individual parameters, the first term represents residual or process variability; the second represents variation in the model’s expected future value across plausible individual parameters.

This decomposition explains why a narrow posterior for a parameter does not necessarily imply a narrow prediction interval: residual variability may still be substantial. Conversely, a posterior mean forecast alone conceals how much probability lies on either side of a clinical threshold. The uncertainty measure must match the question. A prediction interval describes a range for a future observation under the model; a credible interval for a parameter describes uncertainty about that parameter conditional on the model and data. They are not interchangeable.

Assess the Information in the Individual Data

Data adequacy is not just a count of concentrations. Information depends on when samples were taken relative to dosing, the parameters that affect the observed profile, assay and residual error, and whether the concentrations span regions that distinguish competing parameter values. Several samples clustered in a region where the model predictions barely change with clearance may provide little information about clearance. A smaller number of well-timed samples can sometimes be more informative, though sparse data demand caution.

The practical question is whether the likelihood from the individual’s observations materially informs the parameters relevant to the forecast, rather than merely producing a posterior estimate. This can be assessed by comparing prior and posterior distributions, examining uncertainty in the forecast-relevant parameter combinations, and considering whether the data provide leverage at the relevant times. Eta shrinkage, influential observations, and sparse-data limitations can help identify weak individual information, but none should be interpreted in isolation.

Check Predictive Agreement, Not Just Parameter Agreement

Prior-data agreement asks whether the observed patient information is surprising under the population prior and observation model. Predictive diagnostics ask whether the model can reproduce relevant features of the observed concentrations. These address related but different concerns. A patient can have a plausible parameter update while the model still fails to reproduce the concentration-time pattern, for example because of a structural mismatch or a recording error.

Posterior predictive checks compare observed data with data replicated from the model using the patient’s posterior distribution and relevant sampling and dosing conditions. Useful summaries may include residual patterns over time, the maximum deviation, or whether observations fall in predictive intervals. A check is more informative when its summary targets a plausible failure mode. A single observation inside a broad predictive interval is weak evidence; a systematic pattern or extreme discrepancy across relevant observations deserves investigation. Because the data also informed the posterior, posterior predictive checks can be less sensitive to misspecification than checks based on genuinely held-out observations.

Dose-History Quality and Model Applicability

A forecast conditions on a dose history. Incorrect dose amounts, administration times, missed doses, infusion durations, or sample times can distort the inferred parameters and the predicted concentration. Record quality should therefore be assessed as part of the forecast, not as clerical detail. Distinguish what is documented from what is assumed. When adherence or timing is uncertain, consider whether plausible alternatives change the forecast enough to change the decision.

A forecast also assumes that the model remains applicable over the forecast period. Changes in renal or hepatic function, interacting medicines, body size, disease state, or other modelled covariates can matter if they alter relevant parameters. If the model assumes stable clearance but a patient’s clearance is changing, a narrow interval conditional on stable clearance may give false reassurance. A changing covariate or nonlinear process should be represented only if the model and available information support it; otherwise, the limitation should remain explicit.

A practical sensitivity assessment asks whether reasonable alternative assumptions about uncertain inputs or model components alter the clinically relevant prediction. For example, calculate the forecast under two plausible dose-time histories. If one implies a low probability of crossing a threshold and the other implies a high probability, the forecast is not robust to the record uncertainty. This is more informative than reporting one precise result while hiding an uncertain input.

A Structured Assessment and Its Limits

A compact assessment can be organized into linked checks. It is a reasoning aid, not a validated numerical score. The conclusion should say what is supported, what remains uncertain, and which decision the forecast can or cannot support.

The main alternative to this structured synthesis is to rely on a single indicator: a narrow interval, low shrinkage, a good fit statistic, or agreement with the population prior. Each can be useful, but each answers only part of the question. A narrow interval can be conditional on a wrong dose record; low shrinkage does not rule out structural misspecification; a good in-sample fit does not guarantee future calibration; and close prior agreement may mean that sparse data contributed little. Conversely, a visible prior-data difference is not inherently a failure: it may represent real individual variation when the data and model support it.

Calibration across patients can reveal whether forecasts are systematically too high, too low, or overconfident in a population. It cannot establish that a particular patient’s dose record is correct or that this patient’s forecast is adequate for a specific decision. Individual diagnostics and population-level calibration are complementary. Neither removes uncertainty about unrepresented model errors or clinical changes.

Common Mistake: Treating Precision as Trustworthiness

• Wrong reasoning: “The forecast interval is narrow, so the patient’s forecast is reliable.”
• Correction: The interval may be narrow only under the assumed model and recorded history. Verify those assumptions and check predictive agreement before interpreting its width.
• Wrong reasoning: “The observed concentrations agree with the prior, so the model is validated.”
• Correction: Agreement is only absence of an obvious conflict for the data examined. It does not establish predictive adequacy, especially when the observations are sparse or weakly informative.
• Wrong reasoning: “One extreme residual proves that the model is misspecified.”
• Correction: First check data, dose, and sample records and consider expected variability. A substantial unexplained pattern raises concern, but a diagnostic discrepancy is evidence to investigate rather than a standalone diagnosis.