Failure Analysis of Overconfident Sparse-Data Forecasts

When a Forecast Is More Certain Than Its Evidence

An individual forecast can look precise because a computation returns a narrow interval. That appearance is not evidence that the available data identify the patient’s pharmacokinetics well. With only a few observations, a narrow forecast may instead reflect a strong prior, an overly simple error model, or dose information treated as exact when it is not. The important question is not only how the interval was calculated, but whether it includes the important sources of uncertainty.

Let \(\theta\) denote an individual’s model parameters, \(D\) the observed concentration data, and \(H\) the dose history and other inputs. A forecast \(Y_*\) is conditional on the model and the information used to compute it. Its posterior predictive distribution averages over uncertainty in the individual parameters; if inputs such as dose time are uncertain, that uncertainty should also be represented. The examples below use deliberately simple hypothetical models to make the consequences traceable.

A basic decomposition helps locate missing uncertainty. Conditional on parameters \(\theta\) and inputs \(H\), suppose a future observation has mean \(m(\theta,H)\) and residual variance \(\sigma^2(\theta,H)\). The posterior predictive variance is

This follows by writing \(Y_*-E(Y_*\mid D)\) as the sum of \(Y_*-E(Y_*\mid\theta,H,D)\) and \(E(Y_*\mid\theta,H,D)-E(Y_*\mid D)\). The first term has conditional mean zero, so its cross-product with the second term has expectation zero. Taking variances leaves the expected conditional variance plus the variance of conditional means. Thus a calculation that reports only uncertainty in the fitted mean omits residual variability; one that fixes uncertain inputs also omits variation in the forecast mean induced by those inputs.

Weak Identification and Restrictive Priors

A parameter is weakly identified by sparse data when substantially different parameter values give similar predictions for the observations that were collected. Those values can nevertheless imply very different predictions at the future time or dose of interest. A small residual around the observed sample does not rule out this problem: the data may constrain only one combination of parameters, not each parameter separately.

A prior can regularize a weakly identified parameter, but the resulting certainty may come mainly from the prior. In a normal illustration, suppose one noisy observation \(Y=14\) has \(Y\mid\theta\sim N(\theta,4)\), and a prior is \(\theta\sim N(10,0.04)\). Under this conjugate model, the posterior variance is \((1/4+1/0.04)^{-1}\), and its mean is the precision-weighted average of the prior mean and observation. These results rely on the stated normal likelihood, known observation variance, and normal prior; they do not establish that the prior is appropriate for a particular patient.

The assumptions behind this calculation are especially consequential with sparse data: the observation model is unbiased and normal with known variance, the prior is normal and independent of the observation before updating, and the parameter is represented adequately by this scalar model. If those assumptions are doubtful, the arithmetic can be exact while the resulting uncertainty is misleading. Compare plausible prior scales and inspect whether the forecast changes materially; report the influence of the prior rather than presenting a prior-dominated interval as if it were data-driven.

Residual Error and Predictive Uncertainty

Even if parameter uncertainty is small, a future measured concentration varies around its model-predicted mean. When a model’s residual error is too small, predictive intervals contract. This can happen if variability is estimated from too few observations, if an error component is omitted, or if residual variability changes over concentration or time but is treated as constant. Posterior predictive checks and residual diagnostics can reveal discrepancies, but passing a check with very sparse data does not prove that the error model is correct.

A predictive interval should include both uncertainty about the expected concentration and observation-level variability when the target is a future measured concentration. For a latent model-predicted mean, residual observation error may not belong in the interval, but uncertainty in the individual parameters still does. State which target is being forecast: a latent exposure, an individual expected concentration, or a future measured sample. Confusing these targets can make a valid interval answer the wrong question.

Uncertain Dose History Is Part of the Forecast

A dose history is often treated as known, but a dose may have been delayed, missed, taken at a different time, or recorded with uncertainty. If different histories imply different concentrations, using a single convenient history suppresses input uncertainty. Averaging histories into one nominal input is not generally equivalent to averaging their predictive distributions; nonlinear pharmacokinetic models make this especially important.

Failure Analysis in Practice

A useful analysis asks which modeling choices control the apparent precision, then checks whether those choices are supported by the individual’s data and known inputs. Practical checks include the following:

• Inspect parameter dependence and sensitivity. Large posterior correlations, long ridges, or substantial changes in the forecast under plausible parameter or prior alternatives indicate weak identification. A narrow marginal interval for one parameter alone can hide a consequential joint relationship.
• Check residual behavior. Look for systematic time or concentration patterns, outliers, and dispersion inconsistent with the assumed error model. With sparse observations, emphasize that diagnostics have limited power rather than interpreting a lack of visible pattern as proof of adequacy.
• Vary uncertain inputs explicitly. Recalculate or integrate forecasts over plausible dose amounts and times. Compare the result with the forecast based on a single assumed history, and record how input uncertainty affects the target prediction.
• Compare uncertainty targets. Distinguish intervals for parameters, latent individual means, and future observed concentrations. A narrow interval for one is not evidence that the others are narrow.
• Check predictive behavior and calibration. Posterior predictive checks, residual diagnostics, and calibration of predictive intervals address different aspects of model adequacy. None alone guarantees reliable individual forecasts, particularly outside the range of observed data.

These checks complement rather than replace one another. A posterior predictive check asks whether data generated under the fitted model can reproduce relevant features of observed data; calibration asks whether predictive intervals achieve their stated coverage across an appropriate set of predictions. Neither can identify an unrecorded dose history from no evidence, and both can be inconclusive for a single patient with very few samples.

The main alternative to a single plug-in forecast is to report a posterior predictive distribution that averages over parameter uncertainty and represents observation error. If dose history is uncertain, average over plausible histories as well. A maximum a posteriori (MAP) estimate or other point estimate is useful for a concise summary, but a forecast evaluated only at that estimate omits the spread in plausible values. Conversely, a more elaborate distribution is not automatically better: it still depends on the likelihood, prior, error model, and input information. When data are sparse, sensitivity analyses across defensible assumptions may be more honest and informative than a single narrow interval.

The limitation is fundamental: computation cannot create individual information absent from the observations or reliable external knowledge. Additional informative samples or better verified dose records may reduce uncertainty, but they do not guarantee that every parameter becomes identifiable. If a decision depends strongly on a forecast whose uncertainty is assumption-dominated, that dependence should be made explicit rather than concealed by numerical precision.