Why Sparse Data Can Make Forecasts Look More Certain Than They Are
A patient-specific forecast can look precise because a model returns a single estimate, a smooth concentration-time curve, or a narrow interval. None of these appearances establishes that the patient's future concentration is known precisely. With very few measurements, the data may provide limited information about the patient's pharmacokinetic parameters. A forecast can also be too narrow if it treats estimated population parameters as known or omits correlations among uncertain parameters.
The relevant target is the distribution of a future concentration given the patient's available data and the model. It must account for uncertainty that can affect that future result, not just uncertainty in one fitted patient estimate. The earlier material on patient-specific prediction intervals and sequential Bayesian updating provides the context for this distinction; here the focus is on diagnosing and correcting false precision.
A useful distinction is between a patient parameter estimate and a future observation. Even if the patient parameter were known, the future observation can vary because of residual or assay error. If the patient parameter is uncertain, that uncertainty also affects the future concentration. If the population model parameters are uncertain, their uncertainty can contribute as well. A forecast that includes only some of these components should be described as conditional on the omitted quantities, not presented as an unconditional patient-level prediction interval.
Decomposing Predictive Uncertainty
Let \(Y_f\) be a future concentration, \(y\) the patient's observed data, \(\eta\) the patient's individual effects, and \(\theta\) the population parameters. The law of total variance separates uncertainty in the future observation into a conditional component and a component due to uncertainty in its conditional mean:
The first term includes future observation variability, such as residual error, conditional on the patient and population parameters. The second term reflects uncertainty about those parameters after observing \(y\). The decomposition is exact when the expectations and variances exist; it does not require a normal model. In practice, evaluating it requires a specified model for the observations and a suitable representation of parameter uncertainty.
For a locally linear forecast on a scale where an approximate normal distribution is reasonable, let the forecast depend on uncertain quantities collected in a vector \(q\). If its local sensitivity vector is \(g\), then the parameter-related variance is approximately \(g^\mathsf{T}\Sigma g\), where \(\Sigma\) is the covariance matrix of \(q\). This expression includes off-diagonal covariances. Setting those covariances to zero without justification changes the variance, and can change it in either direction.
Worked Examples: Three Routes to False Precision
Worked Example: Sparse Observation Leaves Patient Uncertainty
Consider a hypothetical one-compartment forecast summarized on the log-concentration scale. Suppose the patient effect has prior distribution \(\eta\sim N(0,0.16)\), one log-concentration observation has residual error variance \(0.09\), and the observed residual relative to the population prediction is \(0.30\). For this illustrative calculation, assume the observation is \(\eta+\epsilon\), with independent \(\epsilon\sim N(0,0.09)\). A future log concentration is \(\log(8)+\eta+\delta\), where the future residual \(\delta\sim N(0,0.04)\) is independent of the observation and \(\eta\). The concentration units are mg/L.
These assumptions give a normal prior, a linear observation model with normal error, and independent observation and future residual errors. Normal updating gives posterior variance and mean:
The log future concentration therefore has mean \(\log(8)+0.192\) and variance \(0.0576+0.04=0.0976\). Its approximate central 95% predictive interval is calculated on the log scale, then exponentiated:
The interval is wide because one observation has not eliminated uncertainty about \(\eta\), and future residual variation remains. If the calculation instead fixes \(\eta\) at its posterior mean, its log-scale variance is only \(0.04\). That conditional interval is \(8\exp(0.192\pm1.96(0.20))=[6.55,14.3]\) mg/L. It is not a full predictive interval for this model: it conditions on the patient effect as though known. The narrower result can be reported as conditional, but not as though it captured all uncertainty after one sparse observation.
Worked Example: Treating Population Parameters as Fixed
Suppose, hypothetically, that the log of a future concentration is centered at \(\log(5)\). The posterior standard deviation of the patient's log-scale effect is \(0.15\), and future residual standard deviation is \(0.10\). Assume these components are independent and normal. Separately, suppose the relevant population parameter has an approximately normal estimation uncertainty with standard deviation \(0.20\), independent of the patient effect and residual error. These are illustrative uncertainty summaries, not study results.
If the population parameter is treated as fixed, the log-scale variance is \(0.15^2+0.10^2=0.0325\), so the standard deviation is \(\sqrt{0.0325}=0.1803\). The approximate 95% interval is:
This interval is conditional on the population parameter being fixed. If its estimation uncertainty is propagated under the stated independence assumption, the log-scale variance is \(0.15^2+0.10^2+0.20^2=0.0725\), with standard deviation \(0.2693\). The corresponding interval is:
The second interval is wider because it includes another source of uncertainty. This calculation does not mean that population parameter uncertainty must always be added as an independent variance: the assumed independence is explicit here, and correlations require a joint calculation.
Worked Example: An Omitted Correlation Makes an Interval Too Narrow
For a hypothetical future log concentration, suppose the uncertain contribution is \(a+b+\epsilon\), where \(a\) and \(b\) are population-parameter contributions with means zero, variances \(0.04\) and \(0.0225\), and covariance \(0.025\). Let the independent future residual \(\epsilon\) have variance \(0.01\), and let the forecast's central concentration be 8 mg/L. The correlation between \(a\) and \(b\) is \(0.025/(0.20\times0.15)=0.8333\). The covariance is admissible because its magnitude is less than \(0.20\times0.15=0.0300\).
Because the forecast depends on the sum \(a+b\), the correct log-scale variance includes twice their covariance:
The log-scale standard deviation is \(0.35\), so the approximate 95% interval is \(8\exp(\pm1.96(0.35))=[4.03,15.9]\) mg/L. If the covariance is incorrectly set to zero, the variance becomes \(0.04+0.0225+0.01=0.0725\), giving \(8\exp(\pm1.96\sqrt{0.0725})=[4.72,13.6]\) mg/L. The latter is falsely narrow under the specified joint distribution. With negative covariance, omitting it could instead produce an interval that is too wide; “ignore covariance” is not a generally conservative choice.
Diagnostics, Alternatives, and Reporting
False precision is not diagnosed by the width of an interval alone. A narrow interval can be justified by highly informative data and a well-supported model, while a wide interval can reflect genuine uncertainty. Examine how the reported interval changes when plausible sources of uncertainty are included, and check whether the assumptions used to compute it are supported.
- Check information in the observations. Few concentrations, limited timing information, or measurements taken in a region where different parameter values predict similar concentrations may leave patient parameters weakly distinguished. A precise point estimate does not repair that lack of information.
- Check what was held fixed. Identify whether the calculation conditions on population estimates, patient effects, dose history, or other inputs. An interval conditional on fixed values may be useful, but it answers a narrower question.
- Check joint uncertainty. If more than one parameter contributes to the forecast, inspect the relevant covariance matrix or joint posterior draws. A diagonal-only approximation can misstate uncertainty when the forecast responds to correlated parameters.
- Check sensitivity and calibration. Compare intervals under defensible alternative uncertainty assumptions, and assess forecast calibration across patients when suitable data are available. Persistent undercoverage indicates that intervals are too narrow on average for that setting, though it does not by itself identify the cause.
- Check the scale and distribution. A normal approximation on the log scale yields a lognormal concentration distribution. Its central interval is asymmetric on the concentration scale. If the approximation or model is poor, simulation from the specified predictive distribution may be preferable, but simulation cannot fix omitted uncertainty or a misspecified model.
Common approaches answer different questions. A plug-in forecast substitutes point estimates and is useful as a central summary, but a plug-in point alone does not quantify predictive uncertainty. A conditional interval can include residual error while fixing parameter estimates; it is appropriate only when that conditioning is clear. A full predictive calculation propagates the relevant uncertainty jointly, by analytic formulas when suitable or by drawing parameters and future observations from their conditional distributions. Resampling or other approximations may also be considered, but with very sparse patient data they can be unstable and remain dependent on their assumptions.
No interval method guarantees reliable coverage when the structural model, residual model, dosing record, or population distribution is inappropriate. Sparse data also limit how much patient-specific information can be recovered, regardless of computational sophistication. A forecast should therefore separate the estimated central value from the uncertainty statement and disclose important conditioning choices.
Check Your Understanding
Use the stated assumptions in each question. Solutions follow the questions.
- A forecast interval includes future residual error but fixes the patient's individual effect at its estimate. What uncertainty is omitted, and how should the interval be described?
- In a log-scale forecast, two uncertain parameter contributions have variances \(0.01\) and \(0.04\), covariance \(0.01\), and an independent residual variance of \(0.01\). Find the total variance. What variance would result from incorrectly setting the covariance to zero?
- Why does increasing the number of simulated forecasts not correct an interval that omits uncertainty in population parameters?
- A 95% prediction interval is described as conditional on fixed population parameters. Does it claim 95% coverage over repeated estimation of those parameters? Explain.
Solutions. (1) It omits posterior uncertainty in the patient's individual effect. Describe it as conditional on that effect fixed at its estimate, rather than as a full patient-level predictive interval. (2) The total variance is \(0.01+0.04+2(0.01)+0.01=0.08\). Omitting covariance gives \(0.01+0.04+0.01=0.06\), which is too small for this positive covariance. (3) More simulations reduce numerical Monte Carlo error for the distribution being simulated; they do not add an uncertainty source that the simulation model excludes. (4) No. It is a 95% interval under the stated conditioning on fixed population parameters; it does not include their repeated-estimation uncertainty and therefore does not claim 95% coverage across that uncertainty.