Calibration of Individual Predictive Intervals

What Interval Calibration Measures

An individual posterior predictive interval summarizes uncertainty about a future concentration for a particular person, given that person’s available data and a forecasting model. If the interval is labelled 90%, its nominal coverage is 90%: across an appropriate set of forecast occasions generated under the conditions the interval is intended to represent, about 90% of the future concentrations should fall inside the intervals. Calibration asks whether this stated coverage is borne out by evaluation.

The evaluation must define its target. A forecast may concern a trough after a dose change, a concentration at a fixed time after dosing, or another specified occasion. Coverage pooled across all such occasions can conceal poor performance in a clinically important subgroup or time window. Moreover, several forecasts from one person are not generally independent: they share that person’s data, parameters, and clinical history.

The denominator determines the question being answered. If every occasion receives equal weight, a person with many evaluated concentrations contributes more than a person with few. To estimate coverage for an average person instead, first compute each person’s coverage rate and then average those rates, giving each person equal weight. Neither target is universally preferable; the evaluation should match how forecasts will be used.

Population-average calibration does not imply that every person has the same long-run coverage. For example, intervals might cover too rarely for people with impaired clearance and too often for those with typical clearance, while pooled coverage happens to be nominal. A single future concentration per person cannot establish that person’s long-run coverage rate. Individualized interval labels describe predictions conditioned on individual data; empirical calibration still requires a clearly defined collection of forecast outcomes.

Assumptions and a Calibration Justification

A useful evaluation requires more than counting whether observations fall inside intervals. The forecast must be frozen before the evaluated outcome is known, the outcome and interval must refer to the same concentration definition and occasion, and the evaluated cases must represent the target population or be appropriately weighted to it. If forecasts are updated using the outcome being evaluated, the assessment is optimistic and no longer tests prospective prediction.

There is a precise model-based reason for expecting nominal coverage under ideal conditions. Suppose the individual information \(D_i\) and future concentration \(C_i\) are generated from the joint prior-predictive process of a correctly specified Bayesian model. Given \(D_i\), let \(F_i\) be the posterior predictive cumulative distribution function for \(C_i\). If this predictive distribution is continuous and the interval uses its exact equal-tailed quantiles, then the interval endpoints are \(F_i^{-1}(\alpha/2)\) and \(F_i^{-1}(1-\alpha/2)\). Under the joint process, \(F_i(C_i)\), conditional on \(D_i\), is uniform on \([0,1]\). Therefore:

Averaging this conditional result over \(D_i\) also gives coverage \(1-\alpha\) under the joint prior-predictive process. This argument depends on the future outcome actually following the model’s conditional predictive distribution and on continuity for the stated quantile calculation. For a distribution with atoms, ordinary quantile intervals can have coverage greater than their nominal level. For example, a central 90% interval for a Bernoulli(0.5) outcome is \([0,1]\) under the usual quantile definition and covers with probability 1. Randomized quantile procedures can attain nominal coverage under an appropriate convention, but ordinary nonrandomized interval coverage need not do so.

This result is a model-based reference, not a guarantee of calibration in clinical use. Real patients may differ from the prior, the observation model may be misspecified, and data collection may select which concentrations are observed. Simulation from a specified model checks behavior under that model’s assumptions. Held-out clinical outcomes assess predictive performance in the evaluated data and population, subject to sampling uncertainty and data quality.

Evaluating Held-Out Forecasts

For independent forecast occasions, the estimated coverage is a sample proportion. A simple approximate standard error is \(\sqrt{\hat{c}(1-\hat{c})/N}\). A confidence interval for coverage describes uncertainty about the population coverage rate under the sampling assumptions; it is not a predictive interval for a future concentration. When occasions are clustered within people, treating all occasions as independent generally understates uncertainty. Use an analysis that preserves patient-level clustering, such as resampling individuals, or a suitable cluster-robust method.

When each person contributes repeated occasions, report the number of people as well as the number of forecasts. Inspect coverage by occasion type, time since dose, concentration range, and relevant patient characteristics, but define important subgroups in advance where possible. Numerous post hoc subgroup comparisons can produce chance findings. A pooled estimate remains useful as a summary, but it should not be the only diagnostic when clinical decisions depend on particular subgroups or time points.

Simulation as a Model-Based Check

A simulation evaluation can test whether a forecasting procedure attains its stated coverage when data are generated according to a specified model. For each simulated individual, generate the available information, fit or update the posterior using only that information, construct the future predictive interval, generate the future concentration, and record whether it is covered. Repeating this process estimates average coverage under the simulation assumptions. To assess a particular clinical deployment, the simulated population, dosing histories, sampling occasions, and observation process should reflect the intended setting.

The Monte Carlo standard error quantifies simulation noise in the estimated coverage. It is approximately \(\sqrt{\hat{c}(1-\hat{c})/N}\) when the simulated evaluation units are independent. If a simulation generates multiple occasions per person, dependence must again be reflected in uncertainty calculations. A simulation that reuses a single parameter value or otherwise fails to reproduce the intended variation may answer a narrower question than expected.

Diagnostics, Alternatives, and Common Pitfalls

Coverage should usually be examined at more than one nominal level, such as 50%, 80%, and 90%. A pattern of undercoverage at several levels suggests that intervals are too narrow overall; correct 90% coverage paired with poor 50% coverage can indicate a different shape or tail-calibration problem. Examine coverage across clinically meaningful occasions and patient groups, alongside interval widths. Coverage alone rewards overly wide intervals, so an interval can be calibrated yet uninformative. Conversely, a narrow interval is not useful if it misses too often.

Calibration complements, rather than replaces, diagnostics of individual residuals or posterior predictive fit. Residual diagnostics can reveal structured forecast errors; predictive checks can compare features of replicated and observed data; interval coverage directly asks whether specified predictive intervals contain held-out outcomes at the stated frequency. Point-prediction error evaluates a different target and cannot establish interval calibration. A fuller evaluation pairs coverage with interval width and relevant scoring rules, while keeping the target outcome and forecast conditions consistent.

A common mistake is to count every concentration occasion as an independent observation and then report a narrow confidence interval for coverage. The reasoning is that there are \(N\) intervals, so the standard error must be \(\sqrt{\hat{c}(1-\hat{c})/N}\). That formula assumes independent Bernoulli coverage indicators. When a person contributes repeated occasions, the indicators may be correlated; the correction is to account for clustering, for example by resampling whole people, and to report both the number of people and occasions.

Another mistake is to conclude that a model is clinically validated because its own simulation yields nominal coverage. Simulation establishes behavior under the data-generating assumptions chosen for the simulation, not under all possible patient populations or observation processes. Held-out evaluation provides a separate empirical check, but it too is limited by sample size, representativeness, outcome availability, and the specific forecast occasions assessed. If measurements below quantification limits or missing concentrations are common, define whether evaluation concerns the latent concentration or the recorded observation and use a compatible observation model and assessment.