Calibration of Individual Forecasts Across Patients

Calibration Is a Property of Forecasts Across Patients

A patient-specific forecast may be a predictive distribution or an interval for a future concentration, conditional on the dosing history, covariates, and concentrations available when the forecast is made. Its uncertainty can differ from one patient to another. Calibration asks whether forecasts and subsequently observed outcomes agree at the stated frequency across an appropriate set of patients. It is not the same as asking whether one patient’s observed concentration falls inside that patient’s interval.

For a meaningful evaluation, forecasts must be recorded before the future concentration is observed, and the target must be defined consistently. For example, the target might be the next trough concentration measured under a specified dosing schedule. The predictive distribution should represent uncertainty relevant to that target, including assay or residual variability when the forecast is intended to predict a measured concentration. Comparing an observed concentration with an interval for a different target, such as a latent typical concentration, does not evaluate the intended forecast.

The word marginally matters: the target is the proportion of outcomes covered across patients, averaged over the population and forecast situations represented in the evaluation. A well-calibrated overall percentage can still conceal poor performance in a clinically important subgroup or at a particular forecast horizon. Calibration is also distinct from sharpness. Two methods might both achieve approximately 90% coverage, while one produces substantially wider intervals. Coverage alone cannot establish that forecasts are as informative as they could be.

Estimating Coverage and Its Uncertainty

If there is one forecast-and-outcome pair per independent patient, the indicators \(I_i\) can be treated as Bernoulli observations with coverage probability \(q\). The empirical coverage \(\hat{q}\) estimates \(q\). Its approximate standard error is \(\sqrt{\hat{q}(1-\hat{q})/n}\), but this approximation can be unreliable with small samples or coverage near zero or one. A Wilson interval is a useful alternative for expressing sampling uncertainty.

For a nominal confidence level with standard normal critical value \(z\), the Wilson interval is

This interval follows by inverting the score test for a binomial proportion: retain values \(q\) for which \((\hat{q}-q)/\sqrt{q(1-q)/n}\) is no farther than \(z\) from zero, then solve the resulting quadratic inequality in \(q\). It quantifies uncertainty about the coverage rate under the binomial sampling assumptions; it does not turn a nominal interval into an individual guarantee.

Standardized Prediction Errors

Coverage reduces each forecast to covered or not covered. Standardized prediction errors retain more information about the direction and magnitude of discrepancies. Let \(m_i\) and \(s_i\) be the predictive mean and predictive standard deviation for patient \(i\)’s future observed concentration. Define the standardized prediction error as

A positive value means the observed concentration exceeded the forecast mean; a negative value means it was below. The scale is the forecast’s predictive standard deviation, in the same units as the concentration, so \(z_i\) is dimensionless. If the predictive distributions are correctly specified and approximately normal, their standardized errors should, across comparable forecasts, have mean near zero and standard deviation near one. These are diagnostic reference patterns, not requirements that a small finite sample match exactly. With skewed or otherwise non-normal predictions, a mean and standard deviation near zero and one do not by themselves establish calibration.

Stratified Summaries Reveal Conditional Miscalibration

Overall coverage averages over the mix of patients and forecast conditions in the validation set. To check whether that average hides a pattern, calculate coverage and standardized-error summaries in clinically meaningful strata. Possible strata include forecast horizon, concentration range, dosing situation, or a prespecified covariate group, provided the groups are defined in a way that is available and interpretable for the intended use. Report each stratum’s numerator and denominator as well as its percentage: a coverage of 80% from 4 of 5 patients is much less precise than 80% from 80 of 100.

Stratification should answer a focused question, not create many small groups until an apparently unusual result appears. When several strata are examined, chance fluctuations are more likely to produce an extreme rate somewhere. Small denominators also make subgroup rates unstable. A descriptive pattern can motivate further evaluation, but it is not by itself proof that a clinical characteristic caused the discrepancy.

Assumptions, Diagnostics, and Alternatives

A useful calibration assessment depends on more than arithmetic. Forecasts must be evaluated against the outcomes they were designed to predict, with the forecast information fixed before those outcomes are known. The validation patients should represent the population and conditions to which conclusions will be applied. For binomial uncertainty calculations, one forecast per independent patient is a simple setting. If there are multiple forecast outcomes per patient, those outcomes may be correlated; treating every row as independent can make uncertainty appear too small. Summaries and uncertainty calculations should then account for patient-level clustering, or use a clearly defined patient-level evaluation.

Look for systematic signs that calibration summaries are inadequate: many more observations outside intervals than their nominal rate allows; standardized errors concentrated above or below zero; unusually large standardized errors; or marked changes in coverage across important strata. Also check that intervals are not compared with outcomes on mismatched scales or time points, and that the reported interval includes the relevant observation error. A high overall coverage rate is not automatically desirable: very wide intervals can cover nearly everything while offering little decision value.

Coverage and standardized errors answer different questions. Coverage is directly interpretable as a frequency but discards how far an outcome lies beyond an interval. Standardized errors show direction and relative size but depend on a meaningful predictive center and scale; their normal reference interpretation is not universal. Point-error measures such as mean absolute error or root mean squared error summarize distances from a point forecast, but do not check whether predictive uncertainty is calibrated. For asymmetric predictive distributions, quantile-based checks or probability integral transform summaries can assess features that a standardized error alone may miss. These alternatives complement, rather than replace, assessment against the intended clinical target.

Population-level calibration also differs from fit to a single patient. A patient’s observed value inside a forecast interval is one covered outcome, not evidence that the forecasting method is calibrated. Conversely, one observed value outside an interval does not establish population-wide failure. Patient-level predictive checks assess whether a particular patient’s observations are compatible with that patient’s forecast; calibration across patients evaluates whether forecast performance has the stated frequency over many patients. The earlier material on patient-level posterior predictive checks addresses the former perspective. Neither perspective should be substituted for the other.

Practical Interpretation

A concise report should identify the forecast target, nominal interval level, number of independent patients, number covered, empirical coverage, and uncertainty around that estimate. It should also state how repeated outcomes were handled and show prespecified strata relevant to use. Standardized errors can be summarized by their mean, spread, and notable extremes, with care about distributional assumptions. These summaries support a measured conclusion: for example, “Observed coverage was close to nominal overall, but long-horizon forecasts showed lower empirical coverage in this small validation sample.” They do not justify claiming that every patient’s forecast is reliable or that a particular subgroup difference is causal.