Assessing Calibration of Individual Prediction Intervals

Why Assess Individual Prediction Intervals?

An individual prediction interval is useful only if its stated uncertainty is credible for the forecasts to which it is applied. A nominal 90% interval is intended to contain the future observation in about 90% of comparable forecasting cases. Checking this claim requires observations that were not used to construct or tune the intervals. In an NLME forecasting setting, each case pairs an individual forecast made using the information available at a specified time with a later, held-out outcome.

Coverage alone is not enough to compare forecasts. An interval can cover nearly every observation simply by being very wide, while an interval that is narrow but misses often may understate uncertainty. Calibration and interval width therefore answer different questions: does the observed frequency of inclusion accord with the nominal level, and how much uncertainty does the forecast report? An interval score combines width and penalties for misses to summarize both.

Definition: Suppose $m$ held-out forecast cases have observed outcomes $y_i$ and prediction intervals $I_i=[L_i,U_i]$, for $i=1,\ldots,m$, where $L_i$ and $U_i$ are the lower and upper endpoints. The empirical coverage is $$ \widehat{C}=\frac{1}{m}\sum_{i=1}^{m}\mathbf{1}\{L_i\leq y_i\leq U_i\}, $$ where $\mathbf{1}\{\cdot\}$ equals 1 when its condition is true and 0 otherwise. For a nominal $100(1-\alpha)\%$ interval, the target marginal coverage is $1-\alpha$.

The target is marginal over the population of forecast cases represented by the validation data. It does not mean that every individual interval has a $1-\alpha$ probability of containing its own outcome after both are known. Rather, before the held-out outcomes are observed, intervals constructed by the forecasting procedure should include outcomes at about the stated rate across comparable cases. A stronger diagnostic asks whether coverage is also close to nominal within relevant subsets, such as short- and long-horizon forecasts.

Conditions for a Meaningful Validation

Coverage summaries are interpretable only when the forecast and its validation outcome are defined consistently. The information used to form each interval must be available at that forecast time; the held-out outcome must not have been used to estimate, select, or tune the interval procedure. The outcome must also correspond to the stated target, including its time, dose scenario, and measurement scale. If the validation cases do not represent the future cases of interest, their empirical coverage may not describe future performance.

The nominal level is a property of the forecasting procedure under its assumptions, not a guarantee that any finite validation set will show exactly that percentage. Model misspecification, an unrepresented change in the population or clinical process, or incorrect assumptions about residual variability can all produce systematic miscalibration. Even when a procedure is well calibrated in the population, a small validation sample can yield a noticeably different observed rate.

For a simple uncertainty calculation, if the $m$ coverage indicators are independent and each has the same coverage probability $C$, then the standard error of the empirical coverage is approximately $\sqrt{\widehat{C}(1-\widehat{C})/m}$. This approximation can be poor for small samples or rates near 0 or 1. Forecasts for the same individual are often dependent, so treating every forecast time as an independent case can understate uncertainty. In that setting, summarize at the independent-subject level or use a resampling method that resamples subjects rather than individual forecast rows.

Worked Example: Overall Coverage and Its Sampling Uncertainty

Suppose 20 hypothetical patients each contribute one held-out concentration forecast at a specified time. Each forecast is a central 90% prediction interval. Seventeen of the 20 observed concentrations fall inside their respective intervals. The empirical coverage is

\widehat{C}=\frac{17}{20}=0.85=85\%.

The observed coverage is 5 percentage points below the nominal 90%, but the difference is not by itself evidence of systematic miscalibration. Under the simplifying assumption of independent cases with common coverage, the estimated standard error is

SE(\widehat{C})\approx \sqrt{\frac{0.85(1-0.85)}{20}} =\sqrt{0.006375}=0.0798,

or about 8.0 percentage points. For another view of the small-sample uncertainty, a 95% Wilson interval for the underlying coverage is approximately 0.639 to 0.947. Thus, this validation set is compatible with a broad range of coverage rates, including 0.90. The result is a noisy estimate, not proof that true coverage is 85% or that the nominal target has been met. If the 20 forecasts came from repeated observations on fewer patients, the independence calculation would be inappropriate without accounting for within-patient dependence.

Interval Scores: Coverage and Width Together

For a central $100(1-\alpha)\%$ interval, the interval score assigns a cost to its width and an additional cost when the observed outcome lies outside it. For case $i$, the score is

IS_{\alpha}(L_i,U_i;y_i) =(U_i-L_i) +\frac{2}{\alpha}(L_i-y_i)\mathbf{1}\{y_i<L_i\} +\frac{2}{\alpha}(y_i-U_i)\mathbf{1}\{y_i>U_i\}.

A smaller score is preferred when comparing forecasts for the same outcomes and nominal level. The first term, $U_i-L_i$, penalizes width. If the outcome falls below the interval, the score adds $2/\alpha$ times the distance from the lower endpoint; if it falls above, it adds the same multiple of the distance beyond the upper endpoint. A more distant miss costs more. The mean interval score is the arithmetic mean of the case-specific scores.

The factor $2/\alpha$ gives misses substantial weight at high nominal coverage. For a 90% interval, $\alpha=0.10$, so the multiplier is 20. The score is expressed in the units of the outcome: for concentration intervals, it has concentration units. Comparisons should use the same outcome scale and nominal level; scores across different scales or nominal levels are not directly interchangeable.

Why the Interval Score Has This Form

For a continuous predictive distribution, the expected interval score is minimized when the endpoints are its $\alpha/2$ and $1-\alpha/2$ quantiles. This property explains why the score penalizes both width and misses: narrow intervals reduce the width term, but endpoints placed at the wrong quantiles incur expected miss penalties.

To see the endpoint argument, hold $U$ fixed and consider the expected score as a function of $L$. Differentiating with respect to $L$, where the predictive cumulative distribution function $F$ is continuous, gives

\frac{\partial E[IS_{\alpha}]}{\partial L} =-1+\frac{2}{\alpha}F(L).

The derivative is zero when $F(L)=\alpha/2$. Similarly, differentiating with respect to $U$ gives

\frac{\partial E[IS_{\alpha}]}{\partial U} =1-\frac{2}{\alpha}\{1-F(U)\},

which is zero when $F(U)=1-\alpha/2$. These are the central interval quantiles. The argument assumes a continuous predictive distribution; with discrete distributions or point masses, a minimum can occur at quantile endpoints where the derivative is not defined, but the quantile interpretation remains the relevant one.

Worked Example: Calculating a 90% Interval Score

Consider four hypothetical held-out concentrations, in mg/L, with their 90% prediction intervals:

Case	Interval (mg/L)	Observed concentration (mg/L)
1	[8, 12]	10
2	[18, 22]	23
3	[28, 32]	26
4	[38, 42]	41

For a 90% interval, $2/\alpha=2/0.10=20$. Case 1 is covered and has width $12-8=4$, so its score is 4. Case 2 is 1 mg/L above its upper endpoint, giving

IS_{0.10}= (22-18)+20(23-22)=4+20=24\ \text{mg/L}.

Case 3 is 2 mg/L below its lower endpoint:

IS_{0.10}= (32-28)+20(28-26)=4+40=44\ \text{mg/L}.

Case 4 is covered and has score $42-38=4$. Two of four observations are covered, so empirical coverage is $2/4=50\%$. The mean interval score is

\overline{IS}_{0.10}=\frac{4+24+44+4}{4}=19\ \text{mg/L}.

The two missed observations are penalized according to how far they fall outside their intervals; the score does not treat a barely missed value as equivalent to a distant one. This four-case result is illustrative only: such a small sample cannot provide a precise calibration assessment.

Stratify by Forecast Conditions

Overall coverage can conceal important failures. Forecast uncertainty often changes with horizon, and individual forecasts can be less informative when the available concentration history is sparse. Report coverage, mean width, and mean interval score within scientifically relevant strata, such as prespecified horizon bands or sampling-density groups. A useful table reports both the number of independent individuals or forecast cases and the number covered, so that a rate based on few cases is not mistaken for strong evidence.

Stratification should be planned and interpreted carefully. If forecast horizon and sampling density vary together, a difference between groups cannot identify which factor accounts for it. Where data permit, examine the factors separately or use a cross-classification with enough cases per cell. Avoid creating many tiny groups after seeing the results: their coverage rates will be highly variable and may invite overinterpretation.

Worked Example: Wide Intervals Can Still Be Calibrated

Suppose a hypothetical validation has 20 dense-history, near-horizon forecasts and 20 sparse-history, near-horizon forecasts. Both groups use nominal 90% intervals. Eighteen outcomes are covered in each group. The mean width is 4 mg/L in the dense-history group and 8 mg/L in the sparse-history group.

The empirical coverages are $18/20=0.90$ in both groups. The sparse-history intervals are twice as wide on average, but that width difference does not imply miscalibration. In this example the observed coverage in both groups matches the nominal rate; the wider intervals express less precise individual forecasts. Whether the groups’ underlying coverage rates are truly equal cannot be established from these finite samples alone.

Now suppose a further 20 hypothetical sparse-history, long-horizon forecasts have 14 covered outcomes and a mean width of 12 mg/L. Their coverage is $14/20=0.70$. Across all three groups, coverage is

\frac{18+18+14}{20+20+20} =\frac{50}{60}=0.833.

The pooled rate of 83.3% obscures the difference between the first two strata and the long-horizon sparse-history stratum. The latter group’s intervals are widest but also show the lowest empirical coverage. Width alone neither establishes nor rules out calibration; the held-out inclusion rate must be examined alongside it. Because horizon and sampling density may differ together here, the pattern is a warning to investigate, not proof that either factor caused undercoverage.

A Practical Validation Sequence

Freeze the forecast target and information set.
Specify the outcome, forecast time, horizon, and observations permitted in constructing each interval.

Reserve held-out cases.
Do not use their outcomes to tune the interval construction or choose a model and then report the same cases as an independent validation.

Summarize coverage and width.
For each nominal level, report the number covered, the total number of cases, empirical coverage, and typical interval width in outcome units.

Compare interval scores.
Calculate the mean score on the same held-out outcomes for each candidate procedure at the same nominal level.

Check relevant strata and uncertainty.
Examine horizon and sampling density, note small groups, and account for repeated forecasts within individuals when judging precision.

Diagnostics, Alternatives, and Limitations

A coverage rate well below nominal suggests undercoverage in the evaluated cases; a rate well above nominal may indicate conservative intervals, though it can also arise by chance. A low mean interval score is desirable only in a like-for-like comparison: it reflects the balance of width and misses, not coverage alone. Always inspect the coverage and width components separately so a score difference can be understood rather than treated as a black-box ranking.

Coverage-only summaries are simple and directly tied to the nominal claim, but they do not distinguish a tight interval from a very wide one. Mean width is useful for informativeness, but it says nothing about whether the interval captures outcomes often enough. The interval score combines those features at one nominal level. Reporting all three—coverage, width, and interval score—usually makes the trade-off clearest. If several nominal levels are available, evaluate each separately rather than assuming performance at one level guarantees performance at another.

A key limitation is that empirical coverage estimates performance only for the validation cases and conditions represented. A change in dosing, assay behavior, population, or clinical practice can make historical calibration less relevant. Another pitfall is to call wide intervals “well calibrated” just because they contain many outcomes: high coverage may reflect over-wide intervals, while calibration requires comparison with the stated nominal rate. Conversely, a wide interval whose coverage is near nominal is not miscalibrated merely because it is less useful for decision-making. Calibration and sharpness are related goals but not synonyms.

Common mistake: “Every observation was inside its interval, so the forecasts are well calibrated.” This confuses high coverage with nominal coverage and ignores interval width. For a nominal 90% procedure, 100% observed coverage may be sampling variation or may reflect overly conservative, very wide intervals. Compare the empirical rate with 90%, report widths, assess uncertainty, and inspect relevant strata before drawing a conclusion.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

For 30 held-out cases, 25 outcomes fall within nominal 90% intervals. What is empirical coverage, and what is one reason not to call this definitive evidence of undercoverage?
A held-out outcome is 3 mg/L above the upper endpoint of a 90% interval with width 5 mg/L. Calculate its interval score.
Two groups each have observed coverage of 90%, but one group has twice the mean interval width. Which group is better calibrated based on these summaries?
Why can pooled coverage hide a problem at long forecast horizons?
What is wrong with treating 40 forecast rows from 8 patients as 40 independent cases when estimating coverage uncertainty?

Solutions. (1) Coverage is $25/30=0.833$, or 83.3%. It is below 90%, but finite-sample variation means it does not alone establish that the underlying coverage is below nominal. (2) The score is $5+(2/0.10)(3)=5+60=65$ mg/L. (3) Neither is shown to be better calibrated: their observed rates are both 90%; the narrower group is more informative by width, subject to uncertainty in the estimates. (4) A large number of well-covered short-horizon forecasts can dominate the pooled rate and conceal lower coverage in a smaller long-horizon group. (5) Forecasts from the same patient may be dependent, so treating rows as independent can understate uncertainty; resampling or summarizing at the patient level is more appropriate.