Why Coverage and Sharpness Must Be Evaluated Together
An external prediction interval is useful only if it performs two jobs: it should contain future observations at about the rate its nominal level promises, and it should be sufficiently narrow to guide decisions. The first property is coverage; the second is commonly described as sharpness. A very wide interval can cover nearly every observation yet communicate little about plausible concentrations. A very narrow interval can look precise while missing too many observations.
This evaluation concerns intervals generated for the external prediction target defined by the prediction task. For example, an interval might target a concentration for a new individual at a specified time and dose, or a future observation for an individual with specified information already available. The target, conditioning information, and uncertainty represented by the interval must match the external observations being assessed. The model and interval-generating procedure should be fixed before those observations are examined.
A nominal 90% prediction interval is intended, under the model and the stated prediction target, to cover about 90% of future observations over repeated applications. It does not promise that exactly 90% of any particular finite validation sample will be covered. Nor does it mean that there is a 90% probability that a fixed parameter or the observed value lies in a particular interval after the interval has been calculated. Those are different probability statements.
A prediction interval for an individual observation must also be distinguished from a confidence interval for a population mean or model parameter. The latter describes uncertainty about a mean or parameter; it is not designed to contain most individual future outcomes. Evaluating it with individual concentrations would therefore answer the wrong question.
Calculating Empirical Coverage and Width
Choose the nominal level or levels before looking at validation outcomes. For a nominal level \(q\), such as 0.90, record whether each matched observed value falls inside its interval. Divide the number of covered observations by the total number of eligible observations. Calculate widths using the same observations and average them in the outcome’s measurement units.
Here \(I_i\) is the coverage indicator, \(\widehat{C}\) is empirical coverage, and \(\overline{W}\) is mean width. Inclusion of an observation exactly on a bound counts as coverage under this definition. For concentration predictions, if \(Y_i\), \(L_i\), and \(U_i\) are measured in mg/L, widths and mean width are also in mg/L.
The calculation is descriptive unless an inferential procedure is specified as well. If observations were independent draws from the stated prediction target, each coverage indicator would have expected value equal to the target coverage when the interval procedure is calibrated. Their average would then estimate that target coverage. Repeated observations from the same individual are generally dependent, however. Treating every concentration as an independent validation unit can make a simple uncertainty calculation too optimistic. Report the number of individuals as well as the number of observations, and account for clustering if uncertainty around the empirical rate is needed.
Worked Example: Calculating Coverage and Mean Width
Suppose a frozen NLME model produces hypothetical 90% prediction intervals for eight external concentration observations. The observed concentrations, intervals, and coverage decisions are:
| Observation | Observed (mg/L) | 90% interval (mg/L) | Covered? | Width (mg/L) |
|---|---|---|---|---|
| 1 | 3.1 | [2.0, 4.0] | Yes | 2.0 |
| 2 | 1.2 | [1.5, 3.5] | No | 2.0 |
| 3 | 4.8 | [3.0, 5.0] | Yes | 2.0 |
| 4 | 6.4 | [4.0, 6.0] | No | 2.0 |
| 5 | 3.9 | [2.5, 5.5] | Yes | 3.0 |
| 6 | 2.0 | [1.0, 3.0] | Yes | 2.0 |
| 7 | 5.2 | [5.0, 7.0] | Yes | 2.0 |
| 8 | 6.1 | [3.5, 6.5] | Yes | 3.0 |
Six of the eight observed concentrations fall between their respective bounds. Thus:
The total width is \(2+2+2+2+3+2+2+3=18\) mg/L, so:
In this hypothetical sample, observed coverage is 75% for nominal 90% intervals, and mean width is 2.25 mg/L. This is a warning worth investigating, not proof that the procedure’s long-run coverage is exactly 75%. Eight observations provide limited information, and dependence among observations would further affect how much evidence the rate provides.
Why the Coverage Calculation Estimates a Target Rate
The basic justification follows from the indicator used in the calculation. For an observation drawn from the intended prediction target, the indicator equals one exactly when the interval covers it. Its expected value is therefore the probability of coverage. If that probability is \(q\), the expected average of \(N\) such indicators is also \(q\), provided each is evaluated under that same target.
Specify what future outcome is being predicted and what information is available when each interval is generated.
For observation \(i\), coverage is the event \(L_i \leq Y_i \leq U_i\); its indicator is \(I_i\).
If the procedure has target coverage \(q\) for each matched prediction, then \(E(I_i)=q\), and linearity of expectation gives \(E(\widehat{C})=E\left(\frac{1}{N}\sum I_i\right)=q\).
This establishes why the empirical proportion is a natural estimate of coverage when the observations represent the target and the intervals are evaluated as specified. Independence is not needed for that expectation calculation, but it matters for the variability of the estimate. In particular, repeated outcomes from one individual may contribute related indicators rather than independent information.
Assessing Sharpness Without Rewarding Overly Broad Intervals
Mean width summarizes the average span of the intervals, but it cannot be interpreted in isolation. A nominal level matters: a 95% interval is ordinarily wider than a 50% interval from the same predictive distribution. Comparisons of width are most informative when they use the same nominal level, outcome scale, prediction target, and external sample. A narrow interval is desirable only if coverage remains adequate for that target.
Worked Example: High Coverage Can Be Uninformative
Consider ten hypothetical external observations with concentrations \(Y_i=i\) mg/L for \(i=1,\ldots,10\). At a prespecified nominal level of 90%, method A gives intervals of width 1 mg/L centered on each of the first eight observations, but gives \([7,8]\) mg/L for \(Y_9=9\) and \([11,12]\) mg/L for \(Y_{10}=10\). Method B gives an interval \([Y_i-3,Y_i+3]\) mg/L for every observation.
Method A covers the first eight observations and misses the last two, so its empirical coverage is \(8/10=80\%\). Every interval has width 1 mg/L, giving mean width \(10(1)/10=1\) mg/L. Method B covers all ten observations, because each observed value is at the center of its interval. Each interval has width \(6\) mg/L, giving mean width \(10(6)/10=6\) mg/L.
Method B has higher observed coverage in this sample, but its intervals are six times as wide on average. The example does not establish which method will perform better in future samples. It shows why reporting coverage alone can make a very broad procedure appear attractive, and why sharpness alone could favor intervals that miss too often.
Where several nominal levels are prespecified, evaluate coverage and width at each level. In a well-behaved family of central intervals, higher nominal levels should generally correspond to wider intervals and higher coverage. A single empirical deviation can arise from sampling variation; a systematic pattern across levels may indicate a mismatch between predicted uncertainty and observed outcomes.
Worked Example: Checking Two Prespecified Levels
Suppose a hypothetical external evaluation includes 20 matched concentration predictions. The prespecified 50% intervals cover 10 observations and have mean width 1.8 mg/L. The prespecified 90% intervals cover 18 observations and have mean width 5.4 mg/L. The calculations are:
The empirical rates equal their respective nominal levels in this hypothetical sample, while the higher-level intervals are wider by \(5.4-1.8=3.6\) mg/L on average. That pattern is consistent with the intended coverage-width trade-off. It does not demonstrate exact calibration in the population or guarantee the same rates in a new sample. The reported results are sample summaries, not proof that either interval procedure will attain its nominal rate in all settings.
Diagnostics, Comparisons, and Limitations
Overall coverage can conceal important failures. A model may cover adequately at low concentrations but miss systematically at high concentrations, or work for one dosing regimen and not another. Examine coverage and width across scientifically relevant groups and prediction conditions, such as time after dose or concentration range, while taking care not to turn a small number of observations into strong subgroup claims. Record the subgroup sample sizes. If almost every observation lies in one region, an overall rate says little about performance elsewhere.
Also inspect the individual misses and interval bounds. Frequent observations below lower bounds or above upper bounds may indicate a directional discrepancy, while misses on both sides can indicate intervals that are too narrow. The coverage rate alone does not distinguish those patterns. Check that all intervals and observations use consistent units and that the prediction target and available covariates are matched. An interval evaluated against an outcome that helped construct it is not an external prediction assessment.
Coverage and width complement, rather than replace, other evaluations. Coverage asks whether stated interval levels match observed inclusion rates; width describes the precision of those intervals on the outcome scale. Point-prediction measures answer a different question, and a predictive scoring rule can combine rewards for concentration and penalties for interval misses. None removes the need to define the target and inspect performance in context.
The principal limitations are finite-sample variation, dependence among repeated observations, and the fact that an average width does not show the full distribution of widths. A seemingly satisfactory overall rate may be driven by a large subgroup while another subgroup is poorly served. Conversely, a small sample can deviate noticeably from nominal coverage even when the procedure is reasonably calibrated. Report the external sample size, number of individuals where relevant, prespecified nominal levels, observed coverage, and width summaries together.
Common Pitfall
Wrong reasoning: “The 95% intervals covered all 20 external observations, so the model is perfectly calibrated, and those intervals are excellent because they cover everything.” The sample rate is \(20/20=100\%\), not evidence of perfect long-run calibration. If those intervals are extremely broad, their coverage may be achieved with little practical precision.
Correction: Describe the result as 100% empirical coverage among 20 observations, report the widths in the outcome units, and consider the uncertainty and representativeness of the validation sample. Evaluate coverage and sharpness jointly at the levels and for the target chosen in advance.
Check Your Understanding
Answer each question, then compare with the solutions.
- Seven of ten external observations fall within their nominal 80% intervals. What is empirical coverage, and how should it be compared with the nominal level?
- For six concentration intervals, the sum of widths is 15 mg/L. What is the mean width, and what are its units?
- A method has 100% empirical coverage but a mean width of 40 mg/L, while another has 90% coverage and a mean width of 4 mg/L at the same nominal level. What can and cannot be concluded from these summaries?
- Why can treating 30 repeated concentrations from three individuals as 30 independent observations overstate the information in a coverage estimate?
- Why is a confidence interval for the population mean not generally suitable for assessing coverage of individual future concentrations?
Solutions. (1) Coverage is \(7/10=70\%\). It is 10 percentage points below the nominal 80% level in this sample; that difference warrants contextual interpretation and does not alone establish the procedure’s long-run coverage. (2) Mean width is \(15/6=2.5\) mg/L. (3) The first method covered more observations in the evaluation but is much broader; it may be less informative. These summaries do not establish future performance or determine which method is preferable for every use. (4) Outcomes within an individual may be dependent, so their coverage indicators need not provide as much independent information as 30 observations from 30 unrelated individuals. (5) A confidence interval for a mean targets uncertainty about the mean, whereas an individual prediction interval targets future individual outcomes, which vary around that mean.