Why Summarize Predictive-Interval Coverage?
A visual predictive check shows whether observed outcomes fall in regions generated by model simulations, but a visual impression can be difficult to compare across time ranges or patient groups. A numerical coverage summary makes one specific feature countable: the fraction of observed outcomes that lie inside a chosen simulated prediction interval. Calculating that fraction by time bin or subgroup can help locate where predictive intervals do, and do not, contain the observations at the intended rate.
Coverage is a descriptive calibration summary, not a stand-alone test of model adequacy. A rate near the nominal level is reassuring about interval inclusion in that bin, but does not establish that the full distribution is well described. For example, the model may miss the shape or dependence of the data while still placing roughly the expected fraction inside a broad interval.
Definition and Calculation
Suppose \(y_{ij}\) is the observed outcome for subject \(i\) at record \(j\), and \([L_{ij},U_{ij}]\) is the simulated prediction interval assigned to that record. The interval is constructed from simulated outcomes using a prespecified predictive-check design. For example, a central 90% prediction interval has lower and upper predictive quantiles of 0.05 and 0.95. The observed record is covered when \(L_{ij}\leq y_{ij}\leq U_{ij}\).
Intervals must correspond to the predictive target being evaluated. A record-specific interval uses simulations aligned to that record’s covariates and observation time. Another design may construct a predictive distribution for outcomes pooled within a time bin. Either way, define the interval construction clearly and apply the same rule to every observed record in the summary. Do not compare an observed value with bounds that were created for a different time, subgroup, outcome scale, or predictive-check target.
Here \(b\) indexes a time bin or subgroup, \(n_b\) is the number of observed records in it, \(k_b\) is the number covered, and \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. The nominal rate \(q\) is the reference for a central interval; the descriptive departure is \(C_b-q\), usually stated in percentage points. State the denominator as well as the rate: 18 of 20 records covered is more informative than reporting 90% alone.
What the Coverage Rate Means
If the predictive distribution is appropriate for the records under study and the interval is calibrated to contain a proportion \(q\) of outcomes, its expected coverage is \(q\). To see why, define an indicator \(I_{ij}\) equal to 1 when record \(j\) for subject \(i\) is covered and 0 otherwise. If each interval has predictive coverage \(q\), then \(E(I_{ij})=q\). By linearity of expectation,
This justification concerns the expected rate under the predictive model; it does not say that every finite data set must have exactly that rate. Nor does it require the indicators to be independent for the expectation calculation. Repeated records from one subject can be dependent, so the raw coverage proportion should not automatically be treated as if it came from independent Bernoulli trials. Here the purpose is to describe where the observed records fall relative to the simulated intervals, not to attach an unqualified significance test to each rate.
Worked Example: Coverage Across Time Bins
Consider a hypothetical concentration check using central 90% simulated prediction intervals. Each observed concentration is compared with the interval assigned to its record. In an early time bin, 18 of 20 observations are inside their intervals; in a middle bin, 15 of 20 are inside; and in a late bin, 19 of 20 are inside.
For the early bin, \(k_b=18\) and \(n_b=20\), so \(C_b=18/20=0.90\), or 90%. Its descriptive departure from nominal coverage is \(90\%-90\%=0\) percentage points.
For the middle bin, \(C_b=15/20=0.75\), or 75%. The departure is \(75\%-90\%=-15\) percentage points. For the late bin, \(C_b=19/20=0.95\), or 95%, a departure of \(95\%-90\%=+5\) percentage points.
The middle-bin rate is the clearest descriptive shortfall in this example. It is a reason to inspect the predictive-check display in that time region, including whether observations tend to fall below or above the interval. The rate alone does not show which tail is responsible or establish why coverage is low.
Subgroups and Aggregation
The same calculation applies to a subgroup, such as dose level, renal-function category, or study arm: identify the observed records in the group, count those inside their corresponding intervals, and divide by that group’s record count. The intervals should be generated for the subgroup’s predictive target. A subgroup comparison is most interpretable when the interval level and simulation design are consistent across groups.
An overall coverage rate formed by pooling records weights each bin or subgroup according to its number of records. It is not generally the same as the unweighted average of the bin-specific rates. Both quantities can be useful, but they answer different questions: the pooled rate describes the fraction covered among all records, whereas the unweighted average gives equal weight to each bin, regardless of its size.
Worked Example: Comparing Coverage by Subgroup
In a hypothetical study, a central 90% interval is evaluated separately for two dose groups. In the lower-dose group, 36 of 40 observed records are covered. In the higher-dose group, 25 of 40 are covered. Assume that the simulated intervals were constructed for the appropriate dose group and each included record is counted once.
For the lower-dose group, \(C_{\mathrm{low}}=36/40=0.90\), or 90%. For the higher-dose group, \(C_{\mathrm{high}}=25/40=0.625\), or 62.5%. Relative to nominal 90% coverage, the descriptive departures are 0 and \(-27.5\) percentage points, respectively.
The higher-dose group therefore has a substantially lower observed inclusion rate in this hypothetical example. A useful next step is to inspect the visual check for that group: do observations fall mainly below the lower bound, above the upper bound, or on both sides? The numerical rate identifies a coverage discrepancy but does not describe its direction. It also should be reported with its denominator; here the 62.5% rate represents 25 covered records out of 40.
Worked Example: Pooled Coverage Is Not the Average Bin Rate
Suppose a hypothetical central 90% predictive interval is checked in two time bins. The first bin has 10 records, all covered. The second has 90 records, of which 72 are covered. The bin-specific rates are \(10/10=1.00\), or 100%, and \(72/90=0.80\), or 80%.
The unweighted average of the two bin rates is \((100\%+80\%)/2=90\%\). But the pooled observation-level rate is calculated from all records: \((10+72)/(10+90)=82/100=0.82\), or 82%. The arithmetic is weighted by the bin sizes because the larger bin contributes more records.
Thus, saying “average coverage was 90%” would be ambiguous unless the averaging rule were stated. For an overall fraction of observed records covered, use the pooled count divided by the pooled denominator. If equal importance for each time bin is the intended summary, report the unweighted average explicitly and retain the bin-specific rates so the pattern remains visible.
Reading Coverage Alongside Visual Checks
Coverage summaries complement, rather than replace, visual predictive-check summaries. A visual check can show whether the observations track the simulated distribution’s center and spread across time. Coverage condenses interval inclusion into a rate for a defined set of records. Read the two together: a low rate may coincide with visible observations extending beyond the simulated bounds, while a rate near nominal does not guarantee that observed quantiles or the shape of the distribution match the simulations.
This is also distinct from NPDE summaries. NPDE standardizes predictive deviations and can summarize their marginal behavior; coverage instead asks whether outcomes lie within selected predictive bounds. The methods answer related but different questions, so agreement or disagreement between them should be described rather than collapsed into a single conclusion.
For a transparent report, state the interval level, how and for which records its bounds were produced, the binning or subgroup rule, the numerator and denominator, the observed rate, and the nominal rate. A compact table can make the pattern easy to inspect:
| Bin or subgroup | Covered records | Total records | Observed coverage | Nominal coverage |
|---|---|---|---|---|
| Early time | 18 | 20 | 90% | 90% |
| Middle time | 15 | 20 | 75% | 90% |
| Late time | 19 | 20 | 95% | 90% |
Diagnostics, Pitfalls, and Limitations
Before interpreting a coverage departure, check the data-to-interval matching and the denominator. Common implementation errors include comparing a concentration with bounds on a log scale, using bounds from another time or subgroup, counting a record twice, or omitting observations without explaining why. For a central interval, an observation exactly equal to a bound is inside if the stated rule is \(L_{ij}\leq y_{ij}\leq U_{ij}\); use the same boundary convention throughout.
- Confusing an interval’s nominal level with observed coverage. A central 90% interval is designed for a 90% predictive inclusion rate; it does not force 90% of the observed records to be inside it.
- Reporting a rate without its denominator. A rate based on a small number of records is less informative than the same rate based on many records. Always give the count and the total.
- Pooling unlike records without checking the target. A pooled rate can hide a time- or subgroup-specific pattern. Show the component rates when those patterns matter.
- Treating nominal coverage as a complete model check. Correct inclusion alone does not establish that the predicted center, tails, or within-subject pattern are appropriate.
Coverage summaries are limited by the choices that define them: interval level, predictive simulation design, time-bin boundaries, and subgroup definitions. Changing these choices can change the rates. Very narrow bins or many subgroup splits may leave few records per summary, making the displayed proportions unstable as descriptions of the study data. The observed rate should therefore be interpreted with the visual check and the underlying counts, not as a universal pass-or-fail criterion.
State its nominal level, outcome scale, and the predictive-check target for which its bounds were generated.
Set time bins or subgroups before counting, and ensure each included observation is matched to the appropriate interval.
For each bin or group, calculate covered records divided by total eligible records, then report both counts and the rate.
Compare the observed rate with nominal coverage and inspect the visual check for the direction and location of any departure.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A central 80% interval covers 24 of 30 records in a time bin. What is observed coverage and its departure from nominal coverage?
- Two bins have rates of 70% and 90%. Why might their unweighted average differ from pooled coverage?
- What does a low coverage rate tell you, and what does it not tell you by itself?
- Name two checks to make before interpreting a subgroup coverage departure.
Solutions. (1) \(24/30=0.80\), or 80%; the departure is \(80\%-80\%=0\) percentage points. (2) The bins may contain different numbers of records, so pooled coverage weights their rates by their denominators, whereas the unweighted average assigns equal weight to each bin. (3) It indicates that fewer observed outcomes than the nominal rate were inside the specified intervals; alone, it does not identify which tail accounts for the misses or explain their cause. (4) Check that observations are matched to intervals for the correct time, subgroup, and outcome scale, and verify the interval level and the included-record denominator.