Coverage of External Prediction Intervals

Why Prediction-Interval Coverage Matters

An external prediction interval describes a range intended to contain a future PK outcome, such as an observed concentration, at a stated frequency under the model and its prediction target. In external validation, the model is applied to a separate data set and its intervals are formed without using the external outcomes to refit or tune the model. The central question is direct: do the observed outcomes fall inside the intervals as often as their nominal level implies?

Coverage is a property of a set of predictions across repeated outcomes, not a guarantee about any one patient. A 90% prediction interval is designed to contain the target outcome in 90% of repeated cases under the assumptions used to construct it. In an external data set, the fraction covered is an estimate of that performance and will vary by chance. It can be too low or too high in a finite sample even when the model is behaving as intended.

Definition: For external observation \(i\), let \([L_i,U_i]\) be the prespecified prediction interval for the target outcome \(Y_i\). Define the coverage indicator \(I_i=1\) when \(L_i \leq Y_i \leq U_i\), and \(I_i=0\) otherwise. For \(n\) validation units, the empirical coverage is \(\hat{q}=\frac{1}{n}\sum_{i=1}^{n} I_i\). The nominal coverage of a two-sided interval with tail probability \(\alpha\) is \(1-\alpha\).

The prediction target must be clear. An interval for a future individual concentration generally includes uncertainty from relevant individual variability and residual error; an interval for a population mean or another population-level quantity targets something different. The observed outcome used to assess coverage must match the interval’s target. Comparing individual concentrations with an interval for a mean, for example, does not measure the advertised individual-level coverage. The distinction between population and conditional prediction performance is also important: the prediction setting used to make the intervals must be the setting evaluated externally.

Assumptions Behind a Coverage Summary

A coverage estimate is interpretable only when the validation procedure is aligned with the intended prediction task. Before counting covered observations, check that the model and interval construction were locked, the external outcomes were not used to choose interval widths or analysis cutoffs, and the outcome definition matches the model’s target. Also establish that the external population, dosing and sampling context are relevant to the intended use. Differences in these conditions can affect coverage; the observed rate remains descriptive of the validation data actually collected.

The usual binomial uncertainty calculation makes a further assumption: the validation units are independent and have a common probability \(q\) of being covered. Under this assumption, each \(I_i\) is Bernoulli with probability \(q\), and \(K=\sum I_i\) has a binomial distribution. If a patient contributes many concentrations, those indicators may be correlated. Treating every concentration as an independent unit can then make the apparent precision much greater than the information in the data supports. The analysis should instead respect the sampling structure, for example by defining an appropriate patient-level target or using an uncertainty method that accounts for clustering.

E(\hat{q})=q,\qquad \operatorname{Var}(\hat{q})=\frac{q(1-q)}{n},\qquad SE(\hat{q})\approx\sqrt{\frac{\hat{q}(1-\hat{q})}{n}}.

The first two results follow from writing \(\hat{q}\) as the average of independent Bernoulli indicators: each has expectation \(q\) and variance \(q(1-q)\). The estimated coverage is unbiased for \(q\), and its standard error is estimated by replacing the unknown \(q\) with \(\hat{q}\). This standard error quantifies sampling uncertainty in the coverage proportion; it does not account for model uncertainty that was omitted from the intervals, a mismatched validation population, or dependence that violates the assumptions.

Calculating and Interpreting Coverage

Worked Example: Overall Coverage of a 90% Interval

Suppose, hypothetically, a locked PK model produces individual 90% prediction intervals for one prespecified concentration per patient in an independent external sample of 100 patients. The intervals and outcome are in the same concentration units. Eighty-six observed concentrations fall within their intervals.

There are \(K=86\) covered outcomes out of \(n=100\), so the empirical coverage is

\hat{q}=\frac{86}{100}=0.860=86.0\%.

The nominal coverage is \(1-\alpha=0.90\), so empirical coverage is 4.0 percentage points below nominal. Using the plug-in standard error,

SE(\hat{q})=\sqrt{\frac{0.86(1-0.86)}{100}} =\sqrt{0.001204}=0.0347.

A simple approximate 95% interval for the external coverage probability is \(0.86 \pm 1.96(0.0347)\), or approximately \(0.792\) to \(0.928\). This approximation is reasonable here because the estimated covered and uncovered counts are both substantial, but exact or score-based binomial intervals are alternatives, especially with small samples or rates near zero or one. The interval includes 0.90, so this sample alone does not establish that the underlying coverage differs from nominal. It also does not demonstrate that the model is calibrated: the range of plausible coverage remains fairly broad.

The interpretation is conditional on the independent-patient assumption and the specified prediction task. If the 100 observations were repeated concentrations from far fewer patients, this calculation would generally overstate precision.

The comparison with nominal coverage should include both the point estimate and its uncertainty. A small observed discrepancy can be unimportant relative to sampling variability; a large discrepancy in a large sample may be difficult to attribute to chance alone. Neither a confidence interval containing the nominal rate nor a non-significant test proves that the model has the intended coverage. Precision, validation design, and clinical consequences matter alongside the numerical comparison.

Coverage Patterns Across the Validation Data

An overall rate can conceal where intervals fail. A useful first diagnostic is to tabulate empirical coverage in groups defined before examining results, such as clinically meaningful sampling times, concentration ranges, or populations. For each group, report the number of validation units, covered count, empirical rate, and an uncertainty summary where the design supports one. Small groups produce unstable percentages, so the denominators must remain visible.

These grouped rates are diagnostic summaries, not proof that a particular mechanism caused the difference. Time- or concentration-specific patterns can point to parts of the prediction task needing investigation; population-specific patterns can show that aggregate performance does not represent every group. Detailed assessment of time- and concentration-dependent performance and formal subgroup analyses require attention to design, multiplicity, and precision beyond a simple table.

Worked Example: Comparing Prespecified Sampling Times

In a hypothetical external evaluation, 30 independent patients contribute one prespecified early-time outcome each and 30 different independent patients contribute one late-time outcome each. Both groups are evaluated using 90% prediction intervals. Twenty-eight early outcomes and 22 late outcomes are covered.

For early time, empirical coverage is \(28/30=0.933\), or 93.3%. For late time, it is \(22/30=0.733\), or 73.3%. The difference is

\frac{28}{30}-\frac{22}{30} =\frac{6}{30}=0.200,

so the late-time estimate is 20.0 percentage points lower than the early-time estimate. Both can be compared with the 90% nominal target, but the group sizes are small and the estimates are uncertain. The pattern motivates checking the late-time predictions and the data supporting them; it does not by itself identify a cause. Because the groups contain different independent patients, the calculation avoids treating repeated measurements from the same patient as independent evidence.

Worked Example: How Population Mixing Can Hide Uneven Coverage

Consider a hypothetical external data set with two prespecified populations, each containing 50 independent patients. For 90% intervals, 46 patients in population A are covered and 34 in population B are covered. The overall rate is

\hat{q}_{\text{overall}} =\frac{46+34}{50+50} =\frac{80}{100}=0.800.

The group rates are \(46/50=0.920\), or 92.0%, in population A and \(34/50=0.680\), or 68.0%, in population B. Thus the aggregate estimate is 80.0%, below the 90% nominal level, and it masks a 24.0 percentage-point difference between the groups. If the validation sample had a different mix of populations, its overall coverage could change even if the within-population rates did not. Reporting group denominators and rates therefore adds important context to the pooled summary. These figures identify a performance concern to investigate, not a causal explanation for it.

Why Coverage Alone Is Not Enough

Coverage does not measure how wide the intervals are. An excessively wide interval can contain nearly every observation and still offer little useful precision. Conversely, narrow intervals may be desirable but under-cover if they omit too many outcomes. Interpret coverage alongside interval width and the external prediction-error measures already used for validation. A high coverage rate is not, by itself, evidence of accurate or clinically useful predictions.

Coverage is also distinct from calibration summaries that compare predicted and observed values. A model can have reasonable average agreement while prediction intervals are too narrow; it can also achieve an acceptable overall coverage rate with errors concentrated in one part of the data. The evaluations answer related but different questions: calibration concerns agreement in predictions, while coverage concerns the frequency with which specified intervals contain their target outcomes.

Key takeaway: Compare empirical coverage with the stated nominal level, quantify uncertainty under an appropriate sampling model, and show relevant group-specific rates with their denominators. Coverage is informative only for the outcome and prediction target the interval was designed to contain.

Common Mistakes and Failure Modes

Calling the nominal rate a guarantee. A 90% interval does not have to cover exactly 90% of a finite external sample. Random variation is expected.
Using the wrong outcome target. Assessing individual observations against intervals for a mean does not evaluate individual prediction coverage.
Counting repeated samples as independent patients. Within-patient dependence can invalidate simple binomial standard errors and make precision look too high.
Revising intervals after seeing external outcomes. Choosing a width or subgroup definition to improve the observed rate turns validation into model adjustment and makes the reported coverage optimistic as an evaluation.
Reporting only the pooled percentage. Different populations or sampling conditions can have different coverage rates; a single aggregate can hide that variation.
Equating high coverage with good prediction. Very broad intervals can achieve high coverage while being uninformative. Include interval width and relevant prediction-error summaries.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A hypothetical model’s 80% intervals contain 72 of 90 independent external outcomes. What is empirical coverage and how does it compare with nominal coverage?
Why might 120 concentration measurements from 20 patients not justify a standard binomial standard error calculated with \(n=120\)?
In a hypothetical data set, 18 of 20 outcomes are covered in group A and 12 of 20 in group B. Calculate each rate and their percentage-point difference.
Can 100% empirical coverage establish that the prediction intervals are useful? Explain.

Solutions. (1) \(72/90=0.800\), or 80.0%, exactly equal to the nominal rate in this sample; sampling uncertainty means this does not establish that the underlying coverage is exactly 80%. (2) Measurements from the same patient may be correlated, violating the independence assumption behind the binomial standard error; the analysis must account for clustering or define a suitable patient-level target. (3) Group A has \(18/20=90.0\%\) coverage and group B has \(12/20=60.0\%\); the difference is 30.0 percentage points. The rates are imprecise because each denominator is small. (4) No. The intervals might be excessively wide, and a finite sample does not establish future coverage. Assess interval width and the intended prediction target as well.