Checking Predictive-Interval Coverage

What Predictive-Interval Coverage Measures

A predictive interval is intended to contain a future concentration at a specified sampling design and covariate profile with a stated frequency. Checking its coverage asks a direct question: what fraction of the observed concentrations fall inside the intervals that the model predicted for them? This complements summaries that compare observed and simulated distributions. A model can have reasonable overall summaries yet miss its stated coverage at particular sampling times or for particular groups.

Coverage is a repeated-sampling property, not a probability assigned after observing one fixed concentration. For example, a correctly calibrated 90% predictive interval is designed to include about 90% of future observations generated under the target conditions. In a finite validation sample, the empirical fraction will generally differ from 90% because of sampling variability.

Definition: For observation \(i\), let \(Y_i\) be the observed concentration and \([L_i,U_i]\) the model-based predictive interval constructed for that observation's design and covariates. Define the coverage indicator \(I_i=1\) if \(L_i\leq Y_i\leq U_i\), and \(I_i=0\) otherwise. For \(n\) observations, empirical coverage is \(\hat{p}_{\text{cov}}=\frac{1}{n}\sum_{i=1}^{n} I_i\). If the interval has nominal coverage \(c\), calibration means that the probability of inclusion is \(c\) under the stated target conditions.

The target conditions matter. Intervals must be generated for the observed sampling times, dosing histories, and covariates using a predictive distribution appropriate to those conditions. A central 90% interval, for instance, is usually formed from the 5th and 95th percentiles of that predictive distribution. The endpoints can differ from one observation to another; the coverage indicator is evaluated against each observation's own interval.

In a basic coverage calculation, the fitted model and its parameter estimates are treated as fixed. Predictive simulations should represent the observation-level variability specified by the model, including relevant between-subject and residual variability. This check does not, by itself, account for uncertainty in the estimated model parameters. The distinction is important when interpreting narrow intervals.

Estimating Coverage and Its Uncertainty

If the \(n\) coverage indicators are independent and each has the same probability \(p_{\text{cov}}\) of being one, then their sum follows a binomial distribution. The empirical coverage is an unbiased estimate of \(p_{\text{cov}}\), and its estimated standard error is

SE(\hat{p}_{\text{cov}})=\sqrt{\frac{\hat{p}_{\text{cov}}(1-\hat{p}_{\text{cov}})}{n}}.

This standard error describes sampling variability in the observed coverage fraction under the binomial assumptions. It is not the variability of individual concentrations and is not the uncertainty in the model's parameter estimates. A simple normal-approximation interval, \(\hat{p}_{\text{cov}}\pm 1.96SE\), can be inaccurate near zero or one, especially for small samples. A Wilson score interval is generally a better default for a binomial coverage proportion.

Definition: For observed coverage \(\hat{p}_{\text{cov}}=x/n\), a two-sided Wilson interval with normal critical value \(z\) has center \(C\) and half-width \(H\):

C=\frac{\hat{p}_{\text{cov}}+z^2/(2n)}{1+z^2/n}, \qquad H=\frac{z\sqrt{\hat{p}_{\text{cov}}(1-\hat{p}_{\text{cov}})/n+z^2/(4n^2)}}{1+z^2/n}.

For a 95% interval, use \(z=1.96\). The resulting interval is \((C-H,C+H)\). It describes uncertainty in the population coverage probability under the binomial sampling model; it does not mean there is a 95% probability that this fixed probability lies in the realized interval.

Why the Wilson Interval Works

The Wilson interval is obtained by inverting a score test for a binomial proportion. Suppose a candidate coverage probability is \(p\), and the observed proportion is \(\hat{p}_{\text{cov}}\). Under the binomial model, the score statistic compares the observed proportion with \(p\), using the standard error calculated under that candidate value. Retaining the values of \(p\) for which the absolute score is no greater than \(z\) gives

\left|\frac{\hat{p}_{\text{cov}}-p}{\sqrt{p(1-p)/n}}\right|\leq z.

Squaring both sides and multiplying by the positive quantity \(p(1-p)/n\) gives

n(\hat{p}_{\text{cov}}-p)^2\leq z^2p(1-p).

Expanding and collecting terms in \(p\) produces a quadratic inequality:

(n+z^2)p^2-(2n\hat{p}_{\text{cov}}+z^2)p+n\hat{p}_{\text{cov}}^2\leq 0.

The two roots of this quadratic are the lower and upper Wilson limits. Writing their midpoint and half-distance in terms of \(\hat{p}_{\text{cov}}\) gives \(C\) and \(H\) above. Unlike the simple normal interval, the Wilson limits remain within zero and one and behave more sensibly when the observed proportion is near a boundary.

Worked Example: Estimating Coverage in a Small Sample

Suppose a hypothetical validation set contains 24 independent subjects, each contributing one concentration. A model provides a nominal 90% predictive interval for each subject at the subject's observed sampling time. Twenty-one concentrations fall inside their corresponding intervals. Estimate coverage and its standard error.

The empirical coverage is

\hat{p}_{\text{cov}}=\frac{21}{24}=0.875.

Under the independent-binomial approximation, the estimated standard error is

SE=\sqrt{\frac{0.875(1-0.875)}{24}} =\sqrt{\frac{0.109375}{24}} =\sqrt{0.0045573} \approx 0.0675.

The estimated coverage is 87.5%, which is 2.5 percentage points below the nominal 90%. The standard error is about 6.75 percentage points, so this sample does not estimate coverage very precisely. The observed difference from nominal coverage alone is not evidence that the model is miscalibrated; its uncertainty and the sampling design must also be considered.

Worked Example: A Wilson Interval for Coverage

In a second hypothetical validation set, 18 of 20 independent concentrations fall inside their corresponding nominal 95% predictive intervals. Here \(\hat{p}_{\text{cov}}=18/20=0.900\). Calculate a 95% Wilson interval using \(z=1.96\), so \(z^2=3.8416\).

The center is

C=\frac{0.900+3.8416/(2\cdot20)}{1+3.8416/20} =\frac{0.900+3.8416/40}{1+3.8416/20} =\frac{0.99604}{1.19208} \approx 0.835548.

The half-width is

H= \frac{1.96\sqrt{0.900(0.100)/20+3.8416/(4\cdot20^2)}}{1+3.8416/20} = \frac{1.96\sqrt{0.004500+0.002401}}{1.19208} \approx 0.1366.

Thus the interval is approximately

(0.835548-0.1366,\;0.835548+0.1366) \approx (0.699,\;0.972).

The point estimate, 90%, is below the nominal 95%, but the Wilson interval is wide and includes 0.95. These data therefore do not establish that the underlying coverage differs from 95%. Including the nominal value in an interval is not proof of calibration; it indicates that this sample is compatible with a range of coverage probabilities that includes the target.

Coverage Across Sampling Conditions

An overall coverage fraction can hide patterns. If predictive intervals are intended to work across a range of sampling times, it is useful to calculate coverage in prespecified time groups or other clinically relevant strata. Each group then has its own estimate and uncertainty. The purpose is to identify where calibration may be inadequate, not to claim that every subgroup estimate must equal the nominal value exactly.

Worked Example: Comparing Coverage at Two Sampling Times

Consider a hypothetical evaluation with 50 independent subjects in each of two sampling-time groups. Among early samples, 45 of 50 concentrations fall within their nominal 90% intervals. Among late samples, 38 of 50 fall within their intervals. Calculate the estimates and 95% Wilson intervals to describe the pattern.

For the early group, \(\hat{p}_{\text{cov}}=45/50=0.900\). Using \(z=1.96\), the Wilson denominator is \(1+3.8416/50=1.076832\), and the center is

C_{\text{early}}=\frac{0.900+3.8416/100}{1.076832} =\frac{0.938416}{1.076832} \approx 0.8715.

Its half-width is

H_{\text{early}}= \frac{1.96\sqrt{0.900(0.100)/50+3.8416/(4\cdot50^2)}}{1.076832} \approx 0.08505,

giving a Wilson interval of approximately \((0.786,\;0.957)\).

For the late group, \(\hat{p}_{\text{cov}}=38/50=0.760\). The center and half-width are

C_{\text{late}}=\frac{0.760+3.8416/100}{1.076832} =\frac{0.798416}{1.076832} \approx 0.7414,

H_{\text{late}}= \frac{1.96\sqrt{0.760(0.240)/50+3.8416/(4\cdot50^2)}}{1.076832} \approx 0.1156.

The late-group interval is approximately \((0.626,\;0.857)\). Its estimate is notably below the nominal 90%, whereas the early-group estimate equals the target. This pattern warrants investigation of the model's predictions at late times. The intervals describe uncertainty in each group's coverage; they are not a formal test of the difference between groups, and overlapping or non-overlapping intervals should not be treated as a substitute for a prespecified comparison.

Assumptions, Diagnostics, and Limitations

A binomial standard error or Wilson interval is justified when the coverage indicators are independent Bernoulli observations with a common coverage probability within the analysis group. This is most straightforward when each independent subject contributes one concentration. In population pharmacokinetic data, however, subjects often contribute repeated concentrations. Those indicators can be correlated because measurements from the same subject share individual characteristics and a dosing history. Counting every record as independent can then make the uncertainty appear too small.

With repeated observations, an observation-level coverage fraction can still describe the fraction of records covered in the data set. But the ordinary binomial uncertainty calculation is not automatically valid for it. State the sampling unit, report the number of subjects as well as the number of records, and use an uncertainty method that respects subject-level dependence when inferential precision is required. Do not silently treat correlated samples as independent replicates.

Several checks help determine whether a reported coverage estimate is meaningful:

Verify the interval definition. Confirm the intended nominal level, the quantiles used, and whether the bounds are evaluated inclusively.
Match predictions to observations. Each interval should reflect the relevant time, dose history, and covariates. A mismatch can create apparent undercoverage unrelated to the intended predictive check.
Inspect subgroup estimates. Overall coverage can conceal overcoverage in one region and undercoverage in another. Examine prespecified, adequately populated groups rather than searching many sparse groups for chance deviations.
Check predictive simulation precision. If interval endpoints are estimated from a limited number of simulations, Monte Carlo noise can shift the endpoints and alter the coverage count. Increase simulation replicates and check whether the estimated quantiles and coverage stabilize.
Keep the uncertainty target clear. A binomial interval quantifies sampling uncertainty in coverage under its assumptions. It does not include uncertainty from estimating model parameters when those parameters are held fixed in the predictive simulation.

Coverage also has limitations as a stand-alone diagnostic. A model can achieve the nominal fraction while producing intervals that are too wide, too narrow in some regions and too wide in others, or poorly located in relation to the observations. Conversely, a modest departure in a small validation sample may be ordinary sampling variation. Interpret coverage alongside the interval width, the sampling design, and other predictive diagnostics, including the earlier material on visual predictive checks and normalized prediction distribution errors. This tutorial focuses on inclusion frequency, not on diagnosing tail behavior or joint profile coverage.

Common Mistake

A frequent error is to calculate \(18/20=90\%\) and conclude, “The model has 90% coverage, so its nominal 95% intervals are calibrated.” The calculation is the observed coverage estimate, not the model's known long-run coverage. In the worked example, its Wilson interval is approximately \((0.699,0.972)\), which is wide and includes 0.95. The correct conclusion is that the observed estimate is 90%, with substantial uncertainty; these data do not precisely determine whether the true coverage is below the nominal target.

Another error is to count repeated concentrations from one subject as if they were independent observations. That may leave the point estimate unchanged but typically understates its uncertainty when within-subject coverage indicators are positively correlated. The correction is to distinguish the descriptive record-level fraction from the inferential calculation and account for the actual independent sampling units.

Key takeaway: Empirical predictive-interval coverage is the proportion of observed concentrations inside their matched intervals. Report the nominal level, the estimate, the uncertainty method, and the independent sampling unit; interpret departures in light of both uncertainty and possible patterns across sampling conditions.

Check Your Understanding

Answer each question, then compare with the solutions below.

Of 40 independent concentrations, 34 fall inside their nominal 90% predictive intervals. What is empirical coverage?
Why is the ordinary binomial standard error questionable if each subject contributes several correlated concentrations?
In a Wilson interval, what is the role of \(n\), and what does the resulting interval quantify?
A group has estimated coverage of 0.82 for nominal 90% intervals. Does that alone prove undercoverage?

Solutions. (1) Empirical coverage is \(34/40=0.850\), or 85.0%. (2) The indicators from one subject may be correlated, so the data do not contain as many independent Bernoulli observations as the record count suggests; the usual binomial standard error may be too small. (3) \(n\) is the number of independent observations under the binomial model. The Wilson interval quantifies sampling uncertainty in the underlying coverage probability, not parameter-estimation uncertainty or the probability that a fixed probability lies in the realized interval. (4) No. The estimate is below nominal, but its uncertainty, sample size, dependence structure, and consistency across relevant conditions must be considered before concluding that coverage is inadequate.