Coverage Calibration Of Bayesian Credible Intervals

What Coverage Calibration Measures

A Bayesian credible interval is constructed from a posterior distribution conditional on observed data. Its stated credibility level describes posterior probability for the parameter, given the model, prior, and observed data. Coverage calibration asks a different question: if the same trial design and analysis were repeated many times under a specified data-generating truth, how often would the resulting interval contain that truth?

This repeated-sampling property matters in clinical trial planning and review. A procedure may produce intervals with a nominal 95% posterior probability while its actual coverage at a particular treatment effect is appreciably above or below 95%. Calibration examines that behavior under explicit scenarios; it does not replace the posterior interpretation of an interval.

Definition: Let $\theta_0$ be a fixed generating value for the estimand, and let $D$ denote data generated under a specified trial scenario at $\theta_0$. If $C(D)$ is the Bayesian credible interval produced by the prespecified analysis, its repeated-sampling coverage at $\theta_0$ is $P_{\theta_0}\{\theta_0\in C(D)\}$. The nominal level $1-\alpha$ is the posterior credibility level used to construct the interval; it is not, by itself, a guarantee that this repeated-sampling probability equals $1-\alpha$.

To estimate coverage, simulate $M$ independent trial data sets under the same scenario, apply the same analysis to each one, and record whether the interval contains the generating value. The observed fraction of successful containments estimates the coverage probability. The target estimand, interval construction, trial design, data-generating assumptions, and analysis rules should all be fixed before interpreting the result.

I_j=\mathbf{1}\{\theta_0\in C(D_j)\},\qquad \widehat{\operatorname{Cov}}(\theta_0)=\widehat{c} =\frac{1}{M}\sum_{j=1}^{M} I_j

Here $I_j$ is 1 if replicate $j$'s interval contains $\theta_0$, and 0 otherwise. Under independent, identically generated replicates and a fixed analysis, the indicators are Bernoulli variables with success probability $c=P_{\theta_0}\{\theta_0\in C(D)\}$. Thus $M\widehat{c}$ has a binomial distribution with parameters $M$ and $c$, and the Monte Carlo standard error of the estimated coverage is

SE_{\mathrm{MC}}(\widehat{c}) =\sqrt{\frac{c(1-c)}{M}} \approx \sqrt{\frac{\widehat{c}(1-\widehat{c})}{M}}.

This standard error describes simulation noise in estimating coverage. It does not describe uncertainty about $\theta_0$, nor does it quantify uncertainty from an incorrect data-generating model or an inadequately implemented analysis.

From Simulated Trials to a Coverage Estimate

Specify the scenario.
Set the generating value of the estimand and all other design and data-generating inputs relevant to the calibration question.

Generate a replicate.
Simulate a trial data set under that scenario, following the specified allocation, follow-up, and analysis population rules.

Apply the fixed analysis.
Fit the prespecified Bayesian model and compute the chosen credible interval at the chosen nominal level.

Record containment.
Set $I_j=1$ when the interval contains $\theta_0$, and $I_j=0$ otherwise. Repeat for all $M$ replicates.

Summarize and check.
Report $\widehat{c}$, its Monte Carlo uncertainty, the nominal level, and the scenario. Check that simulation and interval computation behaved as intended.

The interval must target the same estimand whose generating value is used in the containment check. For example, if the analysis interval is for a log odds ratio, comparing it with a risk difference is not a coverage calculation. Choose the interval definition in advance as well: equal-tailed and highest-posterior-density intervals can differ, especially for skewed or constrained posteriors. A closed interval convention is standard; for continuous distributions, whether an endpoint is included has probability zero, but the convention can matter for discrete posterior distributions.

Worked Example: Counting Containments in a Small Simulation

Suppose a hypothetical calibration run generates $M=10$ independent trial data sets under one fixed treatment-effect scenario. A prespecified analysis produces a 95% credible interval for each data set. The containment indicators are $1,1,0,1,1,1,0,1,1,1$. The generating value is contained in 8 of the 10 intervals, so

\widehat{c}=\frac{8}{10}=0.80.

The estimated repeated-sampling coverage in this small run is 80%, compared with the 95% nominal credibility level. The plug-in Monte Carlo standard error is

SE_{\mathrm{MC}}(\widehat{c}) \approx\sqrt{\frac{0.80(1-0.80)}{10}} =\sqrt{0.016} =0.126.

That large standard error makes the run far too imprecise to establish a meaningful departure from 95%. The calculation is valid only if each data set was generated under the stated scenario and the same estimand, interval rule, and analysis were used for every replicate. Ten containments are useful for illustrating the procedure, not for making a stable calibration claim.

Why Nominal Credibility Need Not Equal Fixed-Value Coverage

The distinction can be seen in a simple normal model. Suppose $Y_i\mid\theta\sim N(\theta,\sigma^2)$ independently, with known $\sigma=10$, and use the prior $\theta\sim N(0,100)$. For $n=25$, the sample mean has sampling distribution $\bar{Y}\mid\theta_0\sim N(\theta_0,4)$. The posterior is normal with mean $w\bar{Y}$ and variance $V$, where

w=\frac{100}{100+4}=\frac{25}{26},\qquad V=\left(\frac{1}{100}+\frac{1}{4}\right)^{-1} =\frac{100}{26}\approx 3.846.

A central 95% posterior credible interval is $w\bar{Y}\pm 1.96\sqrt{V}$. Its half-width is $h=1.96\sqrt{100/26}\approx3.844$. At a fixed generating value $\theta_0$, this interval contains $\theta_0$ precisely when

\frac{\theta_0-h}{w}\leq \bar{Y}\leq\frac{\theta_0+h}{w}.

The bounds are shifted by the prior-driven shrinkage of the posterior mean. Because $\bar{Y}$ has mean $\theta_0$, that shift can affect coverage, particularly for generating values far from the prior mean.

Worked Example: Exact Coverage Under a Normal Model

Use the model and prior above, and consider the hypothetical fixed generating value $\theta_0=20$. The interval contains 20 when

\frac{20-3.844}{25/26}\leq\bar{Y}\leq \frac{20+3.844}{25/26}, \qquad\text{or approximately}\qquad 16.802\leq\bar{Y}\leq24.798.

Under this scenario, $\bar{Y}\sim N(20,4)$, so its standard deviation is 2. Standardizing the two bounds gives

P_{20}\{20\in C(D)\} =P\left(\frac{16.802-20}{2}\leq Z\leq \frac{24.798-20}{2}\right) =P(-1.599\leq Z\leq2.399) \approx0.937,

where $Z\sim N(0,1)$. Thus, in this fully specified model, the 95% posterior interval has approximately 93.7% repeated-sampling coverage at $\theta_0=20$. This calculation checks the containment rule analytically; a simulation under the same assumptions should estimate a value near 0.937, allowing for Monte Carlo variation. The normal sampling model, known variance, prior, sample size, and interval construction are all part of this conclusion.

Prior-Averaged Coverage and Fixed-Scenario Coverage

There is an important contrasting result. Under a proper Bayesian model, if the parameter is first drawn from the prior and data are then drawn from the model, the prior-averaged coverage of an exact posterior credible set is its posterior credibility level. This is a prior-predictive average, not a promise of that coverage at every fixed parameter value.

Derivation: Let $C(D)$ be a credible set satisfying $P\{\theta\in C(D)\mid D\}=1-\alpha$ for the observed data, under the prior and sampling model. Averaging its posterior probability over the prior-predictive distribution of $D$ gives $$ P\{\theta\in C(D)\} =E_D\!\left[P\{\theta\in C(D)\mid D\}\right] =E_D(1-\alpha) =1-\alpha. $$ This probability averages over both the prior draw of $\theta$ and the data conditional on that draw. It does not imply $P_{\theta_0}\{\theta_0\in C(D)\}=1-\alpha$ for each fixed $\theta_0$.

Consequently, the calibration target must match the scientific question. A fixed-value analysis evaluates coverage at a stated treatment effect and nuisance-parameter configuration. Prior-averaged coverage evaluates behavior over the prior’s distribution of parameter values. Those summaries can differ; reporting one as if it were the other obscures which scenarios drive the result.

Worked Example: Comparing Coverage Across Two Scenarios

Suppose 1,000 independent hypothetical trial simulations are run at each of two fixed values under the normal model above. At $\theta_0=2$, 954 intervals contain the generating value; at $\theta_0=20$, 937 do. The estimates and plug-in Monte Carlo standard errors are

\widehat{c}_{2}=\frac{954}{1000}=0.954,\qquad SE_{\mathrm{MC}}(\widehat{c}_{2}) \approx\sqrt{\frac{0.954(0.046)}{1000}}=0.00662; $$ $$ \widehat{c}_{20}=\frac{937}{1000}=0.937,\qquad SE_{\mathrm{MC}}(\widehat{c}_{20}) \approx\sqrt{\frac{0.937(0.063)}{1000}}=0.00768.

The first estimate is close to the exact coverage at $\theta_0=2$, about 0.954, and the second is close to the exact value at $\theta_0=20$, about 0.937. The difference illustrates why calibration at a single parameter value can be incomplete. Each estimate has nontrivial Monte Carlo uncertainty, so small differences should not be overinterpreted. The results describe these two fixed scenarios only; they do not establish coverage for every possible treatment effect or for a prior-averaged population.

Diagnostics, Alternatives, and Common Pitfalls

A coverage estimate is only as trustworthy as the simulation and interval computation behind it. Useful checks include confirming that every replicate uses the intended generating value, verifying the interval endpoints and containment indicator on selected data sets, and checking that independent replicates are actually independent. If posterior computation uses a numerical sampler, monitor convergence and Monte Carlo error within each replicate: noisy or failed posterior summaries can distort interval endpoints and therefore the coverage rate. Apply a prespecified rule for failed fits and report how often it occurs rather than silently dropping those trials.

When coverage is near 0 or 1, the plug-in normal approximation $\widehat{c}\pm1.96SE_{\mathrm{MC}}$ may be poor or extend beyond the possible range from 0 to 1. A binomial confidence interval, such as the Wilson interval, is a more reliable summary in such cases. Any such interval quantifies uncertainty due to the finite number of simulations, not the probability that the Bayesian interval contains the unknown parameter in a particular realized trial.

Approach	What it answers	Main qualification
Fixed-scenario simulation	How often intervals contain a specified generating value under a specified trial scenario	Requires enough independent replicates for useful Monte Carlo precision
Analytic repeated-sampling calculation	The coverage probability implied by a tractable model and interval rule	Available only when the probability can be derived or computed reliably
Prior-predictive averaging	Average containment probability over parameter values drawn from the prior and data from the model	Does not establish coverage at each fixed parameter value

An analytic benchmark is particularly valuable when available: agreement between simulation and an exact calculation can help expose coding errors. Simulation remains useful when the model is too complicated for a tractable coverage formula, but its conclusions are conditional on the chosen scenarios and implementation. Neither approach alone establishes robustness to data-generating mechanisms that were not examined.

Wrong reasoning: “The interval is 95% credible, so it must contain the true effect in 95% of repeated trials.” Correction: 95% is a posterior probability conditional on the model, prior, and observed data. Repeated-sampling coverage at a fixed generating value is a separate probability that must be evaluated.
Wrong reasoning: “The simulated coverage is 94%, so the method is definitely under-covering.” Correction: Compare the difference with Monte Carlo uncertainty, and consider whether the run is precise enough to distinguish it from the nominal level.
Wrong reasoning: “Coverage averaged over the prior is 95%, so each treatment effect has 95% coverage.” Correction: An average can equal 95% while coverage varies across fixed values, as the normal example demonstrates.

Key takeaway: Coverage calibration counts interval containments across repeated trials under a specified truth. Report the scenario, nominal level, number of replicates, estimated coverage, and Monte Carlo uncertainty together; do not confuse a prior-averaged guarantee with fixed-value performance.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In 2,000 independent simulations at a fixed generating value, 1,880 intervals contain that value. What is the estimated coverage, and what is its approximate Monte Carlo standard error?
Why does a 95% posterior credible interval not necessarily have 95% repeated-sampling coverage at every fixed treatment effect?
A simulation averages coverage over parameter values drawn from the prior. What probability does this estimate target, and how does it differ from fixed-value coverage?
Two analyses produce different types of 95% credible intervals from the same posterior. Name one reason their simulated coverage could differ.

Solutions. (1) The estimate is $1880/2000=0.940$. Its approximate Monte Carlo standard error is $\sqrt{0.940(0.060)/2000}=\sqrt{0.0000282}\approx0.00531$, or about 0.53 percentage points. (2) Credibility is the posterior probability conditional on observed data and the prior-model combination; coverage is the repeated-data probability at a fixed generating value. Prior information can shift posterior intervals, so the latter need not equal the nominal level. (3) It estimates prior-predictive average coverage, averaging over both parameter values drawn from the prior and data generated conditional on those values. It does not give coverage conditional on one particular fixed value. (4) Equal-tailed and highest-posterior-density intervals can have different endpoints, especially for an asymmetric posterior; different containment rates can result.