Credible-Set Probability and Calibration

Two Different Probabilities

A fine-mapping credible set summarizes uncertainty about which candidate variant is causal, conditional on the observed data and a specified statistical model. Its nominal level is a statement about posterior probability. Repeated-sampling coverage asks a different question: if the causal variant were fixed and the study were repeated, how often would the resulting set contain it? These probabilities can agree under an average over the model's prior distribution while differing substantially for particular variants.

For this tutorial, consider the single-causal-variant setting: exactly one of \(m\) candidate variants is causal. Let \(C\in\{1,\ldots,m\}\) denote its index, let \(D\) denote the observed data, and let \(q_j(D)=P(C=j\mid D)\) be the posterior probability for variant \(j\). A credible set \(S(D)\) is a data-dependent collection of variants. Its posterior mass is the sum of the posterior probabilities of its members.

Definition: The posterior mass of a set \(S(D)\), given data \(D\), is \(Q(S(D)\mid D)=\sum_{j\in S(D)}q_j(D)\). A set with nominal level \(1-\alpha\) has posterior mass at least \(1-\alpha\), under the fitted model and the observed data. The repeated-sampling coverage for fixed variant \(j\) is \(c_j=P_{D\mid C=j}\{j\in S(D)\}\): the probability, over data generated when \(j\) is truly causal, that the analysis produces a set containing \(j\).

The posterior mass is conditional on the observed data; coverage is conditional on a fixed true causal variant and averages over possible repeated data sets. A set may contain variants with substantial posterior probability yet miss the true variant in some repeated studies. Conversely, a set may include the true variant even when its posterior mass is not exactly the nominal level.

What a Nominal Level Guarantees

The key relationship is an averaging identity. Suppose the single-causal-variant model is correctly specified: the candidate list includes the true causal variant, the prior probabilities and data likelihood describe the relevant data-generating process, and the posterior is calculated from that prior and likelihood. Then the probability that the set contains the causal variant, averaged over the prior distribution of \(C\) and the resulting data, equals the expected posterior mass of the set.

\begin{aligned} P\{C\in S(D)\} &=\sum_d\sum_{j=1}^{m}P(C=j,D=d)\,I\{j\in S(d)\}\\ &=\sum_d P(D=d)\sum_{j\in S(d)}P(C=j\mid D=d)\\ &=E_D\!\left[Q(S(D)\mid D)\right]. \end{aligned}

Here \(I\{\cdot\}\) equals 1 when its condition is true and 0 otherwise; the sums are integrals when the data are continuous. The first equality partitions the joint probability by possible causal variant and data. The second uses \(P(C=j,D=d)=P(D=d)P(C=j\mid D=d)\). The final sum is precisely the expected posterior mass of the selected set.

Therefore, if the set has posterior mass at least 0.95 for every possible data set, its prior-predictive inclusion probability is at least 0.95. Prior-predictive means that the causal variant is first drawn from the model's prior and then data are generated under the model. It does not follow that \(c_j\geq0.95\) for every fixed variant \(j\). The identity only implies the prior-weighted average:

P\{C\in S(D)\}=\sum_{j=1}^{m}P(C=j)c_j.

Thus, posterior credibility and repeated-sampling coverage are connected, but they are not interchangeable. A nominal set can have coverage above 95% for some variants and below 95% for others while its prior-weighted average still meets or exceeds 95%.

Worked Example: Calculating Posterior Mass

Suppose a hypothetical single-causal-variant analysis assigns posterior probabilities 0.62, 0.24, 0.09, and 0.05 to four candidate variants. A reported set contains the first three. What is its posterior mass, and what does that number say?

The mass is the sum of the included variants' posterior probabilities:

Q(S\mid D)=0.62+0.24+0.09=0.95.

Conditional on these data and the fitted model, the posterior probability that the causal variant is among these three is 95%. This is not a statement that this particular set will contain the causal variant in 95% of hypothetical repetitions with any one of the four variants fixed as causal. That latter claim requires evaluating the repeated-sampling coverage for each fixed variant.

Average Calibration Is Not Variant-by-Variant Calibration

The distinction can be seen by considering a hypothetical repeated-sampling assessment. Suppose variants A and B have prior probabilities 0.80 and 0.20 in a correctly specified single-causal model. Across repeated data sets generated with A fixed as causal, the nominal set contains A in 990 of 1,000 runs. With B fixed as causal, it contains B in 790 of 1,000 runs. The estimated coverages are \(990/1000=0.99\) and \(790/1000=0.79\), respectively.

The prior-weighted average of these coverage rates is:

$$ 0.80(0.99)+0.20(0.79)=0.792+0.158=0.950. $$

The average is 95%, even though the estimated coverage for B is 79%. This illustrates why an average calibration statement cannot be reported as a guarantee for each variant. In a finite simulation, these rates are estimates and have Monte Carlo uncertainty; the arithmetic illustrates the distinction, not a proof that any particular fitted analysis has these rates.

Coverage can differ across variants because their evidence may be easier or harder to distinguish from that of nearby alternatives, or because the analysis behaves differently in different parts of the candidate set. A posterior set can be well calibrated on average under its prior while remaining poorly calibrated for a particular causal state. When the prior assigns little weight to a region of causal states, average calibration can also hide performance in that region.

Worked Example: Using the Calibration Identity

Imagine a hypothetical model with two possible causal variants. The prior probability of A is 0.70 and of B is 0.30. A repeated-sampling calculation under the model gives coverage 0.98 when A is fixed as causal and 0.88 when B is fixed as causal. Calculate the prior-predictive inclusion probability and interpret it.

Weight each fixed-variant coverage by that variant's prior probability:

P\{C\in S(D)\} =0.70(0.98)+0.30(0.88) =0.686+0.264 =0.950.

The prior-predictive inclusion probability is 95%. But the conditional coverage for B is 88%, not 95%. The average reaches the nominal level because coverage for A is higher and A receives greater prior weight. This calculation relies on the model's prior and likelihood being the data-generating mechanism; it is not a universal guarantee about real studies.

Assessing Calibration in Practice

A calibration assessment should specify what is held fixed and what varies. For repeated-sampling coverage, fix a causal variant \(j\), generate or obtain repeated data sets under a stated data-generating model, apply the same fine-mapping analysis and set rule to every data set, and estimate the fraction of sets containing \(j\). Repeating this for different variants estimates \(c_j\) across the candidate list. If simulation draws the causal variant from the prior instead, the resulting inclusion rate estimates prior-predictive coverage; it does not replace variant-specific results.

The assessment is only as informative as its assumptions. In the ideal calibration identity, the model's candidate variants, prior, and likelihood match the process that generated the data. If the candidate list omits the true causal variant, or if the assumed data-generating process is wrong, the identity does not establish calibration for the actual process. In genomic applications, features such as imperfectly represented variant relationships, effect-size assumptions, and data quality can affect conclusions. A simulation that reproduces the fitted assumptions can reveal behavior under those assumptions, but cannot by itself verify that they hold in the data.

Useful diagnostics include:

Report inclusion rates by fixed causal variant, or by relevant groups of variants, rather than only one aggregate rate.
State whether the simulation averages over a prior or fixes the causal variant in each run.
Record the distribution of posterior set masses and set sizes alongside inclusion rates; a high inclusion rate achieved by very large sets has a different practical cost from one achieved by compact sets.
Check simulation uncertainty. With \(R\) independent runs and estimated inclusion proportion \(\hat p\), a rough Monte Carlo standard error is \(\sqrt{\hat p(1-\hat p)/R}\). It quantifies simulation noise, not uncertainty about whether the model is correct.
Compare results across plausible data-generating conditions. Strong performance under only one assumed condition is limited evidence about performance elsewhere.

A common mistake is to read “95% credible set” as “this procedure covers every causal variant in 95% of repeated studies.” That reasoning treats posterior probability conditional on observed data as if it were a fixed-variant sampling guarantee. The correction is to state the target precisely: the set has at least 95% posterior mass under the fitted model for the observed data; repeated-sampling coverage for a fixed variant is a separate operating characteristic that must be evaluated.

Comparison and Limitations

A Bayesian credible set is constructed from posterior probabilities and is interpreted conditional on the data, prior, and model. A frequentist confidence-set procedure instead targets a repeated-sampling coverage property, typically under specified assumptions and for a defined parameter space. A procedure designed to achieve coverage for every fixed causal variant may use a different construction or produce larger sets. Neither description makes a procedure automatically reliable: Bayesian calibration depends on the adequacy of the prior and likelihood, while a coverage target for a frequentist procedure depends on its assumptions and implementation.

The posterior-mass and coverage distinction also does not settle whether the single-causal-variant model is appropriate. The results here condition on that setting; they do not establish that only one variant is causal. Nor does a high posterior mass establish that a particular member of the set is the causal variant. It quantifies uncertainty across the set under the model. Calibration is a property of behavior across data-generating repetitions, not a substitute for examining the scientific and statistical assumptions.

Worked Example: Interpreting a Coverage Report

In a hypothetical simulation, a nominal 95% set has estimated coverage 96% across runs that draw the causal variant from a specified prior. A separate evaluation fixing one low-prior variant as causal gives 81% coverage. What can be concluded?

The 96% rate describes the simulated prior-predictive average under the chosen data-generating conditions. It is consistent with the set having at least 95% posterior mass under the model, allowing for simulation variability and any excess mass from the set rule. The 81% rate indicates that, under the fixed-variant simulation condition, the set included that particular variant in 81% of runs. The aggregate result does not erase this lower conditional rate.

The results do not show that the procedure is calibrated for every possible causal variant, nor do they establish that the simulated model matches real genomic data. A careful report gives both the averaging scheme and the fixed-variant result, and treats the latter as a simulation estimate with Monte Carlo uncertainty.

Key takeaway: A nominal 95% credible set has at least 95% posterior mass conditional on the observed data and fitted model. Under a correctly specified Bayesian model, its prior-predictive inclusion probability equals its expected posterior mass, but its repeated-sampling coverage can vary across fixed causal variants.

Check Your Understanding

Answer each question by distinguishing the probability being discussed. Solutions are included below.

A set contains variants with posterior probabilities 0.55, 0.30, and 0.10. What is its posterior mass, and what does that mass mean?
State the difference between posterior mass and fixed-variant repeated-sampling coverage.
Two variants have prior probabilities 0.60 and 0.40, with simulated fixed-variant coverages 0.98 and 0.90. What is the prior-weighted coverage?
Why does a prior-predictive inclusion rate of 95% not guarantee 95% coverage for each variant?
Name one condition required for the calibration identity to describe the actual data-generating process.

Solutions. (1) The mass is \(0.55+0.30+0.10=0.95\). Conditional on the observed data and fitted model, the causal variant has 95% posterior probability of being in the set. (2) Posterior mass sums posterior probabilities for the observed data; fixed-variant coverage is the probability, across repeated data sets with a specified variant truly causal, that the set includes it. (3) The weighted coverage is \(0.60(0.98)+0.40(0.90)=0.588+0.360=0.948\), or 94.8%. (4) The identity constrains the prior-weighted average, so lower coverage for one variant can be offset by higher coverage for another. (5) For example, the candidate list must include the true causal variant, and the prior and likelihood must represent the data-generating mechanism.