What Empirical Coverage Measures
A credible set is intended to contain a causal variant with a specified posterior probability under a fine-mapping model. That posterior statement and an empirical coverage estimate answer different questions. Empirical coverage asks: across repeated datasets with known causal variants, how often does the reported set include the truth? It is a property of a defined evaluation procedure over a defined collection of datasets, not a posterior probability for any one variant.
The evaluation must specify what counts as one observation. A common choice is one signal in one simulated replicate, paired with its corresponding credible set and known causal variant. If a dataset contains several signals, a signal-to-set correspondence must be defined before counting coverage. Otherwise, the same true variant could be counted more than once, or a set for one signal could be credited for capturing a variant belonging to another.
This definition is for one causal variant per evaluation unit. If an evaluation unit has several known causal variants, there are at least two distinct targets: whether a particular signal’s set contains its matched causal variant, or whether a set contains all causal variants in a specified truth set. The second target has indicator \(Y_i=1\) only when every required variant is included. These estimands can have different values, so the target and matching rule should be stated explicitly. Do not silently treat “contains at least one causal variant” as equivalent to “contains the causal variant for this signal.”
The denominator matters just as much as the numerator. Some simulation designs produce a set for every replicate; others produce sets only when a signal passes a reporting or purity filter. If coverage is computed only among reported sets, it is conditional on being reported. Report how many units were simulated, how many produced an eligible set, and which units form the denominator. A failure to produce a set is not automatically a covered observation.
Estimating Coverage and Its Uncertainty
Suppose the \(N\) evaluation units are independent draws from the target simulation or benchmark distribution, and each uses the same prespecified fine-mapping and set-construction procedure. Each \(Y_i\) is then a Bernoulli variable with coverage probability \(c=P(Y_i=1)\). The sample mean \(\hat{c}\) estimates \(c\). Since a Bernoulli variable has variance \(c(1-c)\), the variance of the mean is \(c(1-c)/N\); replacing the unknown \(c\) by \(\hat{c}\) gives the familiar estimated standard error.
A large-sample Wald interval is \(\hat{c}\pm 1.96\widehat{SE}(\hat{c})\) for an approximate 95% confidence interval. Its repeated-sampling interpretation is that intervals constructed this way would contain the target coverage probability about 95% of the time under the assumptions. It is not a 95% probability statement about the fixed coverage probability in the interval. Wald intervals can perform poorly with small samples or estimates near zero or one; Wilson or exact binomial intervals are generally safer in those settings.
The binomial calculation assumes independent evaluation units with a common coverage probability. It is reasonable, for example, when each unit is generated by an independent simulation replicate under the same design. It may fail if multiple signals or sets from one replicate are treated as separate independent observations, or if benchmark loci are related or otherwise clustered. In those cases, the point estimate can still describe the observed fraction, but the ordinary binomial standard error can understate uncertainty.
Worked Example: Overall Coverage Across Simulations
Suppose 100 independent hypothetical simulation replicates each contain one causal variant and produce one corresponding credible set. The true variant is in 92 of those sets. The simulation design and set-construction rule are the same across replicates, so the replicate-level indicators are the evaluation units.
The estimated coverage is
Thus, 92.0% of these simulated sets contained their known causal variant. The estimated standard error is
The approximate Wald 95% interval is
The estimate is 92.0%, with an approximate 95% confidence interval from 86.7% to 97.3%. The interval includes 95%, but that alone does not establish that coverage equals 95%; it indicates that this sample is not precise enough for the interval to exclude that value under this calculation. A nominal posterior level and observed empirical coverage should be reported separately.
Stratifying by Set Size or LD
Overall coverage averages over the mix of conditions in the evaluated data. It can conceal differences among conditions that matter scientifically. For example, coverage may vary with the number of variants in a set, or with the LD structure of the region. Stratification estimates coverage separately in prespecified groups, while retaining the same evaluation unit and truth-matching rule in every group.
For stratum \(g\), let \(I_g\) be the indices of evaluation units in that group, and let \(N_g\) be the number of those units. The stratum-specific estimate is
A set-size stratum can be defined by the number of variants in \(C_i\), using cut points chosen before examining coverage results. An LD stratum might use a prespecified summary of LD among variants in the set, such as a maximum pairwise \(r^2\) category. The exact definition, thresholds, and treatment of sets for which the summary cannot be calculated should be documented. Since set size and LD may be associated, a difference between size groups alone does not show that size caused a coverage difference.
Worked Example: Coverage by Credible-Set Size
Consider 100 hypothetical independent evaluation units, classified in advance by set size. Among 50 sets containing at most 5 variants, 42 include the known causal variant. Among 50 larger sets, 46 include it. These counts yield:
The pooled estimate is the total number covered divided by the total number evaluated:
The observed difference, large minus small, is \(0.920-0.840=0.080\), or 8.0 percentage points. Under independent units in both groups, an approximate standard error for this difference is
The approximate 95% interval for the difference is \(0.080\pm1.96(0.0645)=(-0.046,\ 0.206)\). This interval is wide and includes zero, so these hypothetical data do not provide a precise estimate of the difference. The pooled estimate is 88.0%, a weighted average reflecting the equal stratum sizes; it is not evidence that every set-size group has coverage of 88.0%.
When strata have unequal sizes, the pooled estimate weights each stratum by its share of evaluation units: \(\hat{c}_{\mathrm{pooled}}=\sum_g(N_g/N)\hat{c}_g\). An unweighted average, \(\sum_g\hat{c}_g/G\), answers a different question: the average of the stratum-specific rates when each stratum receives equal weight, regardless of how common it is. Choose the summary to match the target population and label it accordingly.
Uncertainty With Repeated Signals and Benchmark Loci
When a simulated replicate contributes several signals, their coverage indicators may be dependent because they share the same genotypes, LD structure, phenotype, or fitting result. Counting every signal as independent can produce an overly narrow interval. The unit of resampling or variance estimation should reflect the independent sampling unit: often the replicate, locus, or benchmark region rather than each set.
One practical approach is a cluster bootstrap. Resample independent replicates or loci with replacement, retaining all their associated signals and sets together. Recalculate the coverage estimate in each resampled dataset, and use the distribution of those estimates to form an interval. This preserves within-cluster dependence. It does not fix a nonrepresentative benchmark set or a poorly defined target; it quantifies sampling variation relative to the clusters and population represented.
Worked Example: Coverage in an LD Stratum
Suppose 40 hypothetical independent benchmark regions were classified before analysis as having high LD, using a stated threshold for the maximum pairwise \(r^2\) among variants in each set. Each region contributes one matched causal-variant/set pair. The causal variant is included in 28 sets, so estimated coverage in this stratum is
The estimated binomial standard error is \(\sqrt{0.700(0.300)/40}=\sqrt{0.00525}=0.0725\). A Wald interval is \(0.700\pm1.96(0.0725)=(0.558,\ 0.842)\). This approximate interval describes uncertainty in the high-LD stratum coverage under independent, similarly sampled regions; it is not uncertainty about whether a particular region’s causal variant is in its set.
If instead the 40 set records came from 10 regions with four dependent signals per region, the calculation treating all 40 as independent would not be justified. Resampling the 10 regions as clusters would keep each region’s four records together. With only 10 independent clusters, uncertainty may be substantial; the apparent number of set records should not be mistaken for 40 independent sources of information.
Diagnostics, Comparisons, and Limitations
Before interpreting an empirical coverage estimate, check that the known truth is available and consistently represented, that each set is matched to the intended causal signal, and that inclusion is evaluated using variant identity rather than a nearby proxy. Confirm the denominator, set-generation rule, nominal level, and any exclusions. Tabulate the number of units in each stratum as well as each coverage numerator; a high rate based on very few units is imprecise. Also inspect whether strata were specified in advance and whether some groups contain too few independent replicates for useful comparisons.
Empirical coverage complements rather than replaces posterior probabilities. A posterior inclusion probability or a credible set’s nominal level summarizes model-based uncertainty for observed data under the fitted model. Empirical coverage evaluates repeated outcomes against known truth in simulations or benchmarks. The latter depends on which causal configurations, effect sizes, LD patterns, sample sizes, and analysis choices were represented. A benchmark result therefore need not generalize to settings absent from that benchmark.
A frequent pitfall is to say, “The sets are 95% credible, so 95% of them must contain the true variant,” and then interpret a finite-sample estimate below 95% as proof of failure. The correction is to treat 95% as the stated posterior target under the model and assess observed frequency with an interval and an explicit evaluation design. A low or imprecise estimate is information about performance in that design; it is not by itself a diagnosis of the cause.
Check Your Understanding
Answer each question, then compare with the solutions.
- In 80 independent hypothetical evaluation units, 68 sets contain their matched causal variants. Calculate the empirical coverage.
- Why can the unweighted mean of two stratum-specific coverage estimates differ from pooled coverage?
- A benchmark has 30 regions and 90 signal-level sets, with three sets per region. What is a concern with a binomial standard error that treats all 90 sets as independent?
- State one distinction between empirical coverage and a nominal posterior credible level.
- For multiple causal variants in one evaluation unit, name two possible coverage targets that must not be conflated.
Solutions. (1) \(\hat{c}=68/80=0.850\), so observed coverage is 85.0%. (2) The pooled estimate weights strata by their sample sizes, whereas the unweighted mean gives each stratum equal weight; they coincide when stratum sizes are equal or the rates happen to yield the same average. (3) Sets from the same region may be dependent. The independent units may be regions, so uncertainty should account for clustering rather than count all 90 sets as independent. (4) Empirical coverage is the observed repeated-sample fraction of known truths included in sets under a specified evaluation design; the nominal posterior level is a model-based probability statement for a set under the fitted model. (5) One target is whether a matched causal variant for a signal is in its corresponding set; another is whether a set contains all causal variants required by the truth definition.