Calibration Under Model Misspecification

Why Calibration Must Be Checked Under the Data-Generating Process

A fine-mapping procedure can return a credible set with 95% posterior mass and still fail to include the causal variant in 95% of comparable analyses. Posterior mass is computed under the fitted model; empirical coverage asks how often the reported set contains the truth across repeated data sets. These quantities agree in an important average sense when the model and its assumptions are correct, but agreement is not guaranteed when the fitted model misrepresents the causal-variant count, effect sizes, or linkage disequilibrium (LD).

Calibration under misspecification is therefore an evaluation problem: specify the truth-generating process, fit the proposed fine-mapping model, and measure how often its reported sets capture the defined target. This does not establish that one particular data set is wrong, nor does undercoverage prove which assumption failed. It shows that the procedure’s stated uncertainty is too optimistic for the simulated conditions being evaluated.

Definition: For replicate $r$, let $C_r$ be the set of truly causal variants and $S_r$ the reported credible set or union of sets being evaluated. The empirical coverage for the all-causal-variants target is $$ \widehat{\operatorname{Cov}}=\frac{1}{R}\sum_{r=1}^{R} I(C_r\subseteq S_r), $$ where $R$ is the number of independent simulated replicates and $I(\cdot)$ equals 1 when the event is true and 0 otherwise. For a single causal variant $J_r$, the target instead is $I(J_r\in S_r)$. State which target is used: capturing at least one causal variant is not the same as capturing every causal variant.

A nominal 95% credible set is intended to represent a posterior probability statement under the fitted model. In repeated simulations, compare its empirical coverage with 0.95, and report uncertainty in the estimate. When causal counts vary, report both a target appropriate to the scientific question and coverage stratified by the true count. Otherwise, good performance in mostly single-causal replicates can hide poor coverage when multiple variants are causal.

Why Correct Models Can Calibrate, and What Misspecification Changes

Consider a correctly specified Bayesian model for a single causal variant $J$, data $D$, and a procedure that reports $S(D)$ with posterior probability $P(J\in S(D)\mid D)$ at least 0.95. Under the joint prior-and-data-generating model, the law of total probability gives the following justification for average coverage:

P(J\in S(D)) =E_D\left[P(J\in S(D)\mid D)\right] \geq 0.95.

The equality averages over data sets generated by that same model and over the model’s distribution for the causal variant. It does not say that every repeated-sampling subgroup has exactly 95% coverage, or that coverage is exactly 95% rather than higher. Most importantly, if the actual causal-count distribution, effect-size distribution, or LD differs from the model used to compute the posterior, this argument no longer guarantees coverage under the actual process.

For multiple causal variants, use the event that matches the claim. If the claim is that every causal variant is contained in the reported union $S(D)$, the relevant posterior event is $C\subseteq S(D)$. A per-signal 95% set does not automatically have 95% probability of containing all causal variants jointly. Calibration comparisons are meaningful only when the simulated truth, fitted output, and coverage event refer to the same target.

Worked Example: Estimating Coverage and Its Simulation Error

Suppose 1,000 independent hypothetical regions are simulated with exactly one causal variant each. The fitted procedure reports a nominal 95% set in every region, and the true variant is inside 936 of the sets. Estimate coverage and quantify its Monte Carlo uncertainty.

The estimate is the number of successful captures divided by the number of replicates: $$ \widehat{\operatorname{Cov}}=\frac{936}{1000}=0.936. $$ Thus the empirical coverage is 93.6%, 1.4 percentage points below the nominal 95%.

Treating the capture indicators as independent Bernoulli outcomes, the estimated standard error is $$ SE=\sqrt{\frac{\widehat{\operatorname{Cov}}(1-\widehat{\operatorname{Cov}})}{R}} =\sqrt{\frac{0.936(0.064)}{1000}} =\sqrt{0.000059904} \approx 0.00774. $$ An approximate 95% interval is $0.936\pm1.96(0.00774)$, or about 0.921 to 0.951. This interval describes simulation uncertainty in the estimated coverage under this simulation setup; it is not a posterior interval for a causal effect. The observed difference from 0.95 is small relative to this uncertainty, so these 1,000 replicates alone do not establish substantial undercoverage.

Departures That Can Make Sets Overconfident

A misspecification can affect which variants receive posterior support and how concentrated that support appears. Undercoverage is not an automatic consequence of every mismatch: its direction and size depend on signal strength, LD structure, sample size, and the particular fitted model. The following departures are useful to test because each can make posterior uncertainty appear smaller than it is.

Causal-count mismatch. If the data contain more causal signals than the fitted model can represent, the fitted model may explain multiple patterns with too few signals. A reported set can then be narrow around a selected explanation while omitting another causal variant. Assess all-causal coverage as well as per-signal capture; these measure different failures.
Effect-size distribution mismatch. A fitted prior that assigns too little probability to the effect sizes actually generated can alter Bayes factors and posterior concentration. For example, a tail-heavier true distribution than the fitted distribution may produce signals whose evidence is poorly represented by the assumed model. The consequence is not universally undercoverage, so evaluate it in the intended setting rather than infer it from the direction of the mismatch alone.
LD mismatch. Posterior comparison among variants depends on their correlation structure. If the reference LD differs from the LD that generated the summary statistics, the fitted model can misallocate evidence among correlated variants and assign excessive support to the wrong location. A high posterior probability conditional on incorrect LD is not evidence that the LD assumption is harmless.

Diagnostics should expose the conditions under which performance changes. Record coverage separately by true causal count, signal strength, and other simulation strata relevant to the design. Also compare empirical capture with the posterior confidence assigned to the same event, such as the fitted signal’s inclusion probability for its set. A mismatch between the event used for posterior confidence and the event used for empirical coverage makes the comparison uninterpretable.

Worked Example: Extra Causal Variants and All-Causal Coverage

In a hypothetical simulation, each of 1,000 regions has two causal variants. A fitted model allows at most one signal. The evaluated output is the union of the reported sets, and the success criterion is that this union contains both true variants. Suppose both are captured in 742 regions.

The empirical all-causal coverage is $$ \widehat{\operatorname{Cov}}_{\mathrm{all}}=\frac{742}{1000}=0.742. $$ Its estimated standard error is $$ SE=\sqrt{\frac{0.742(1-0.742)}{1000}} =\sqrt{\frac{0.191436}{1000}} \approx 0.0138. $$ The approximate 95% interval is $0.742\pm1.96(0.0138)$, or about 0.715 to 0.769. This is substantially below 0.95 in this hypothetical setting.

Suppose the fitted model’s single-signal sets have an average posterior mass of 0.97 for the event that the modeled signal’s causal variant lies in its set. That number does not contradict the 74.2% all-causal coverage: the posterior statement concerns one modeled signal, whereas evaluation requires capturing two true variants. The model’s restricted causal count makes the distinction especially important. To diagnose the cause, compare with a simulation in which the fitted model can represent two signals while holding the other generating conditions fixed.

Worked Example: Effect-Size Misspecification

Consider 800 hypothetical single-causal regions generated with a heavy-tailed effect-size distribution. The fitted model instead uses a narrower effect-size distribution. Define success as inclusion of the true variant in the reported nominal 95% set. Suppose 704 sets capture it.

The empirical coverage is $$ \widehat{\operatorname{Cov}}=\frac{704}{800}=0.88. $$ The estimated standard error is $$ SE=\sqrt{\frac{0.88(0.12)}{800}} =\sqrt{0.000132} \approx 0.0115. $$ The approximate 95% interval is $0.88\pm1.96(0.0115)$, or about 0.857 to 0.903. If the procedure also reports average posterior mass of 0.97 for its nominal sets, the gap between 97% reported mass and 88% empirical capture indicates overconfidence for this specified generating process.

This result is conditional on the hypothetical simulation design; a different mismatch need not have the same direction or magnitude. A useful diagnostic is to repeat the evaluation with the fitted and generating effect-size distributions matched, leaving the causal count, LD, and other features unchanged. If coverage improves, the controlled comparison supports an effect-distribution contribution rather than proving it is the only source of error.

Worked Example: LD Mismatch and Misallocated Posterior Support

Suppose 1,000 hypothetical single-causal regions are generated using one LD structure, but fine-mapping uses a reference LD structure that differs from it. Of the nominal 95% sets, 870 contain the true variant. The empirical coverage is $$ \widehat{\operatorname{Cov}}=\frac{870}{1000}=0.870. $$ The estimated standard error is $$ SE=\sqrt{\frac{0.870(0.130)}{1000}} =\sqrt{0.0001131} \approx 0.0106. $$ An approximate 95% interval is $0.870\pm1.96(0.0106)$, or about 0.849 to 0.891.

If the same simulation with matched generating and fitted LD gives 948 captures out of 1,000, its coverage is $948/1000=0.948$. The contrast is evidence that the LD mismatch is associated with undercoverage in this setup. It does not show that every reference mismatch will cause undercoverage, and it does not identify which particular LD feature caused it. Inspecting coverage by relevant LD or signal strata can help localize where the failure occurs.

Designing a Useful Calibration Study

A calibration study is informative only when its simulation and scoring rules are explicit. In particular, distinguish variation due to the fitted method from variation due to changing several generating assumptions at once. A controlled comparison changes one feature while keeping the others fixed; a combined stress test can show performance under a challenging joint departure but cannot attribute the result to one cause.

Define the target.
Specify whether success means capturing one signal, every causal variant, or a designated signal’s variant. Match that event to the posterior quantity being assessed.

Specify the generating conditions.
State the causal count, effect-size distribution, LD used to generate data, and relevant sampling conditions. Simulate known causal variants so capture can be scored.

Fit and record the procedure’s output.
Use the intended fine-mapping workflow and record each set, its stated posterior mass, and the replicate’s truth. Do not exclude difficult or weak-signal replicates after seeing whether they were captured.

Estimate and stratify coverage.
Compute the capture proportion and its simulation uncertainty, then summarize by prespecified features such as causal count or signal strength. Compare matched and deliberately mismatched conditions.

Coverage estimates also have limitations. They apply to the simulated distributions and assumptions, not automatically to every real genomic region. Finite replicate counts add Monte Carlo error, and pooling heterogeneous regions can conceal subgroup undercoverage. Conversely, a single low estimate may be chance variation. Report the number of replicates, the success definition, the nominal level, and uncertainty; avoid treating a small numerical difference as decisive without considering its precision.

Common Pitfalls and Alternatives

Wrong reasoning: “The set has posterior mass 0.95, so it must capture the truth in 95% of analyses.” Correction: that posterior statement is conditional on the fitted model and data. Repeated-simulation coverage under the actual or specified generating process must be evaluated separately.
Wrong reasoning: “The set contains at least one causal variant, so coverage is successful.” Correction: this only evaluates an at-least-one target. If the claim concerns all causal variants, score $C\subseteq S$.
Wrong reasoning: “Undercoverage proves LD mismatch caused the failure.” Correction: undercoverage establishes a performance problem for the tested conditions. Controlled comparisons are needed to attribute it to a particular departure.

Compared with relying only on posterior mass from the fitted model, repeated simulation directly checks capture when the truth is known. Compared with a single combined stress test, matched and one-feature-at-a-time comparisons are more useful for identifying a likely source of failure. Both approaches are limited by how well the chosen simulations represent the settings where the fine-mapping results will be used.

Key takeaway: A credible set’s nominal posterior mass is not a guarantee of empirical coverage under misspecification. Define the capture event precisely, simulate known causal truths under explicit departures, quantify Monte Carlo error, and look for undercoverage in the conditions that matter.

Check Your Understanding

Answer each question, then compare with the solutions below.

In 500 independent simulations, 455 sets contain the single true causal variant. What is the empirical coverage?
Why can a 95% per-signal credible set fail to provide 95% coverage of all causal variants in a region with multiple causal variants?
A model reports posterior mass 0.98 for a set, while repeated simulations under a specified LD mismatch give 84% capture. What does this establish, and what does it not establish?
Why is it useful to compare a matched simulation with one in which only the effect-size distribution changes?
If the estimated coverage is 0.90 from 100 independent replicates, calculate its approximate standard error using the binomial formula.

Solutions. (1) Coverage is $455/500=0.91$, or 91%. (2) Per-signal coverage concerns a particular signal’s variant; capturing every causal variant is a joint event and can have lower probability. (3) It establishes undercoverage for that simulation setup relative to the reported 98% mass; it does not prove that LD mismatch is the sole cause or that all LD mismatches behave this way. (4) Holding other conditions fixed makes a coverage difference more informative about the effect-size departure than a comparison that changes several features at once. (5) $SE=\sqrt{0.90(0.10)/100}=\sqrt{0.0009}=0.03$, or 3 percentage points.