Simulation-Based Calibration of Fine-Mapping Posteriors

Why Simulate to Check Fine-Mapping Posteriors?

A fine-mapping analysis assigns posterior probabilities to candidate causal variants or configurations. Those probabilities depend on the likelihood, the prior, the LD information, and the inference algorithm. A simulation-based calibration check asks a direct question: when data are generated from the model being fitted, do the reported posterior probabilities behave like probabilities of the simulated causal states?

This is not a test of whether a variant is truly causal in an external study. In a simulation, the causal configuration is known because it was generated. Repeating the process lets us compare posterior inclusion probabilities with the frequency that variants actually were causal in those simulated replicates. The check is meaningful only relative to the model used to generate the simulations.

Specify the Generative Model Before Simulating

Let \(C\) denote a causal configuration: the set of variants designated causal in a replicate. Let \(R\) be the LD matrix used by the fine-mapping model, and let \(D\) denote the association data supplied to the inference procedure, such as a vector of summary statistics. The simulation must specify a prior \(p(C)\), an effect-size distribution conditional on \(C\), and a data-generating likelihood \(p(D\mid C,R)\). The same assumptions must be used when computing the posterior being checked.

For example, a simulation may draw \(C\) from a prior on variant configurations, draw nonzero effects for variants in \(C\) from the chosen effect-size prior, and then generate association statistics using the likelihood that accounts for \(R\). The exact likelihood depends on the fine-mapping model and data representation. Whatever likelihood is used, the simulation must preserve the LD structure and other modeled features that influence the association statistics.

For each replicate \(r\), record the true inclusion indicator \(C_{rj}\), which equals 1 if variant \(j\) was designated causal and 0 otherwise, and the posterior inclusion probability \(\widehat{p}_{rj}\) computed from the simulated data. A hat is used here to emphasize a computed posterior quantity; it is not a population proportion. With \(M\) simulated replicates, the empirical causal frequency for variant \(j\) is \(M^{-1}\sum_{r=1}^{M} C_{rj}\), while the mean reported PIP is \(M^{-1}\sum_{r=1}^{M}\widehat{p}_{rj}\).

The order matters. If causal configurations are drawn from one prior but inference uses another, the exercise no longer checks calibration under a single matched Bayesian model. It may still be a useful robustness or sensitivity analysis, but a mismatch should be intentional and reported as such.

What Calibration Predicts

Under the specified Bayesian model, the posterior inclusion probability for variant \(j\) in data set \(D\) is \(P(C_j=1\mid D)\). If the simulation draws \(C\) from the same prior and \(D\) from the same likelihood used for inference, then the posterior probability has a precise average relationship to the simulated truth.

Here \(B\) is any PIP range, such as 0.7 to 0.8. The first equality uses the fact that \(C_j\) is a 0-or-1 indicator: its conditional mean is its conditional probability of being 1. The second line applies iterated expectation, and the last equality follows because \(\widehat{p}_j\) is determined by the observed data and therefore is fixed when conditioning on \(D\). Since membership in \(B\) is also determined by the PIP, the result says that among simulated variant-replicate records with PIPs in a range, the mean PIP should match the causal frequency, up to simulation error.

This is calibration, not a demand that each individual probability be correct or that every bin match exactly in a finite run. It is also a statement under the assumed generative model. If the true data-generating process differs from that model, a calibration discrepancy may reflect that difference, an inference approximation, or an implementation error.

Probability Bins and Empirical Frequencies

A practical calibration plot groups variant-replicate records into PIP ranges. For each bin, calculate the average PIP and the fraction of records whose true indicator is 1. Plot the causal fraction against the average PIP. Perfect calibration corresponds to points on the diagonal. Bins should be chosen and described clearly; bins that are too narrow can contain too few records for a stable frequency estimate.

Pooling records across variants can be useful, but it changes the target: the result describes calibration for the pooled set of variant-replicate events. Report how records were pooled, especially if variants have very different prior probabilities or roles in the design. A pooled plot can conceal variant-specific miscalibration, so stratified summaries may be informative when the simulation has meaningful groups.

Assessing the Whole Simulation and the Inference Procedure

Calibration should be examined at more than one level. The overall mean causal frequency for a variant should agree, in expectation, with its mean PIP under matched prior and likelihood. Calibration bins check whether agreement holds across low, medium, and high probabilities rather than only on average. A method can have a correct overall average while systematically assigning probabilities that are too extreme in some ranges and too conservative in others.

The simulation should also check the complete implementation, not only a final summary. Confirm that the causal configuration is drawn from the intended prior, that variant indexing and allele orientation are consistent, that the correct LD matrix enters both generation and inference, and that posterior probabilities are computed for the same event recorded as truth. If configurations allow multiple causal variants, compare each variant’s causal indicator with its marginal PIP; do not compare a configuration-level probability with a variant-level indicator.

Monte Carlo Variation and Interpretation

Finite simulations produce random differences between a bin’s mean PIP and its observed causal frequency, even when the model and inference are correctly matched. The number of independent replicates, the number of records in each bin, and dependence among records all affect how much variation to expect. Report bin counts and avoid interpreting a noisy extreme bin as decisive evidence of failure.

A useful diagnostic is to repeat the simulation with more independent replicates or to calculate uncertainty intervals for bin frequencies. For a bin with \(m\) independent records and observed causal fraction \(\hat{q}\), a rough standard error is \(\sqrt{\hat{q}(1-\hat{q})/m}\). This formula is an approximation and is not appropriate without adjustment when records are clustered or otherwise dependent. Calibration plots should be read alongside the simulation design and uncertainty, not as a binary pass-or-fail test.

SBC can also use posterior-rank diagnostics. For a scalar quantity drawn from the prior, generate data, draw posterior samples, and rank the simulated true value among those posterior draws. With exact posterior computation, appropriately independent draws, and randomized handling of ties for discrete quantities, the ranks should be uniform over their possible positions under the matched model. Such rank checks can reveal some implementation problems, but they answer a different diagnostic question from a PIP reliability plot. A uniform rank pattern does not replace checking whether variant-level PIPs match simulated causal frequencies.

Common Pitfall

• Wrong reasoning: “The simulated PIPs average to 0.20, and 20% of variants are causal, so every part of the posterior is calibrated.”
• Correction: Agreement of overall averages is necessary under the matched model but can hide errors across probability ranges. Compare mean PIP with observed causal frequency in bins, report bin sizes, and account for simulation uncertainty and dependence.
• Wrong reasoning: “If calibration fails under one simulation, the method is universally invalid.”
• Correction: The conclusion is conditional on the simulation assumptions. A mismatch may reveal model misspecification, an incorrect implementation, an inadequate approximation, or simply Monte Carlo variation. Check which assumptions were used to generate and analyze the data before generalizing.