Diagnostics for Unstable Fine-Mapping Results

Why Fine-Mapping Diagnostics Matter

A fine-mapping result can look precise in one output and still depend strongly on choices or computational details. The leading variant may change when a reasonable prior or linkage disequilibrium (LD) input changes; a variant may enter one credible set but not another; or separate computational runs may give noticeably different posterior inclusion probabilities (PIPs). These are different kinds of instability, and they call for different checks.

The goal is not to find one summary that declares a result “stable” or “unstable.” Instead, examine which conclusions survive relevant comparisons. A broad conclusion that a locus contains a signal may be well supported even if the exact variant ranking is not. Conversely, agreement in one summary does not establish that every variant assignment is reliable. The earlier material on prior sensitivity and LD sensitivity motivates perturbing assumptions; here the emphasis is on diagnosing and interpreting the resulting changes.

Separate the Sources of Instability

A useful first step is to distinguish three sources. Computational variability occurs when repeated runs of the same method, data, and model produce different estimates because of finite simulation, random initialization, or incomplete numerical search. Model sensitivity occurs when estimates change after a deliberate change to an assumption, such as a prior or LD input. Data or model limitations arise when the available information cannot distinguish variants well, or when the fitted model does not adequately represent the data. The first calls for convergence checks, the second for sensitivity comparisons, and the third for restrained interpretation and possibly further analysis.

For a variant \(j\), the PIP is the posterior probability that it is causal under the specified model and data. It is not a probability that the variant is causal in an assumption-free sense. A ranking sorts variants by their PIPs, while a credible set collects variants according to a stated posterior-mass rule for a signal. Each summary answers a different question. Ranking diagnostics focus on order and separation; membership diagnostics focus on set overlap; convergence diagnostics ask whether computation reliably estimates the specified posterior.

Before comparing outputs, hold fixed what should be fixed. To assess computational convergence, use the same data, model, priors, and LD input. To assess sensitivity, change the intended input while keeping other choices as comparable as possible. Record the changes made, and compare like with like: PIPs for the same variants and assignments for corresponding signals. If the model permits multiple signals, signal labels such as “signal 1” may be arbitrary output labels rather than biological identities.

Compare Rankings Without Hiding Important Swaps

Rank agreement can be summarized with a rank correlation, but it should not replace inspection of the PIPs and the leading variants. For two rankings of \(n\) variants with no tied ranks, Spearman’s rank correlation is

where \(d_j\) is the difference between the two ranks assigned to variant \(j\). The statistic ranges from \(-1\) (opposite rankings) to \(1\) (identical rankings). With tied ranks, calculate the ordinary correlation of the two rank vectors instead. A high correlation can coexist with a change in the top-ranked variant, especially when most lower-ranked variants retain their order. Therefore report top-variant identity, relevant PIP changes, and rank agreement together.

Credible-Set Membership and Signal Assignments

For a given signal, compare the membership of its credible sets across analyses. If the sets are \(C_1\) and \(C_2\), their Jaccard index is the size of their intersection divided by the size of their union:

The index ranges from 0 for disjoint sets to 1 for identical sets. It is a descriptive overlap measure, not a test of whether the sets are equivalent. State the set construction and the comparison being made; a Jaccard value alone does not tell whether the shared variants are the most strongly supported ones.

When there are multiple signals, first match corresponding signals across fits. Output labels can switch even when the underlying signals are similar. Matching can use the overlap of credible sets, the overlap of high-PIP variants, or the similarity of the full variant-level assignments. If two signals are too similar to match clearly, that ambiguity is itself a diagnostic: report it rather than forcing a one-to-one correspondence. Compare memberships only after considering this matching step.

Check Computational Convergence

Repeated runs under the same model should agree within their computational uncertainty. For simulation-based methods, inspect trace plots and between-chain agreement, and calculate convergence and effective-sample-size diagnostics appropriate to the algorithm. A trace that remains in a limited region, drifts over time, or differs systematically across chains can indicate inadequate exploration. A convergence statistic near its ideal value is useful evidence, but no single diagnostic proves convergence; examine important quantities such as variant PIPs and signal summaries, not only a global statistic.

For a PIP estimated as the mean of inclusion indicators, let \(I_m\) equal 1 if variant \(j\) is included in sampled configuration \(m\), and 0 otherwise. The estimate is \(\hat p_j=M^{-1}\sum_{m=1}^M I_m\). If the draws are correlated, the variance of this average depends on their autocorrelation. An approximate effective sample size is

where \(\rho_k\) is the lag-\(k\) autocorrelation of the indicator sequence. In practice, autocorrelation sums are estimated with suitable truncation or other standard estimators. Positive autocorrelation generally reduces effective sample size; more generally, effective sample size accounts for autocorrelation and can be either smaller or larger than the raw draw count. This Monte Carlo standard error describes numerical uncertainty in estimating the PIP, not uncertainty about whether the variant is causal.

Decide Which Conclusions Remain Supportable

A useful report separates stable findings from fragile details. For example, analyses may consistently identify the same locus and the same small group of leading variants, while the first- and second-ranked variants exchange places and credible-set membership changes at the boundary. In that situation, report the stable group and make the unresolved variant-level distinction explicit. Do not turn stability into a stronger claim than the analysis supports: stable rankings are not proof of causality, and a stable credible set is conditional on the fitted model and its assumptions.

When presenting diagnostics, specify the comparisons, quantities, and computational evidence. A compact report might state which prior or LD inputs were varied; whether the top variant changed; how much the leading PIPs changed; credible-set overlap; how signals were matched; and whether repeated runs agreed within Monte Carlo error. If model misspecification or limited LD resolution is plausible, these diagnostics cannot repair it. They can show where conclusions are robust and where they are conditional or unresolved.

Practical Diagnostic Sequence

These checks complement rather than replace one another. Rank correlation is a compact comparison of order; set overlap summarizes membership; signal matching prevents arbitrary labels from masquerading as changes; and computational diagnostics assess whether a run estimates its intended result reliably. No one metric subsumes the others.