Simulation Diagnostics for Trans-Ancestry Fine-Mapping

Why Simulate Trans-Ancestry Fine-Mapping?

A fine-mapping method produces evidence about which variant or variants may be causal, but a single locus analysis cannot show how dependable that evidence is under different data conditions. Simulation provides a controlled check: the causal variant is known by construction, so the method's reported uncertainty can be compared with the truth. The aim is not to reproduce every feature of a real population. It is to vary specified features in a way that reveals when performance changes and why.

For trans-ancestry fine-mapping, four especially informative conditions are the contrast between ancestry-specific linkage disequilibrium (LD) patterns, heterogeneity in causal effects, imbalance in sample sizes, and error in the LD reference panels. Evaluate these conditions jointly with several diagnostics. In particular, strong apparent precision is not reassuring if the method often excludes the true causal variant or assigns posterior probabilities that are systematically too extreme.

Specify the Data-Generating Process First

A useful starting point is a shared-causal-variant setting: there are \(K\) candidate variants, and the same variant \(c\) is causal in each of two ancestry groups. Let \(X_a\) be the genotype matrix for group \(a\), with columns standardized within group, and let \(R_a\) denote its population LD correlation matrix. The matrices \(R_1\) and \(R_2\) need not be equal. For a quantitative phenotype, one transparent base model is

Here, \(\beta_a\) is the vector of genetic effects in group \(a\). In a one-causal-variant scenario, its entries are zero except for \(\beta_{a,c}\). The base scenario can set \(\beta_{1,c}=\beta_{2,c}\); heterogeneity scenarios deliberately change their magnitudes or directions while keeping \(c\) causal. Effects are in the phenotype units per standardized genotype unit. For example, if \(y_a\) is measured in standard-deviation units, then \(\beta_{a,c}=0.10\) means an expected 0.10 standard-deviation difference per one-standard-deviation increase in that genotype, under this model.

The genotype data should reflect the intended LD structure. One option is to generate haplotypes from specified ancestry-specific haplotype frequencies and form diploid genotypes. Another is to resample observed genotypes within ancestry groups for a simulation intended to preserve realistic LD. State how allele frequencies, missingness, imputation, and phenotype noise are generated. In the clean base case, assume a common, correctly harmonized variant set, no unmodeled population structure or phenotype confounding, and a correctly specified analysis likelihood. Then add complications intentionally rather than allowing them to vary unnoticed.

The method under evaluation must also be specified: for example, whether it uses summary statistics or individual-level data, which ancestry-specific LD matrices it receives, and which causal-effect model it assumes. A simulation comparison is interpretable only relative to these choices. If the method assumes a shared effect but the data generator gives different effects by ancestry, the scenario tests robustness to that mismatch; it does not test the method under its own ideal assumptions.

Vary Factors in a Planned Design

A simulation should vary the factors of interest systematically. For instance, specify low and high LD contrast; equal effects and several degrees of effect heterogeneity; balanced and imbalanced sample sizes; and well-matched versus error-prone reference panels. Use a factorial design when feasible, so that combinations are observed rather than changing one factor while leaving the others confounded. If computation limits require a subset, state which combinations were run and why.

For heterogeneity, define the values in advance, such as equal effects, unequal but same-direction effects, and unequal effects with opposite directions. Keep the causal variant fixed when the question is effect portability. This separates effect heterogeneity from a change in which variant is causal. If different causal variants are also of interest, treat that as a distinct data-generating scenario rather than folding it into the effect-heterogeneity label.

Represent sample-size balance explicitly. For two groups with sample sizes \(n_1\) and \(n_2\), one simple descriptor is the ratio of the larger sample to the smaller sample, \(B=\max(n_1,n_2)/\min(n_1,n_2)\). Hold the total sample size fixed when asking about allocation alone; vary the total separately when asking about overall power. Otherwise a difference attributed to imbalance might actually be due to having more observations.

Reference-panel error can be varied by using finite panels sampled from the target ancestry, panels from a mismatched ancestry, or both. Estimate LD from the simulated reference genotypes in each replicate if the intended workflow would estimate it. Do not treat an arbitrary perturbed matrix as a realistic reference panel unless it is guaranteed to be a valid correlation matrix and the perturbation has a justified interpretation. Record the panel sizes and ancestry sources, and distinguish population LD from the estimated panel LD supplied to the method.

Coverage and Its Monte Carlo Uncertainty

Let \(C_r\) be the credible set from replicate \(r\), and let \(c\) be the known causal variant. Define \(I_r=1\) if \(c\in C_r\), and \(I_r=0\) otherwise. With \(R\) independent replicates, empirical coverage is

For a method intended to produce a 95% credible set, coverage near 0.95 is a useful benchmark under the simulated data-generating process. It is not guaranteed to equal 0.95: finite simulation error, model mismatch, prior choices, and data conditions can all affect the result. Coverage substantially below the nominal level indicates that the reported sets exclude the truth too often in that scenario.

A derivation clarifies why the number of replicates matters. If each replicate is independent and has the same true coverage \(q\), then \(I_r\) is Bernoulli with variance \(q(1-q)\). The sample mean of \(R\) such indicators has variance \(q(1-q)/R\). Replacing the unknown \(q\) with the observed coverage gives the estimated Monte Carlo standard error:

This standard error quantifies simulation noise in the estimated coverage, not uncertainty about the causal variant in an individual locus. More replicates reduce this Monte Carlo error; they do not correct a misspecified generator or analysis.

Credible-Set Size and Posterior Calibration

Coverage alone does not describe resolution. A method can achieve high coverage by returning nearly every candidate variant. For each replicate, record \(|C_r|\), the number of variants in the credible set, and report its mean or median as well as coverage. Smaller sets are more informative only when coverage remains adequate. Also report whether a set could not be produced, if that can occur; do not silently omit such replicates from one metric while retaining them in another.

Posterior calibration asks a different question. In a one-causal-variant simulation, let \(q_{rj}\) be the posterior probability assigned to variant \(j\) in replicate \(r\), and let \(Z_{rj}=1\) if \(j\) is truly causal and 0 otherwise. Across repeated simulated loci, calibrated posterior probabilities should satisfy the frequency relationship

In practice, group variant-replicate pairs into bins of similar posterior probability, then compare the mean posterior probability in each bin with the observed proportion of true causal variants. For example, among entries assigned probabilities around 0.8, roughly 80% should be causal if those probabilities are calibrated under the simulation's truth distribution. Calibration depends on the simulated distribution of causal variants, effect sizes, and other assumptions, including the prior used by the method. A model can be calibrated under one generator and not another.

Compare Scenarios Without Confounding Their Effects

When possible, use paired scenario comparisons: use the same causal variant, effect settings, and random-number seeds for corresponding conditions, changing only the factor being examined. For example, compare a balanced and an imbalanced sample allocation at a fixed total sample size, with the same LD matrices and causal effect. This makes the contrast easier to interpret. Replicate pairing does not remove all Monte Carlo variation, but it can reduce noise in the difference between conditions.

Summarize results in a table or plot with coverage, average set size, calibration summaries, and replicate counts together. A lower mean set size accompanied by lower coverage is not an unqualified improvement. Similarly, good average coverage can mask poor performance in a particular combination, such as high LD contrast plus severe reference-panel mismatch. Inspect interaction patterns rather than reporting only an average across all scenarios.

Common Mistakes and Interpretation

• Calling a small set successful without checking coverage. A tiny set may simply exclude the causal variant. Report size and coverage together.
• Changing several factors at once. If LD contrast, sample size, and panel ancestry all change together, the reason for a performance difference is ambiguous. Use controlled comparisons or a clearly specified factorial design.
• Treating simulation coverage as a universal property. Coverage is conditional on the generator, analysis model, variant set, and credible-set construction. State the conditions under which it was measured.
• Confusing posterior probabilities with frequentist p-values. Posterior calibration compares assigned posterior mass with truth frequencies across repeated simulations; it is not a test of a null hypothesis.
• Ignoring simulation error. A coverage estimate from a finite number of replicates varies by chance. Report the replicate count and a Monte Carlo standard error or interval.

Simulation diagnostics are most useful when each result answers a defined question: does LD contrast help separate correlated candidates, does effect heterogeneity expose an assumption mismatch, does an imbalanced sample reduce the contribution of the smaller group, or does reference-panel error distort inference? The answers are scenario-specific. Together, coverage, set size, calibration, and carefully controlled comparisons show both how informative a method is and how much confidence its reported evidence deserves under the conditions tested.