Simulation-Based Calibration of Donor Inference

Why Calibrate Donor Inference by Simulation?

A donor-aware analysis is intended to respect the fact that cells from the same donor are related. But whether a method does so adequately depends on the design, the assumptions, and the quantity being estimated. With few donors, unequal cell counts, or substantial between-donor variation, a method that appears reasonable can still produce biased estimates, intervals that miss too often, or tests that reject a true null hypothesis too frequently.

Simulation-based calibration evaluates these properties under repeated data sets whose generating process is specified in advance. Because the simulated truth is known, performance can be compared directly with that truth. This is not proof that a method will work for every biological study: it shows how the method behaves under the simulated conditions, and therefore makes the assumptions and limitations of the evaluation explicit.

Here, “calibration” means agreement between a procedure’s repeated-sampling behavior and its advertised behavior. For example, a nominal 95% confidence interval is calibrated in a scenario if it contains the true parameter in about 95% of independently simulated studies. A false-positive rate is calibrated at level \(\alpha\) if a test rejects the null in about an \(\alpha\) fraction of studies generated under that null.

A Nested Data-Generating Model

A useful starting point is a simple continuous-outcome model with cells nested within donors. Let \(j=1,\ldots,D\) index donors and \(i=1,\ldots,m_j\) index cells from donor \(j\). Let \(T_j\) indicate the donor’s condition, with \(T_j=0\) for control and \(T_j=1\) for treatment. One possible data-generating model is

Here, \(Y_{ij}\) is the outcome for a cell, \(\beta_0\) is the control mean, and \(\beta_1\) is the treatment-minus-control difference in the cell-level mean. The donor random effects \(b_j\) are independent across donors, and the cell errors \(\epsilon_{ij}\) are independent across cells and independent of the donor effects. In this model, cells from one donor share \(b_j\), while cells from different donors do not share a donor effect. The total marginal variance of a cell outcome is \(\tau^2+\sigma^2\), giving intraclass correlation

To use this simulation to evaluate an analysis, the simulated design must represent the feature being studied. Specify the number of donors per condition, how condition is assigned, and the cell counts \(m_j\). Hold these choices fixed or deliberately vary them across scenarios. The donor effects and cell errors should be generated according to the stated model; the analysis method should then be applied to each simulated data set as it would be to observed data.

This model is a deliberate simplification. It assumes a continuous outcome, normal donor effects and cell errors, and a common treatment effect across donors. A simulation intended to assess an analysis for another outcome type or design must generate data with assumptions appropriate to that setting. In every case, the analysis is fitted separately to each simulated study; it must not be given the simulated truth as if it were observed.

Performance Measures Across Repeated Studies

Let \(B\) denote the number of simulated studies in a scenario. In study \(r\), let \(\hat{\beta}_{1r}\) be the method’s estimate, \(L_r\) and \(U_r\) the endpoints of its nominal 95% interval, and \(R_r\) equal 1 if its test rejects \(H_0:\beta_1=0\) and 0 otherwise. The same simulation can assess several distinct properties; none should be treated as a substitute for the others.

The indicator \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. Bias is measured in the units of the effect estimate. Coverage is a proportion. False-positive rate is also a proportion and should be compared with the prespecified significance level \(\alpha\). A method may have small bias but poor coverage, for example if its standard errors are too small. It is therefore important to report these measures separately.

Simulation results themselves are noisy because they use a finite number of replications. If a binary outcome occurs in \(K\) of \(B\) independent replications, its estimated proportion is \(\hat{q}=K/B\), with approximate Monte Carlo standard error

For estimated bias, an estimate of Monte Carlo standard error is \(s_{\hat{\beta}_1}/\sqrt{B}\), where \(s_{\hat{\beta}_1}\) is the sample standard deviation of the \(B\) estimates. This uncertainty describes the precision of the simulation summary, not the uncertainty in an estimate from an actual biological study.

A Calibration Workflow and Its Justification

The core logic is repeated sampling under known conditions. A complete evaluation specifies the truth and design first, generates data from that specification, applies the analysis without changing its rules after seeing results, and summarizes repeated performance. The flow below is useful because it separates a method’s behavior from random variation in any single simulated data set.

The justification for the coverage calculation follows directly from its definition. In a repeated-sampling scenario, each interval either contains the fixed true value \(\beta_1\) or does not. The indicator \(I(L_r\leq\beta_1\leq U_r)\) is therefore a binary observation of coverage in replication \(r\). Averaging these indicators estimates the probability of coverage under that scenario; with independent replications, its expected value is the scenario’s true coverage probability.

When all replications use the same procedure and scenario, each probability on the right is the same, so the expected empirical coverage equals that probability. If the procedure produces 95% coverage under the scenario, the empirical coverage fluctuates around 0.95 rather than equaling it exactly in every run.

Varying Donors, Cells, and Donor Correlation

A single scenario answers a narrow question. A useful calibration study varies the features that plausibly alter performance. For example, compare low and higher donor counts, several cell counts per donor, and several ICC values. Change one factor at a time for clear comparisons, or use a planned grid to assess combinations. Keep the target effect and analysis rules fixed when the purpose is to isolate design effects.

The number of donors and the number of cells answer different information questions. More cells can reduce the cell-sampling component of uncertainty in donor summaries, but do not create more independent donors. As \(\rho\) rises, cells from one donor share more of their variation, making the effective information gain from additional cells smaller. A simulation that increases only cells per donor may therefore show limited improvement in coverage or test calibration when donor count is small.

Consider evaluating a donor-level summary analysis and a random-intercept analysis under the same generated studies. Each method should be assessed using the same outcomes and target parameter, and the simulation should record the precise interval and test procedure used. A cell-level analysis that treats all cells as independent can be included as a contrast, but it does not become donor-aware simply because the underlying data were simulated with donor effects.

Results can differ across methods and scenarios. A donor-level summary may be transparent and robust to some cell-level distribution details, but may discard information or behave differently when cell counts vary. A mixed model can represent within-donor dependence directly, but relies on its specified random-effects and residual structure and can encounter estimation problems with few donors. Neither method should be declared universally superior from one favorable scenario.

Diagnostics, Failure Modes, and Limits

Calibration results are only as useful as the simulation that produced them. Check the generated data before interpreting method performance: verify the intended donor counts, condition allocation, cell-count distribution, mean, variance components, and ICC. In a small pilot run, donor summaries and empirical within-donor correlations can reveal coding errors such as accidentally drawing a new donor effect for every cell.

Record convergence and other warnings in every replication. If a method fails to fit in a nontrivial fraction of studies, reporting performance only among successful fits can be misleading: the retained studies may be systematically easier. Report the failure frequency and describe the rule used for failed fits. Also inspect whether extreme estimates, boundary variance estimates, or unusually wide intervals are concentrated in particular scenarios.

Monte Carlo uncertainty is another diagnostic. If two estimated coverage rates differ by less than their simulation noise, the evidence for a performance difference is weak. Increasing \(B\) improves precision of the simulation summaries, with Monte Carlo standard error decreasing approximately in proportion to \(1/\sqrt{B}\). It does not add biological donors to the simulated study, and it does not make a misspecified data-generating process more realistic.

A common mistake is to simulate many cells from a small number of donors, observe a narrow interval, and conclude that the method has abundant independent information. The number of simulated replications \(B\) is also sometimes confused with the number of biological replicates. Neither large cell counts nor a large \(B\) changes the donor count within each study. The correction is to define the independent sampling units in the data-generating process and evaluate the method at the donor counts relevant to the intended application.