Calibration of Component-Wise Abundance Tests

Why Calibration Needs to Be Checked

A component-wise abundance analysis tests each cell type separately, even though the proportions of all types in a sample must sum to one. A test can appear reasonable for one component while producing p-values that are too small under its null hypothesis, or while interacting poorly with the multiplicity procedure across components. Calibration asks whether the reported evidence behaves as its nominal probabilities imply.

This matters especially when the independent experimental units are donors, but each donor contributes many cells. Cells from one donor are not independent biological replicates. Increasing the number of cells may improve the measurement of that donor’s composition, but it does not create more independent donors. Overdispersion between donors and dependence among components can therefore make a cell-level analysis look more certain than the design supports.

For a simulation, let \(C_{ik}\) be the count of cell type \(k\) from donor \(i\), let \(N_i\) be that donor’s total cell yield, and let \(\pi_{ik}\) be the underlying proportion for that donor. There are \(K\) components, and their proportions satisfy \(\sum_{k=1}^{K}\pi_{ik}=1\). A null simulation for a two-group comparison sets the group-level abundance distributions equal for the component or components being tested. The simulation should still include realistic donor-to-donor variation and the same sampling structure intended for the analysis.

Designing a Useful Calibration Simulation

A useful simulation reproduces the features that matter for the planned analysis: the number of donors per group, the distribution of cell yields, the number of components, the null or non-null effects, and the dependence structure among component counts. One standard way to represent extra donor-to-donor variability is a Dirichlet-multinomial model. In one parameterization, a donor’s probability vector is drawn from a Dirichlet distribution with mean vector \(\boldsymbol{\pi}\) and concentration \(\kappa\), then the donor’s counts are drawn from a multinomial distribution conditional on that vector. Smaller \(\kappa\) gives greater between-donor variation; as \(\kappa\) grows, the model approaches multinomial sampling with a fixed probability vector.

The components remain dependent: if one component’s share rises, the shares available to other components must collectively fall. A beta-binomial model for one component versus the rest can represent extra variation in that component’s proportion, but separately fitting such models does not remove the compositional dependence across the collection of tests. The simulation should preserve that dependence when evaluating multiple-testing behavior.

Set up at least two kinds of simulation. Under the global null, no component differs between groups; this evaluates the joint behavior when every test is null. Under partial alternatives, specify effects for some components while leaving the others null; this permits assessment of false discoveries among a mixture of true and false nulls. Vary one design feature at a time when possible, so changes in performance can be attributed to donor count, cell yield, or overdispersion rather than to several simultaneous changes.

What to Measure Across Components

For a particular component whose null hypothesis is true, estimate its type I error rate across simulated studies. If null status differs by component, pool only the p-values for true nulls when estimating the marginal null rejection rate. Report the number of simulated null tests as well as the estimated rate, because rates from a small simulation are imprecise. A p-value histogram or empirical cumulative distribution can reveal broad departures from uniformity, but neither replaces a numerical estimate at thresholds relevant to the intended analysis.

When many component-wise hypotheses are tested, calibration also concerns the discoveries after multiplicity adjustment. For a simulated study, let \(R\) be the number of rejected hypotheses and \(V\) the number of rejected hypotheses that were actually null. Define the false-discovery proportion as \(V/\max(R,1)\). The false-discovery rate (FDR) is the expectation of this proportion over repeated studies. It is not the proportion of all tested hypotheses that are false, nor is it generally the ratio of total false rejections to total rejections after pooling studies.

For partial alternatives, calculate the false-discovery proportion separately in each simulated study and then average those values. This targets FDR directly. Also report power or the true-discovery proportion if it is relevant, but do not trade away calibration silently: a method that rejects more often may have greater power and a larger false-discovery rate.

Comparisons, Diagnostics, and Limits

Calibration simulation complements, rather than replaces, analytic methods. A binomial model can be a useful reference when its sampling assumptions are appropriate, but treating cells as independent when donors are the replicates can understate uncertainty. A beta-binomial model accommodates extra variation for one component versus the rest; joint models such as the Dirichlet-multinomial or logistic-normal approach represent dependence among components more directly. A component-wise method may still be convenient, but its marginal p-value calibration and its multiple-testing behavior should be evaluated separately.

Multiplicity procedures also rely on conditions. For example, the Benjamini–Hochberg procedure has standard FDR control under independent test statistics and under certain forms of positive dependence. Compositional constraints can induce dependence, and the direction and structure need not satisfy those conditions automatically. Simulation is a practical way to examine the procedure under a specified dependence model; it does not prove control outside that model. Permutation-based assessment is another option when the design permits valid label exchangeability, but it likewise depends on the randomization or exchangeability assumptions being appropriate.

Useful warning signs include a null rejection rate materially above \(\alpha\), an excess of very small p-values in null simulations, or an FDR estimate above the target in scenarios where control is intended. Check that the simulated group labels, donor structure, covariates, and multiplicity adjustment match the analysis plan. Also examine whether rates change sharply as cell yield rises while donor count stays fixed, or as overdispersion increases. A result that looks calibrated only at one convenient parameter setting is not a broad calibration assessment.

Simulation results have limits. They inherit the assumptions chosen for the data-generating process, and a misspecified simulation can provide reassurance that does not transfer to the study. Monte Carlo error can obscure modest departures from nominal rates. Finally, good calibration for observed relative proportions does not establish calibration for absolute cell numbers or remove biases in cell capture and annotation. Calibration answers a specific question: how the planned testing procedure behaves when data follow the simulated design and model.

Common Mistake

Wrong reasoning: “There are thousands of cells, so the test has thousands of independent observations; if its p-values are small, the component is well calibrated.” This confuses cell yield with independent biological replication and ignores donor-level variability and compositional dependence.

Correction: Simulate the donor as the experimental unit, generate the within-donor cell counts under a stated compositional model, and evaluate both marginal null rejection rates and the FDR after the planned adjustment. More cells per donor and more donors are different design changes, so vary and report them separately.