Sensitivity Analysis for Design Assumptions in Single-Cell Data

Why Sensitivity Analysis Matters

A design calculation is conditional on its inputs. Power may look adequate for a chosen effect size and variance estimate, yet fall sharply if the true effect is smaller, donor-to-donor variation is larger, fewer donors are retained, or a cell population yields fewer usable cells than expected. Sensitivity analysis makes those dependencies visible by recalculating power and precision under a range of plausible assumptions.

The goal is not to find one forecast that guarantees success. It is to identify which assumptions drive the design, which are uncertain, and whether the planned study remains useful when those assumptions change. This is especially important in single-cell studies: many cells from one donor can improve measurement of that donor, but they do not substitute for independent donors when the target is a population-level condition difference.

Definition: A sensitivity analysis for a study design recalculates operating characteristics—such as power or the expected width of a confidence interval—under alternative, explicitly stated values of uncertain design inputs. A design is fragile when modest, plausible changes to an input lead to a consequential loss of power, precision, or feasibility.

Set the Comparison and Assumptions

For a basic planning calculation, consider two independent groups of donors and one prespecified cell population and outcome, such as a donor-level expression summary in log2 units. Let \(n_0\) and \(n_1\) be the numbers of retained donors in the two groups, and let \(\delta\) be the condition difference the study aims to detect. Suppose the donor-level outcome is approximately normal, has a common standard deviation \(\tau\) in both groups, and donors are independent within and between groups. For a two-sided test at significance level \(\alpha\), a normal-approximation standard error is

SE(\bar{x}_1-\bar{x}_0)=\tau\sqrt{\frac{1}{n_1}+\frac{1}{n_0}}.

This calculation treats the donor as the independent unit. The donor-level outcome might be a pseudobulk summary or another prespecified donor-by-cell-population measurement; the formula is not a claim that individual cells are independent replicates. It also assumes that the variance estimate represents the target population and that the comparison and test are correctly specified. If the design is paired, has covariates, unequal variances, multiple planned contrasts, or complex allocation, this simple formula may not represent its uncertainty; use a calculation matched to that design, often by simulation.

Under the normal approximation, the noncentrality parameter for the planned effect is \(\lambda=\delta/SE\). The approximate two-sided power is

\operatorname{Power}\approx \Phi(-z_{1-\alpha/2}-\lambda)+1-\Phi(z_{1-\alpha/2}-\lambda),

where \(\Phi\) is the standard normal cumulative distribution function and \(z_q\) is its \(q\)-quantile. The approximation is most useful for transparent scenario comparisons; with small donor counts, a \(t\)-based calculation or design-specific simulation may be preferable. Use the significance level appropriate to the planned analysis, including any multiplicity adjustment. Precision can be summarized by the approximate confidence-interval half-width \(z_{1-\alpha/2}SE\), with the full width twice that amount.

Why Donor Counts and Cell Yields Behave Differently

A useful simplified variance model separates between-donor variation from cell-level variation. Suppose a donor’s cell-level measurements have a donor-specific component with variance \(\tau^2\), plus independent within-donor residual variation with variance \(\sigma^2\). If \(m\) usable cells are measured for each donor, the donor mean has variance

\operatorname{Var}(\bar{x}_{\text{donor}})=\tau^2+\frac{\sigma^2}{m}.

This expression illustrates diminishing returns from more cells: the within-donor component shrinks as \(m\) grows, but the between-donor component remains. For \(n_0\) and \(n_1\) independent donors, the contrast variance under this simplified model is

\operatorname{Var}(\bar{x}_1-\bar{x}_0) = \left(\tau^2+\frac{\sigma^2}{m}\right) \left(\frac{1}{n_1}+\frac{1}{n_0}\right).

The decomposition is a planning approximation, not a universal single-cell likelihood model. Correlated cells, unequal cell counts, cell-level quality differences, or a more complex estimand can require simulation. Its central design lesson is still useful: once \(\sigma^2/m\) is small relative to \(\tau^2\), increasing cell yield only slightly improves precision, while adding independent donors can directly reduce the donor-level contrast standard error.

Justification: If donor \(j\)'s cell measurements are \(x_{jk}=\mu+b_j+e_{jk}\), with independent \(b_j\) of variance \(\tau^2\) and independent \(e_{jk}\) of variance \(\sigma^2\), then the donor mean is \(\bar{x}_j=\mu+b_j+m^{-1}\sum_{k=1}^m e_{jk}\). Independence gives \(\operatorname{Var}(\bar{x}_j)=\tau^2+m^{-2}(m\sigma^2)=\tau^2+\sigma^2/m\). The difference between independent group means has variance equal to the sum of their variances, yielding the contrast formula above.

Worked Scenarios

Worked Example: Effect Size and Donor Variability

Suppose a hypothetical balanced study retains 12 donors per condition. For one prespecified contrast, assume a donor-level standard deviation of \(0.60\) log2 units and use a two-sided \(\alpha=0.05\) normal approximation. Compare target differences of \(0.50\) and \(0.80\) log2 units, then assess what happens if the standard deviation is \(0.80\) instead.

With \(n_0=n_1=12\), the standard error under \(\tau=0.60\) is

SE=0.60\sqrt{\frac{1}{12}+\frac{1}{12}} =0.60\sqrt{\frac{1}{6}} \approx 0.2450\text{ log2 units}.

For \(\delta=0.50\), \(\lambda=0.50/0.2450\approx2.041\). With \(z_{0.975}=1.960\), approximate power is \(\Phi(2.041-1.960)+\Phi(-2.041-1.960)\approx\Phi(0.081)+\Phi(-4.001)\approx0.532\). Thus, under these assumptions, the probability of rejecting the null hypothesis when the true difference is \(0.50\) log2 units is about 53%.

For \(\delta=0.80\), \(\lambda=0.80/0.2450\approx3.266\), and power is approximately \(\Phi(1.306)+\Phi(-5.226)\approx0.904\). The same design has substantially higher power for the larger target effect; it does not have one power value independent of the effect being targeted.

Now retain the \(0.50\)-unit target but increase donor variability to \(\tau=0.80\). The standard error becomes \(0.80\sqrt{1/6}\approx0.3266\) log2 units, so \(\lambda\approx1.531\) and power is approximately \(\Phi(-0.429)+\Phi(-3.491)\approx0.334\). The design is sensitive to the assumed donor variability. At the original \(\tau=0.60\), the approximate 95% interval half-width is \(1.960(0.2450)\approx0.4801\) log2 units, or full width \(2(0.4801)\approx0.9602\) log2 units.

Worked Example: Additional Cells Have Diminishing Returns

Consider a hypothetical design with 10 independent donors per condition. Assume the simplified variance decomposition above, with between-donor standard deviation \(\tau=0.35\) log2 units and within-donor standard deviation \(\sigma=0.80\) log2 units. Compare 100 with 400 usable cells per donor for a target difference of \(0.50\) log2 units.

For \(m=100\), donor-mean variance is \(0.35^2+0.80^2/100=0.1225+0.0064=0.1289\), giving a donor-mean standard deviation of \(\sqrt{0.1289}\approx0.3590\) log2 units. The contrast standard error is

SE=\sqrt{0.1289\left(\frac{1}{10}+\frac{1}{10}\right)} =\sqrt{0.02578} \approx0.1606\text{ log2 units}.

Then \(\lambda=0.50/0.1606\approx3.114\), and approximate two-sided power at \(\alpha=0.05\) is \(\Phi(3.114-1.960)+\Phi(-3.114-1.960)\approx0.876\). The approximate 95% interval half-width is \(1.960(0.1606)\approx0.3147\) log2 units.

For \(m=400\), donor-mean variance is \(0.1225+0.64/400=0.1241\). Therefore \(SE=\sqrt{0.1241(0.2)}=\sqrt{0.02482}\approx0.1575\) log2 units. The noncentrality parameter is about \(3.174\), giving power \(\Phi(1.214)+\Phi(-5.134)\approx0.888\); the approximate interval half-width is \(1.960(0.1575)\approx0.3088\) log2 units. Quadrupling cell yield changes these planning summaries only modestly because the between-donor variance dominates. The calculation assumes the stated variance components and independent residuals; actual cell dependence should be represented in a design-specific model or simulation.

Worked Example: Attrition Can Undermine Planned Precision

A hypothetical study wants 12 retained donors per condition and anticipates 20% attrition. It therefore plans to recruit 15 per condition, since \(15(1-0.20)=12\) expected retained donors. Suppose the target difference is \(0.60\) log2 units, the donor-level standard deviation is \(0.70\) log2 units, and \(\alpha=0.05\).

If 12 donors are retained in each group, then

SE=0.70\sqrt{\frac{1}{12}+\frac{1}{12}} =0.70\sqrt{\frac{1}{6}} \approx0.2858\text{ log2 units}.

Thus \(\lambda=0.60/0.2858\approx2.099\), and approximate power is \(\Phi(2.099-1.960)+\Phi(-2.099-1.960)\approx0.555\). The approximate 95% interval half-width is \(1.960(0.2858)\approx0.5601\) log2 units.

If actual retention is 10 donors per group instead, \(SE=0.70\sqrt{0.2}\approx0.3130\) log2 units. Then \(\lambda\approx1.916\), power is approximately \(\Phi(-0.044)+\Phi(-3.876)\approx0.483\), and the interval half-width is \(1.960(0.3130)\approx0.6136\) log2 units. These are conditional calculations for specified retained counts, not a complete probability model for random attrition. If retention varies randomly, a fuller analysis should consider the distribution of retained counts, including the possibility that a group falls below the intended size. The 20% inflation is a planning expectation, not a guarantee of retaining 12 donors in each group.

How to Build and Review Scenarios

A useful analysis varies assumptions one at a time first, then combines plausible adverse values to assess whether several uncertainties can compound. Keep the estimand, contrast, analysis method, and significance level fixed unless one of those is itself under review. Report the values used in each scenario; a statement that a design is “robust” is not informative without the assumptions being varied.

Define the target.
Specify the condition contrast, cell population, outcome scale, and a scientifically meaningful range of effect sizes.

Set plausible variance and feasibility inputs.
Use pilot evidence or defensible ranges for donor variability, within-donor variation, cell yield, and attrition. State the uncertainty in those inputs rather than treating a noisy pilot estimate as known.

Recalculate operating characteristics.
For each scenario, report retained donor counts, standard error, power at the planned \(\alpha\), and an interval half-width or other precision measure. Apply the planned multiplicity burden where relevant.

Identify the drivers and safeguards.
Find which changes materially reduce power or precision. Consider whether recruitment targets, cell-yield criteria, replacement rules, or allocation should be revised, without assuming that extra cells solve a lack of donor replication.

For unequal groups, replace the balanced-group term by \(1/n_1+1/n_0\); do not use a simple average group size in its place. For planned multiple contrasts, the relevant threshold may be more stringent than \(0.05\), so sensitivity calculations should use the analysis plan’s actual threshold or a simulation reproducing its testing procedure. The earlier material on simulation-based power is useful when analytic approximations cannot represent the design, and the earlier material on attrition supplies planning concepts for losses and replacement rules.

Diagnostics, Alternatives, and Limitations

A scenario table is often the clearest diagnostic. Look for abrupt power loss across plausible values, interval widths that exceed a scientifically acceptable range, or a design whose apparent adequacy depends on optimistic cell yields or unusually low donor variability. Check that assumed retained donor counts are feasible, that the cell population is expected to be observed in enough donors, and that the proposed analysis uses the same unit of replication as the calculation.

Approach	Useful when	Main caution
Analytic sensitivity calculations	The design is simple enough for transparent formulas and a small number of scenarios.	Approximate formulas may not capture small-sample testing, complex allocation, or random attrition.
Design-specific simulation	The analysis has complex variance structure, unequal yields, or a nonstandard testing procedure.	Results depend on the simulated data-generating model; uncertainty in that model still needs scenarios.
Single “best guess” calculation	A quick reference point is needed.	It hides sensitivity and can give false reassurance when inputs are uncertain.

Sensitivity analysis cannot establish that an assumed effect size is biologically important, rescue a confounded or rank-deficient design, or prove that the eventual data will follow the planning model. It also does not turn a power calculation into the probability that the study will produce a significant result under every possible truth. Power is conditional on a specified effect, variance, sample size, test, and significance level. Precision summaries are likewise conditional on their variance assumptions.

Common Mistake: Treating Cells as Independent Donors

Wrong reasoning: “We expect thousands of cells in each condition, so the study has thousands of independent replicates and must have high power.” This incorrectly treats cells from the same donor as independent evidence for a donor-level condition contrast. Shared donor biology creates within-donor dependence, and the condition is often assigned or observed at the donor level.

Correction: Base replication on independent donors for a donor-population comparison. Model cell yield as a contributor to how precisely each donor is measured, while retaining between-donor variation in the uncertainty. Then recalculate across realistic donor counts, donor variability, cell yields, and attrition. A large cell yield may help, but the variance decomposition shows why its benefit can level off.

Key takeaway: A credible design is not one that meets a power target only under a single favorable assumption. It is one whose power and precision have been checked across plausible effects, donor variability, retained donor counts, and cell yields—and whose weaknesses are explicit.

Check Your Understanding

Work each question, then compare with the solutions below.

For a balanced two-group design, what happens to the standard error if donor counts in both groups are multiplied by four while donor variability stays fixed?
In the variance model \(\tau^2+\sigma^2/m\), what happens to donor-mean variance as cell yield \(m\) becomes very large?
A design’s power is calculated for a difference of \(0.8\) log2 units. May that power be interpreted as the probability that the true difference is at least \(0.8\)? Why or why not?
A study expects 25% attrition and wants 18 retained donors per group. How many should it recruit per group using simple expected-count inflation, rounding up?
Why is it important to recalculate power using the actual significance threshold when several contrasts are planned?

Solutions. (1) The standard error is proportional to \(\sqrt{1/n_1+1/n_0}\). Multiplying both counts by four divides the quantity inside the square root by four, so the standard error is halved. (2) The within-donor term \(\sigma^2/m\) approaches zero, but the variance approaches \(\tau^2\); more cells do not remove between-donor variation. (3) No. Power is the probability of rejecting \(H_0\) under the specified assumed effect and model, not a probability statement about the unknown effect. (4) \(18/(1-0.25)=24\), so recruit 24 per group under this simple inflation rule. Actual retained counts can still vary. (5) A stricter threshold generally requires stronger evidence to reject \(H_0\), reducing power at the same sample size and effect. Using the unadjusted threshold would overstate power relative to the planned testing procedure.