Why Examine More Than One QC Threshold?
A quality-control decision can depend on where a cutoff is placed. A stricter cell-calling cutoff may exclude barcodes with weaker evidence of cell-associated RNA, while a stricter mitochondrial-fraction cutoff may exclude more cells with high mitochondrial fractions. A single selected pair of cutoffs can hide how much the retained data depend on those choices. Sensitivity analysis makes that dependence visible by repeating the same filtering procedure over a plausible range of thresholds and comparing the resulting cell sets and summaries.
The purpose is not to find a threshold that produces a preferred cell count or cell-group composition. Rather, the analysis asks whether important summaries are reasonably stable across defensible choices, and which barcodes or groups account for any changes. The conclusions are descriptive: a sensitivity analysis alone does not establish that retained cells are biologically valid or that excluded cells are invalid.
The cell-calling statistic need not be a raw UMI count; it could be another fixed score or statistic used in the cell-calling procedure. Its direction must be made explicit. If the statistic is instead oriented so that smaller values indicate stronger evidence, reverse the inequality before interpreting changes. Likewise, mitochondrial fractions should use the same gene annotation, denominator, and scale throughout the comparison.
A useful analysis evaluates a grid of threshold pairs, not just a collection of unrelated alternatives. Choose ranges that are plausible given the barcode-rank curve, cell-calling diagnostics, and the observed QC metrics. Vary one threshold at a time to understand its effect, and include joint changes to see how the cutoffs operate together. Keep the input data, metric definitions, and all non-varied filters fixed. If multiple samples are present, inspect results by sample as well as in aggregate so that a large sample does not conceal a problematic smaller one.
What to Compare Across Thresholds
At minimum, report the number of retained cells for each threshold pair and how that number changes from a clearly stated reference pair. Also compare distributions of relevant per-cell metrics, such as total counts, detected genes, and mitochondrial fraction. Means can be sensitive to a small number of extreme observations, so medians and quantiles are often useful alongside them. Show the denominator for every proportion and distinguish changes in a group’s retention rate from changes in its share of the retained data.
If cell-group labels are already available from an independent source or a fixed annotation, compare group representation across the grid. Avoid recomputing labels separately at each threshold if the goal is to isolate the effect of filtering: changing both the retained cells and the labeling procedure makes the source of a difference harder to identify. If labels are incomplete, state which cells have labels and how they are handled.
Worked Example: Counting Cells Across a Threshold Grid
Suppose eight hypothetical barcodes have the following cell-calling scores and mitochondrial fractions. The cell-calling score is higher for barcodes with stronger cell-associated evidence.
| Barcode | Score \(s_i\) | Mitochondrial fraction \(q_i\) |
|---|---|---|
| A | 10 | 2% |
| B | 8 | 4% |
| C | 7 | 8% |
| D | 6 | 12% |
| E | 5 | 18% |
| F | 4 | 6% |
| G | 3 | 25% |
| H | 2 | 3% |
Compare cell-calling cutoffs \(c=4\) and \(c=6\), and maximum mitochondrial fractions \(m=10\%\) and \(m=20\%\). At \(c=4\), barcodes A through F pass the cell-calling condition. Of those six, A, B, C, and F have \(q_i\leq10\%\), giving \(N(4,10\%)=4\). All six have \(q_i\leq20\%\), so \(N(4,20\%)=6\).
At \(c=6\), only A, B, C, and D pass the cell-calling condition. Three of those four have \(q_i\leq10\%\), so \(N(6,10\%)=3\); all four have \(q_i\leq20\%\), so \(N(6,20\%)=4\). The complete count grid is therefore:
| Cell-calling cutoff \(c\) | Maximum mitochondrial fraction \(m\) | Retained-cell count |
|---|---|---|
| 4 | 10% | 4 |
| 4 | 20% | 6 |
| 6 | 10% | 3 |
| 6 | 20% | 4 |
Relative to the reference pair \((c,m)=(4,10\%)\), the count increases by \((6-4)/4=0.50\), or 50%, at \((4,20\%)\). At \((6,10\%)\), it decreases by \((3-4)/4=-0.25\), or 25%. These percentages describe changes in this hypothetical barcode set, not sampling uncertainty or a probability that a cutoff is correct.
Why Counts Change: A Set-Based Justification
With all other conditions fixed, increasing the cell-calling cutoff cannot add a barcode: a barcode that meets \(s_i\geq c_2\), where \(c_2>c_1\), also meets \(s_i\geq c_1\). Thus, for a fixed mitochondrial cutoff, \(R(c_2,m)\) is a subset of \(R(c_1,m)\), and \(N(c_2,m)\leq N(c_1,m)\). The cells lost are precisely those that pass the mitochondrial condition and have scores between the two cutoffs:
Similarly, increasing the allowed mitochondrial fraction from \(m_1\) to \(m_2\), with \(m_2>m_1\), cannot remove cells when \(c\) is fixed. The added cells are those that pass the cell-calling condition and have \(m_1<q_i\leq m_2\):
These relationships are useful checks on the implementation. If a stricter cell-calling cutoff increases the count, or a more permissive mitochondrial cutoff decreases it, then some other setting, input set, or calculation may have changed. The expected monotonicity applies to these fixed, nested threshold rules; it need not apply if the procedure itself changes across settings.
Worked Example: Comparing Metric Distributions
Consider six hypothetical barcodes that all pass a fixed cell-calling cutoff. Their detected-gene counts and mitochondrial fractions are:
| Barcode | Detected genes | Mitochondrial fraction |
|---|---|---|
| A | 700 | 2% |
| B | 900 | 4% |
| C | 1,100 | 8% |
| D | 1,300 | 12% |
| E | 1,500 | 18% |
| F | 6,000 | 6% |
With a mitochondrial cutoff of 10%, A, B, C, and F are retained. Their mean detected-gene count is
In sorted order, the counts are 700, 900, 1,100, and 6,000, so the median is \((900+1{,}100)/2=1{,}000\) genes. With a 20% cutoff, all six barcodes are retained. Their mean is
The sorted counts are now 700, 900, 1,100, 1,300, 1,500, and 6,000, giving a median of \((1{,}100+1{,}300)/2=1{,}200\) genes. Here the median rises while the mean falls. That is possible because the newly included barcodes change the middle of the distribution, while barcode F’s high count has substantial influence on the mean. Neither shift by itself shows that the more permissive cutoff is better; it describes how the retained set changes.
Worked Example: Group Retention and Representation
Suppose a hypothetical starting set has 50 cells in group A, 30 in group B, and 20 in group C. At a permissive threshold pair, 45, 27, and 18 cells from those groups are retained, respectively. At a stricter pair, the corresponding retained counts are 40, 15, and 16. Assume the group labels are fixed and available for every cell.
The total retained counts are \(45+27+18=90\) at the permissive pair and \(40+15+16=71\) at the stricter pair. Group B’s retention rate changes from \(27/30=90\%\) to \(15/30=50\%\). Its composition among retained cells changes from \(27/90=30\%\) to \(15/71\approx21.13\%\). These calculations show both that group B has a lower retention rate under the stricter thresholds and that its share of the retained set is smaller.
For comparison, group A’s retention rate is \(45/50=90\%\) and then \(40/50=80\%\); its composition changes from \(45/90=50\%\) to \(40/71\approx56.34\%\). Group A’s share rises even though its retention rate falls. This is not a contradiction: composition depends on what happens to every group in the denominator. It would be misleading to report only the increase in group A’s share as evidence that more group A cells were retained.
Practical Workflow, Diagnostics, and Alternatives
Specify the input barcodes, the cell-calling statistic and its direction, the mitochondrial-fraction calculation, and all filters that will remain unchanged.
Use observed distributions and cell-calling diagnostics to choose plausible values. Include a reference pair and vary each threshold separately before examining joint changes.
For each pair, record the retained count, count change from the reference, relevant metric quantiles or distributions, and group-specific retention and composition when labels are available.
Verify expected count monotonicity, inspect which barcodes enter or leave, and examine whether changes are concentrated in particular samples or known groups.
State the threshold range, reference pair, denominators, and summaries that are stable or change materially. Do not choose a cutoff solely because it gives a desired composition or count.
A single-cutoff analysis is simpler, but it cannot reveal how dependent the results are on that choice. An adaptive threshold can account for systematic differences in QC metrics across cell types, while a global threshold applies the same cutoff throughout; comparing plausible alternatives is informative, but the analyses answer different questions and require their own justification. Sensitivity analysis is also distinct from bootstrap or other sampling-uncertainty procedures: it describes changes induced by changing thresholds, not uncertainty from repeatedly sampling biological units.
Several diagnostics help identify problems. Retained counts should follow the monotonicity rules above when only nested cutoffs change. Unexpected reversals can indicate a changed input set, an inconsistent percentage-versus-proportion scale, an incorrectly oriented score, or a coding error. Large shifts in a metric distribution may reflect selection induced by filtering rather than a change in the underlying cells. If results differ sharply by sample, report that heterogeneity instead of relying only on a pooled summary.
Common Pitfalls and Limitations
- Treating a grid as a search for the best-looking answer. Choosing thresholds because they preserve a preferred group or produce a target number of cells can make the QC decision outcome-driven. Define a plausible range and report the pattern across it.
- Confusing group retention with group composition. Always state the denominator: cells in that group at the start for a retention rate, or all retained cells for composition.
- Changing several parts of the pipeline at once. If thresholds, metric definitions, and annotations all change, the source of a difference cannot be isolated. Hold non-varied choices fixed where possible.
- Interpreting stable counts as proof of valid cells. Many threshold pairs can retain nearly the same number of cells while swapping which barcodes are included. Inspect cell identities and relevant metrics, not counts alone.
- Overstating what the analysis establishes. Sensitivity analysis quantifies dependence on specified choices. It cannot, without additional evidence, prove that a threshold is biologically correct or that observed group differences are caused by filtering.
The approach is limited by the plausibility of the threshold range, the quality of the underlying metrics, and the availability of trustworthy group labels. If no independent labels exist, describe metric distributions and barcode-level changes without implying known group representation. A threshold grid is a transparent diagnostic, not a substitute for the substantive rationale behind a QC rule.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- With the same mitochondrial cutoff, what should happen to the retained-cell count when the cell-calling cutoff increases? State the set relationship that justifies your answer.
- A group has 24 cells in the starting set, of which 18 are retained. There are 60 retained cells in total. Calculate the group’s retention rate and its composition among retained cells.
- At a fixed cell-calling cutoff, 35 cells pass a mitochondrial cutoff of 8% and 43 pass a cutoff of 15%. How many cells were added by relaxing the cutoff? What range of mitochondrial fractions do those added cells satisfy?
- Why can a group’s share of retained cells rise even when fewer cells from that group are retained?
Solutions: (1) The count cannot increase. If \(c_2>c_1\), then \(R(c_2,m)\subseteq R(c_1,m)\), so \(N(c_2,m)\leq N(c_1,m)\). (2) The retention rate is \(18/24=75\%\); the composition is \(18/60=30\%\). The denominators differ because the quantities answer different questions. (3) The cutoff change adds \(43-35=8\) cells. Those cells pass the fixed cell-calling cutoff and have mitochondrial fractions greater than 8% and at most 15%. (4) Composition divides a group’s retained count by the total retained count. If other groups lose a larger share of their cells, the group’s fraction of the remaining cells can increase even while its own retained count decreases.