Diagnosing Disagreement Between Cell-Calling Methods

Why Compare Cell-Calling Methods?

Cell calling is a decision made from barcode-level RNA evidence, not a direct observation of whether every barcode came from a cell. A minimum-count rule, a barcode-rank curve, and an EmptyDrops test use different features of the data. They can therefore disagree even when each has been applied as specified. The useful question is not simply which method “wins,” but which barcodes disagree, where they lie in the count distribution, and whether the disagreement is consistent with each method’s assumptions.

The earlier material on Empty droplets and ambient barcodes, Barcode-rank curves and the cell-calling knee, EmptyDrops tests for cell-associated RNA, and False discovery control in barcode calling provides the individual method definitions. Here, the goal is to compare their calls and interpret the disagreement. The comparison does not supply a gold-standard label, and agreement alone does not establish that a barcode is a cell.

What Is Being Compared?

Let the analysis consider a fixed set of barcodes $U$. For each barcode, let $N_i$ be its total observed UMI count across genes. A method produces a call set: the subset of $U$ it labels as cell-associated. For example, a count-threshold rule calls barcodes with $N_i$ at or above a chosen cutoff. A rank-based procedure identifies a region of the barcode-rank curve interpreted as cell-associated. EmptyDrops tests whether a barcode’s gene-count composition is inconsistent with an ambient-RNA null model, with a multiple-testing rule used to determine calls.

Definition: For two call sets $A$ and $B$ from the same barcode universe, their intersection $A\cap B$ contains barcodes called by both, and their union $A\cup B$ contains barcodes called by at least one. Their Jaccard similarity is $$ J(A,B)=\frac{|A\cap B|}{|A\cup B|} $$ when $|A\cup B|>0$. It ranges from 0 (no shared calls) to 1 (identical, nonempty call sets). If both sets are empty, the formula is $0/0$, so Jaccard similarity is undefined unless an analysis explicitly adopts and reports a convention.

A comparison requires the same barcode universe and consistent preprocessing. If one method was run only on barcodes above a minimum count and another was run on all observed barcodes, their apparent overlap can reflect different eligibility sets rather than different judgments about the same candidates. Record any minimum-count prefilter, tested barcode set, count definition, threshold, rank cutoff, EmptyDrops significance criterion, and multiple-testing procedure.

Worked Example: Comparing Three Call Sets

Suppose a hypothetical experiment has eight barcodes. The count-threshold method calls a barcode if its total UMI count is at least 100. A rank-curve review places the knee after barcode C, so the rank-based set is A–C. EmptyDrops calls barcodes with adjusted $q$-value at most 0.05. The hypothetical results are:

Barcode	Total UMIs	Threshold call	Rank call	EmptyDrops $q$-value	EmptyDrops call
A	1,200	Yes	Yes	0.001	Yes
B	700	Yes	Yes	0.004	Yes
C	260	Yes	Yes	0.180	No
D	140	Yes	No	0.030	Yes
E	85	No	No	0.008	Yes
F	42	No	No	0.220	No
G	18	No	No	0.310	No
H	6	No	No	0.400	No

The call sets are $T=\{A,B,C,D\}$, $R=\{A,B,C\}$, and $E=\{A,B,D,E\}$. For threshold versus rank, the intersection has three barcodes and the union has four: $$ J(T,R)=\frac{3}{4}=0.750 $$ For threshold versus EmptyDrops, the intersection is $\{A,B,D\}$ and the union is $\{A,B,C,D,E\}$, giving $J(T,E)=3/5=0.600$. For rank versus EmptyDrops, the intersection is $\{A,B\}$, while the union is again $\{A,B,C,D,E\}$, giving $J(R,E)=2/5=0.400$. The lowest pairwise similarity is for rank versus EmptyDrops, but these numbers describe overlap, not accuracy. Barcode C, called by the two count-based approaches but not EmptyDrops, and barcode E, called only by EmptyDrops, are especially useful candidates for targeted inspection.

Read Disagreements Against the Data

A barcode-rank plot orders barcodes from largest to smallest total count and displays the count against rank, commonly with logarithmic axes. Add the threshold as a horizontal reference and mark the rank-based cutoff. Then identify EmptyDrops-only calls and barcodes called by the count-based methods but not by EmptyDrops. This makes it easier to see whether disagreements cluster near a knee, occur far below a count cutoff, or span a broad region of the curve.

A steep change in counts can support a rank-based boundary, but the curve does not guarantee that all barcodes on one side are cells or all barcodes on the other side are empty droplets. The knee can be diffuse, absent, or sensitive to which barcodes are included. A threshold is transparent but imposes a fixed count boundary. EmptyDrops can distinguish a low-count barcode whose gene composition is unusual under the ambient model from one consistent with that model; its call is conditional on the test and analysis settings, not a direct verification of cell identity.

Worked Example: Investigating an EmptyDrops-Only Barcode

In the hypothetical table, barcode E has 85 UMIs. It falls below the count threshold of 100 and lies below the selected rank cutoff, but its EmptyDrops adjusted $q$-value is 0.008, below the stated 0.05 criterion. The first conclusion is limited: under the fitted ambient-RNA null and the specified multiple-testing procedure, E meets the rule for an EmptyDrops call. That result does not establish that E is a viable cell.

The appropriate next diagnostic is to inspect E in the barcode-rank plot and review the gene-count profile used by the test alongside the estimated ambient profile. If E’s profile is genuinely distinct from the estimated background, the result is consistent with cell-associated RNA despite its low total count. If the ambient profile is poorly estimated, the tested set is unsuitable, or the profile is not robust to reasonable analysis choices, the call deserves caution. A low count is a reason to investigate, not by itself proof of an empty droplet.

Now consider barcode C: it has 260 UMIs and is included by the chosen count threshold and rank cutoff, but its $q$-value is 0.180. It does not meet the EmptyDrops criterion. This could occur if its observed composition is not sufficiently distinguishable from ambient RNA under the test, but the $q$-value does not prove C is empty. The disagreement identifies a candidate for review; the three procedures do not provide three independent confirmations because all use information from the same count data.

A Short Justification for the Overlap Summary

Jaccard similarity is the fraction of the barcodes called by at least one of two methods that both methods call. The denominator must count each barcode in the union once. Since the intersection is counted in both set sizes, inclusion–exclusion gives:

|A\cup B|=|A|+|B|-|A\cap B|

Thus the Jaccard denominator can also be calculated as $|A|+|B|-|A\cap B|$. For example, if one method calls 500 barcodes, another calls 470, and 440 are shared, the union has $500+470-440=530$ barcodes, so $J=440/530=0.8302$. If the union is nonempty and the call sets are identical, the intersection equals the union and $J=1$. If the union is nonempty and there is no overlap, the numerator is zero and $J=0$. If both sets are empty, the union and intersection are both empty and the ratio is undefined; report it as undefined or state a chosen convention rather than silently assigning a value.

Jaccard similarity omits barcodes that neither method calls. This is useful when the focus is disagreement among candidate calls, but it means that Jaccard is not the same as overall agreement across every barcode in $U$. Always show the call counts and, where useful, the numbers called by only one method as well as the overlap score.

Worked Example: Summarizing Disagreement in a Larger Barcode Set

Suppose three methods were applied to the same hypothetical universe of 10,000 barcodes. There are 400 barcodes called by all three; 40 called by threshold and rank only; 60 called by threshold and EmptyDrops only; 30 called by rank and EmptyDrops only; and 30 called only by EmptyDrops. No barcode is called only by threshold or only by rank. The remaining $10{,}000-(400+40+60+30+30)=9{,}440$ are called by none.

The threshold total is $400+40+60=500$; the rank total is $400+40+30=470$; and the EmptyDrops total is $400+60+30+30=520$. Threshold and rank share $400+40=440$ calls, so their Jaccard similarity is $440/(500+470-440)=440/530=0.8302$. Threshold and EmptyDrops share $400+60=460$, with union $500+520-460=560$, giving $460/560=0.8214$. Rank and EmptyDrops share $400+30=430$, with union $470+520-430=560$, giving $430/560=0.7679$.

The 9,440 barcodes called by none do not change these pairwise Jaccard values. They would matter for a measure of agreement across the full universe. Reporting both the pairwise overlaps and the call counts makes the result more interpretable than reporting a similarity score alone.

Assumptions, Diagnostics, and Failure Modes

Each method’s interpretation depends on its own assumptions and choices. A count threshold assumes a selected total-count boundary is useful for the sample being analyzed; it does not model variation in RNA content across cell types or states. A rank-based call assumes the count ordering and curve contain a boundary that can be identified meaningfully; gradual transitions or multiple populations can make one global cutoff ambiguous. EmptyDrops relies on a suitable ambient-RNA null and the specified test and false-discovery procedure. Its results can be affected by how ambient RNA is estimated, which barcodes are tested, and how eligibility and significance cutoffs are set.

Useful checks include:

Verify that methods are compared on the same barcode identifiers and the same count matrix, and check that count units are consistent.
Plot total counts against rank, mark both count and rank cutoffs, and label the categories of disagreement rather than looking only at a single overlap statistic.
Inspect the counts and gene-count profiles of barcodes called by only one method, especially those close to a threshold or knee.
Check that the EmptyDrops tested barcode set and ambient estimate are appropriate for the data; record any minimum-count eligibility rule and the multiple-testing criterion.
Look for a diffuse or multiple-knee curve, a concentration of disagreements around a cutoff, or a pattern where low-count EmptyDrops calls are numerous enough to warrant checking the ambient model and settings.

These are diagnostics, not automatic criteria for overturning calls. A disagreement concentrated near a cutoff may reflect a genuinely ambiguous boundary; a low-count EmptyDrops call may reflect cell-associated RNA or model misspecification. Conversely, high overlap can occur even when methods share a systematic failure. Do not treat a method’s call as biological truth just because it agrees with another method.

Choosing and Reporting a Comparison

Approach	Primary evidence	Useful diagnostic role	Main limitation
Count threshold	Total counts compared with a fixed cutoff	Provides a simple, reproducible reference boundary	One cutoff may not represent all cell-associated RNA levels
Barcode-rank method	Shape of ordered total counts	Shows distributional transitions and boundary ambiguity	A clear, unique knee may not exist
EmptyDrops	Departure of gene-count composition from an ambient null	Highlights some low-count barcodes not captured by count-only rules	Depends on the adequacy of the ambient model, tested set, and testing procedure

The methods answer related but different questions, so their output should be treated as complementary evidence. A sound report states the rules used, gives the number of barcodes called by each method, summarizes pairwise overlaps, and describes the characteristics of important disagreement groups. If a call set is empty, say so; for its Jaccard comparison with another empty set, state that the value is undefined unless a convention was chosen and documented.

Common Mistake

Common mistake: “The methods agree on most barcodes, so the shared calls are confirmed cells; the Jaccard score is 1 whenever the two sets match.” Agreement is not confirmation, because methods can share biases and none of these comparisons supplies biological truth labels. Also, Jaccard is 1 for identical nonempty sets; if both sets are empty, its defining ratio is $0/0$ and is undefined unless a convention is explicitly adopted. Report the call counts and interpret overlap as agreement between procedures, not validation.

Key takeaway: Use disagreement to locate barcodes and assumptions that need examination. Compare methods on the same barcode universe, inspect the discordant barcodes against the rank curve and count summaries, and keep a method’s statistical call distinct from a claim of biological cell identity.

Check Your Understanding

Answer the questions, then compare your reasoning with the solutions.

Method A calls 12 barcodes, method B calls 10, and 8 are shared. Calculate their Jaccard similarity.
Both methods call no barcodes. What is their Jaccard similarity from the defining formula?
A barcode is below a total-count threshold but passes the EmptyDrops adjusted-$q$ criterion. What can and cannot be concluded?
Why should a comparison state the barcode universe and the number of calls from each method, rather than report Jaccard alone?

Solutions. (1) The union has $12+10-8=14$ barcodes, so $J=8/14=0.5714$. (2) Both the intersection and union have size zero, so the formula is $0/0$ and the similarity is undefined unless a convention is stated. (3) The barcode meets the specified EmptyDrops decision rule under its fitted ambient model and testing procedure; this does not prove it is a cell. Its count and profile should be investigated in context. (4) Methods applied to different universes are not directly comparable, and Jaccard excludes barcodes called by neither method. Call counts and the universe clarify what the overlap represents.