Comparing Doublet Detection Methods

Why Compare Doublet-Detection Methods?

A doublet call depends on the evidence a method uses, the assumptions built into its model, and the threshold used to turn a score into a label. A score-based method and an orthogonal method can therefore disagree even when both are behaving as designed. Comparing them is useful not because one call must automatically be correct, but because the pattern of agreement and disagreement can show where the evidence is consistent and where it needs investigation.

The comparison is meaningful only when both methods refer to the same set of cell barcodes and the same definition of a positive call. A score must be converted to a call using a stated threshold before it can enter a binary overlap table. If a method has no result for a barcode, that missing result is not a negative call. Exclude such barcodes from a paired table or report their missingness separately, and make the resulting comparison set explicit.

This table preserves the pairing: each barcode contributes to exactly one cell. The row and column totals describe the methods’ positive and negative call proportions. The off-diagonal cells, \(b\) and \(c\), identify the discordant calls; their sizes and biological or technical context are often more informative than a single overall agreement number.

Agreement Measures and Their Interpretation

The simplest measure is the observed proportion of agreement, \(P_o\). It counts both shared positive calls and shared negative calls. When doublet calls are uncommon, the many shared negatives can make \(P_o\) look high even if the methods disagree substantially about which barcodes are positive. It is therefore useful to report positive and negative agreement as well.

A chance-adjusted measure sometimes reported is Cohen’s kappa. Let \(p_A=(a+b)/n\) and \(p_B=(a+c)/n\) be the observed positive-call proportions for methods A and B. If two independent sets of labels were assigned with these same marginal proportions, their expected agreement would be \(P_e=p_Ap_B+(1-p_A)(1-p_B)\). Kappa compares observed agreement with this reference value.

The reference calculation treats the methods’ labels as independent while preserving their marginal positive-call proportions. It is a mathematical baseline, not a claim that the methods actually assign labels by chance. The denominator \(1-P_e\) is the normalization used by Cohen’s kappa, corresponding to the hypothetical case \(P_o=1\). It is not generally the maximum excess agreement attainable while holding the observed margins fixed. If the methods have different positive-call proportions, perfect agreement is impossible with those margins. Kappa is consequently a useful descriptive summary, but it should not replace the overlap table or the separate positive- and negative-agreement measures. If \(P_e=1\), kappa is undefined; if \(P_e\) is close to 1, its denominator is small and the statistic can be unstable.

Stratified Diagnostics Reveal What the Pooled Table Hides

A pooled table can summarize agreement across a sample, but its result depends on which cells are included and how they are distributed across samples or cell groups. A large stratum with many easy-to-classify negatives can dominate the total, obscuring weaker agreement among higher-risk cells. It is therefore useful to construct overlap tables within relevant strata, such as sample, sequencing run, cluster, or a defined risk group. These comparisons are descriptive: they show where agreement differs, but do not by themselves explain why.

Stratification should be chosen for a reason, not repeatedly searched until a favorable result appears. If the purpose is to compare performance across samples, show each sample’s denominators and call proportions. If the purpose is to inspect biological patterns, define the groups in advance where possible and state how cells were assigned. When samples differ substantially in size, a pooled summary weights large samples more heavily; a simple average of sample-specific summaries instead gives each sample equal weight. These answer different descriptive questions and should not be presented as interchangeable.

Interpreting Disagreement in Light of Method Assumptions

Agreement is not the same as accuracy. Two methods can agree because they detect the same evidence, share a limitation, or use related assumptions. Conversely, disagreement may arise because methods detect different subsets of doublets or because one call is sensitive to a threshold or data-quality issue. The earlier material on score calibration and genotype evidence provides the relevant context for understanding those call types; here, the comparison asks how their calls align on the same barcodes.

For a score-based method, inspect the underlying score distribution and the threshold used to create the binary call. Calls close to a threshold are more likely to change under modest recalibration. Compare the positive-call proportions as well as the overlap: one method calling far more barcodes positive can have a large A-only or B-only cell even when most barcodes agree. Do not change thresholds after seeing the overlap merely to improve agreement unless that recalibration is a planned part of the analysis and is reported transparently.

An orthogonal method may rely on evidence that the score-based method does not use, but orthogonal evidence is not automatically a perfect reference. Its applicability may depend on data availability, sample composition, or assumptions about the source of the signal. Likewise, score-based detection may be less informative for some doublet types, including cases whose profiles resemble a single cell. Interpret discordances using the plausible failure modes of both methods rather than automatically treating one column as the correct answer.

Assumptions, Failure Modes, and Alternatives

Before interpreting an overlap table, check that the comparison set is valid. The barcodes must be matched correctly; both methods must use a consistent positive-call definition; and missing or inapplicable results must not be silently coded as negatives. If one method was applied only to a selected subset, the comparison describes that subset, not necessarily all retained cells. Report the denominator and call rates so that selection and prevalence are visible.

The main alternatives answer different questions. Observed agreement is easy to understand but can be dominated by shared negatives. Positive and negative agreement separate performance on the two call categories without requiring a gold standard. Kappa adjusts observed agreement using the methods’ marginal call proportions, but its value depends on those margins and it does not identify which method is correct. If a genuinely valid reference classification is available, comparison to that reference can support sensitivity and specificity calculations; without such a reference and its assumptions, those quantities should not be inferred from agreement alone.

Several practical failure modes deserve attention:

• Comparing different barcode sets. Unequal or selected sets can make the overlap misleading. Use matched barcodes or explain the restriction and report missingness.
• Reporting only overall agreement. Shared negatives can inflate \(P_o\). Include the full table and category-specific agreement.
• Ignoring sample or group structure. Cells from the same sample are not independent biological replicates. Do not treat a large cell count as equivalent to many independent samples when making uncertainty claims.
• Calling one method a gold standard without justification. Agreement is symmetric; validation claims require a defensible reference and suitable assumptions.
• Over-interpreting a single summary. Threshold choice, method-specific limitations, and stratum composition can all change the apparent agreement.

If uncertainty estimates are needed, their design must respect the sampling structure. For example, when the target is variation across biological samples, resampling individual cells as though they were independent can produce unjustifiably narrow intervals. A sample-level analysis or resampling scheme may be more appropriate when enough independent samples are available. In many quality-control comparisons, transparent descriptive tables and stratified diagnostics are more informative than a formal test of whether two sets of calls differ.

A Practical Comparison Workflow

A disciplined comparison keeps the table, the methods’ assumptions, and the intended interpretation connected. Use the following sequence before drawing conclusions.