From a Score to a Calling Rule
A doublet-detection method assigns each cell barcode a score that reflects how doublet-like it appears. The score is useful for ordering barcodes, but an ordering alone does not say how likely any one barcode is to be a doublet. A score of 0.8, for example, is not an 80% probability unless the method has been calibrated and validated as a probability model for the population being analyzed.
The practical task is to turn scores into calls: mark some barcodes as likely doublets for downstream handling, while retaining others as likely singlets. This requires a threshold and a reason for choosing it. An expected doublet rate can provide a useful scale for the number of calls, and labeled artificial doublets can show how different thresholds trade sensitivity against false positives. Neither source of information, by itself, guarantees that a particular barcode is truly a doublet.
The direction of the score must be checked: some tools may use a smaller value to indicate stronger evidence. Reverse the inequality if needed. The examples here use the convention that higher scores are more doublet-like. If scores tie at the threshold, define in advance whether the rule includes all tied barcodes; the examples use \(D_i \ge t\).
What Calibration Data Can Tell You
Artificial doublets provide a labeled reference set: barcodes constructed to represent doublets are positive examples, while a chosen reference set of singlets supplies negative examples. At each candidate threshold, calculate the fraction of artificial doublets above the threshold and the fraction of reference singlets above it. These are empirical estimates of sensitivity and false-positive rate for those calibration sets.
Those estimates depend on the calibration data representing the real barcodes to which the rule will be applied. In particular, simulated doublets need to resemble the doublets that occur in the sample, and the reference singlets need to resemble real singlets. Artificial examples may not represent every doublet type equally well. Therefore, measured sensitivity is not automatically the true sensitivity for all doublets in the experiment, and the score distributions should be treated as evidence about threshold behavior rather than as perfect ground truth.
Worked Example: Comparing Two Candidate Thresholds
Suppose a hypothetical calibration set contains 100 artificial doublets and 400 reference singlets. At threshold \(t=0.70\), 72 artificial doublets and 24 reference singlets have scores at least 0.70. At threshold \(t=0.62\), the corresponding counts are 85 and 64. Calculate the sensitivity and false-positive rate at each threshold.
At \(t=0.70\), sensitivity is \(72/100=0.72\), or 72%. The false-positive rate is \(24/400=0.06\), or 6%. At \(t=0.62\), sensitivity is \(85/100=0.85\), or 85%, while the false-positive rate is \(64/400=0.16\), or 16%.
Lowering the threshold calls more barcodes: it captures 13 additional artificial doublets, but also calls 40 additional reference singlets. Thus the choice is a trade-off, not a discovery that 0.62 or 0.70 is intrinsically correct. Whether the added sensitivity is worth the added false positives depends on the purpose of the filtering step and the credibility of the calibration data.
Using an Expected Rate Without Forcing the Answer
The expected doublet rate is an estimate of the overall proportion of barcodes that are doublets under the experimental loading conditions. It helps assess whether the number of calls is broadly plausible. It does not identify which particular barcodes are doublets, and it need not equal the observed call fraction: some doublets may have low scores, while some singlets may have high scores.
One simple rule is to call the highest-scoring fraction of barcodes up to a specified expected rate. This can be a useful operational choice when the rate is credible and the scores meaningfully rank the relevant doublets. But it forces a fixed number of calls even when scores show no clear separation, and it can miss more doublets than expected if the score ranking is imperfect. A fixed-fraction rule is therefore not the same as a threshold justified by measured sensitivity and false-positive behavior.
Worked Example: A Rate-Based Call List
In a hypothetical sample of 1,000 barcodes, suppose the expected doublet rate is 8%. A rate-based rule would flag the 80 barcodes with the highest scores. If the 80th-highest score is 0.74, the resulting rule is to call barcodes with scores at least 0.74, with a tie policy needed if more than 80 barcodes share that score.
This calculation gives \(1{,}000 \times 0.08=80\) candidate calls. It does not show that all 80 are doublets, that no doublets occur among the remaining 920, or that a score of 0.74 represents an 8% probability. The rate-based threshold is defensible only as a ranking-based allocation of an expected number of calls; it is not a validation of those calls.
Choosing a Threshold for a Sensitivity Goal
A different approach starts with the consequence of missing doublets. If retaining doublets is especially costly, specify a target sensitivity and select a threshold that attains it in the calibration data. Among thresholds that meet the target, a higher threshold generally produces fewer false positives, because raising \(t\) can only remove calls. This makes the rule explicit: the sensitivity target is the priority, while the calibration score distribution determines the resulting call burden.
For a finite labeled set, calculate sensitivity at each candidate threshold rather than assuming that an arbitrary score value has a universal interpretation. The selected threshold should also be checked against the false-positive rate and against the expected rate in the actual sample. The target is a design choice, not a mathematical guarantee: sampling variability and mismatch between artificial and real doublets can make actual sensitivity differ from the calibration estimate.
Worked Example: Meeting a Prespecified Sensitivity Goal
Consider a hypothetical calibration set with 20 artificial doublets and 100 reference singlets. The table gives the number scoring at least each candidate threshold.
| Threshold \(t\) | Artificial doublets called | Reference singlets called |
|---|---|---|
| 0.80 | 12 of 20 | 2 of 100 |
| 0.65 | 17 of 20 | 8 of 100 |
| 0.50 | 19 of 20 | 20 of 100 |
Suppose the stated goal is sensitivity of at least 80%. At \(t=0.80\), sensitivity is \(12/20=0.60\), which misses the goal. At \(t=0.65\), it is \(17/20=0.85\), which meets the goal; the false-positive rate is \(8/100=0.08\). At \(t=0.50\), sensitivity is \(19/20=0.95\), but the false-positive rate rises to \(20/100=0.20\). Of these candidates, 0.65 is the highest threshold that meets the stated goal, so it gives the fewest calls while satisfying the empirical sensitivity requirement.
Now suppose the actual sample has 1,000 barcodes and an expected doublet rate of 6%. If the calibration sensitivity and false-positive rate applied exactly to the real sample, the expected called fraction would be \(0.06(0.85)+0.94(0.08)=0.051+0.0752=0.1262\). That corresponds to about \(1{,}000(0.1262)=126\) calls. Of these, the expected number of true doublets called would be \(1{,}000(0.06)(0.85)=51\), and the expected number of singlets called would be \(1{,}000(0.94)(0.08)=75.2\). The calculation is conditional on the calibration rates transferring to the real sample; it is not evidence that exactly 126 calls, or exactly 51 true doublets, will occur.
Why a Score Is Not a Probability
A ranking score and a probability answer different questions. A ranking tells us which barcode appears more doublet-like according to the method. A probability would quantify the chance of a doublet among barcodes with that score, under a specified population and calibration procedure. To see why the expected rate matters, let \(\pi\) denote the prevalence of doublets, let \(Se(t)\) denote sensitivity, and let \(FPR(t)\) denote the false-positive rate. If those rates apply to the sample, the fraction of called barcodes that are true doublets is
This expression follows by counting expected calls from the two classes: the doublet contribution is \(\pi Se(t)\), and the singlet contribution is \((1-\pi)FPR(t)\). The total called fraction is their sum, so the share of calls that are doublets is the doublet contribution divided by that total. This is a threshold-level predictive value under the stated prevalence and rate assumptions, not an individual probability assigned by the raw score.
For the preceding example, using \(\pi=0.06\), \(Se=0.85\), and \(FPR=0.08\), this share is \(0.051/(0.051+0.0752)\approx0.404\), or about 40.4%. It differs from both the 85% sensitivity and the 6% expected prevalence. It is only a useful estimate if the prevalence and calibration rates are appropriate for the target sample.
Calibration Checks and Failure Modes
Before acting on a threshold, compare the score distributions for artificial doublets and reference singlets, and inspect how sensitivity and false-positive rate change over candidate thresholds. Strong overlap means no threshold can simultaneously achieve high sensitivity and a low false-positive rate. A large gap between the threshold-based call fraction and the expected rate is a reason to investigate calibration and assumptions, not a reason to silently adjust the interpretation of the score.
Finite calibration sets also make the estimated rates uncertain. In the sensitivity example, 17 of 20 artificial doublets corresponds to 85%, but each observation changes the estimate by five percentage points. Report the calibration counts along with the percentages, and avoid presenting a small set's empirical rate as exact. Thresholds can be unstable when many scores are tied or clustered near the cutoff; record the tie rule and consider whether small score changes would alter many calls.
A further limitation is that the expected rate describes the overall sample, while a score may detect some kinds of doublets more readily than others. If the calibration examples do not represent the doublets in the experiment, the measured sensitivity may not transfer. A discrepancy between expected and observed call rates can therefore reflect imperfect detection or imperfect assumptions about prevalence, rather than one uniquely identifiable problem.
Comparing Practical Threshold Strategies
| Strategy | What it prioritizes | Main limitation |
|---|---|---|
| Call the top fraction based on expected rate | A call-list size consistent with a chosen rate | Forces a number of calls, even without a clear score separation |
| Choose a threshold to meet a sensitivity target | Capturing a prespecified fraction of calibration doublets | False-positive burden and transfer to real cells must be checked |
| Choose a threshold based on score separation and rate plausibility | A balance between calibration evidence and the sample's expected rate | Requires judgment; neither source alone determines a unique cutoff |
These strategies are not interchangeable. A rate-based rule is most transparent when the goal is to produce a set of candidates of roughly expected size. A sensitivity-based rule is more appropriate when missing doublets has a specified cost and representative labeled calibration data are available. In practice, compare the resulting call fraction with the expected rate and report the rule, the threshold, and the calibration counts so that the decision can be assessed.
Common Mistake
Incorrect reasoning: “The expected doublet rate is 8%, so the top 8% of scores are definitely doublets, and a score of 0.8 means an 80% chance of being a doublet.”
Correction: The expected rate can motivate a top-fraction ranking rule, but it does not verify individual calls. A score's numerical scale is not a probability scale unless a probability calibration has been established for the target population. State the rule as a threshold or ranked call list, report its behavior on calibration data, and describe the expected rate as context rather than ground truth.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A method gives a barcode a score of 0.93. What can be concluded from that number if the score is only a ranking?
- At a threshold, 36 of 40 artificial doublets and 15 of 150 reference singlets are called. Calculate sensitivity and false-positive rate.
- A team calls exactly the top 7% of barcodes because the expected rate is 7%. Does that establish that each called barcode is a doublet? Why or why not?
- At one threshold, \(\pi=0.10\), \(Se=0.80\), and \(FPR=0.05\). Using the threshold-level formula, what fraction of calls are expected to be true doublets, if the rates transfer to the sample?
Solutions. (1) The barcode ranks relatively high in doublet-like evidence; 0.93 alone does not mean a 93% probability. (2) Sensitivity is \(36/40=0.90\), or 90%; false-positive rate is \(15/150=0.10\), or 10%. (3) No. The rule assigns a ranked call list of that size but does not establish the status of individual barcodes or account for ranking errors. (4) The fraction is \(0.10(0.80)/[0.10(0.80)+0.90(0.05)]=0.08/(0.08+0.045)\approx0.640\), or 64.0%.