False Discovery Control in Barcode Calling

From Ambient-Test Evidence to Barcode Calls

An ambient test produces evidence about whether a barcode’s RNA is compatible with an ambient-only explanation. A small p-value is evidence against that barcode’s ambient-only null hypothesis, but it does not by itself specify how many barcodes should be called. When many barcodes are tested, some small p-values can occur by chance. False-discovery-rate (FDR) adjustment addresses this multiplicity by setting a rule for making calls across a defined family of tests.

The result depends on two choices that should be made explicit: the FDR threshold and the ambient-test universe, meaning the complete set of eligible barcodes whose p-values are included in the adjustment. A barcode call is therefore not determined by its p-value alone. It also depends on how that p-value ranks among the tested barcodes and how many hypotheses were tested.

FDR is an average error rate over repeated applications of a procedure under its assumptions. An FDR target of 0.05 does not mean that each called barcode has a 5% probability of being empty, or that exactly 5% of calls in this particular experiment will be false. The realized proportion can be higher or lower. The target concerns the expected proportion of false rejections among all rejections.

The ambient-test p-values must be valid for the hypotheses being adjusted. In particular, under a true null, a valid p-value is super-uniform: for every \(t\) between 0 and 1, \(P(p_i\le t)\le t\). If a test or an eligibility filter makes null p-values systematically too small, applying an adjustment cannot repair that lack of calibration. The introductory material on EmptyDrops tests for cell-associated RNA supplies the ambient-test context; here the focus is how a family of such p-values becomes a set of calls.

The Benjamini–Hochberg Step-Up Rule

Suppose \(m\) eligible barcodes have been tested, and write their p-values in increasing order as \(p_{(1)}\le p_{(2)}\le\cdots\le p_{(m)}\). Choose a target level \(q\), such as \(q=0.05\). The Benjamini–Hochberg (BH) procedure compares each ordered p-value with a rank-specific threshold.

This is a step-up rule: find the last ordered p-value that clears its threshold, then call every barcode with a p-value at least as small. It is not a rule that independently compares each p-value with \(q\), nor does it require every earlier p-value to pass its own threshold. The step-up structure allows a group of moderately small p-values to support calls when the whole ordered set provides sufficient evidence.

Why the Rule Controls an Expected Error Rate

Under independent, valid p-values, BH controls the FDR at no more than \(q\), and more specifically at no more than \(q m_0/m\), where \(m_0\) is the number of true null hypotheses in the tested family. The same control holds under certain positive dependence conditions, including positive regression dependence on the subset of true nulls (PRDS). These are assumptions about the joint behavior of the p-values, not consequences of sorting them.

A short justification shows why the rank thresholds matter. Consider a true-null barcode \(i\), whose p-value is uniform under the null and independent of the other p-values. Temporarily set its p-value to zero and let \(R_i\) be the number of BH rejections in that modified list. If the original procedure rejects barcode \(i\), setting its p-value to zero does not change the number of rejections: \(R_i=R\). Conditional on the other p-values, \(R_i\) is fixed, and rejection of \(i\) occurs when its p-value is at most \(qR_i/m\). Its conditional expected contribution to \(V/\max(R,1)\) is therefore

Summing this contribution over the \(m_0\) true nulls gives an upper bound \(m_0q/m\le q\). With super-uniform rather than exactly uniform null p-values, the corresponding probability is no larger, so the bound still holds. This argument relies on independence; under dependence, the appropriate guarantee needs a condition such as PRDS or a procedure designed for more general dependence.

An adjusted p-value is another way to report the same BH decisions. For the barcode at ordered rank \(i\), its BH-adjusted value is

The minimum over ranks \(j\ge i\) makes the adjusted values nondecreasing with rank. A barcode is called at target \(q\) exactly when its adjusted value is at most \(q\). In the eight-barcode example, the fifth ordered adjusted value is \(0.0496\), so the first five adjusted values are at most 0.05; the sixth is approximately \(0.0587\), so it is not called. An adjusted p-value is a threshold summary for the multiple-testing procedure, not the posterior probability that a particular barcode is a false call.

The Ambient-Test Universe Changes the Calls

The number \(m\) in every BH threshold is the number of p-values in the family. It must count all barcodes in the declared ambient-test universe, not just the barcodes that look promising after their p-values have been inspected. Eligibility rules may be needed to define which barcodes can be tested, but those rules should be specified and applied consistently. If selection uses the same evidence being tested, the validity of the resulting p-values must be justified for that selection; otherwise null p-values may no longer be calibrated.

Adding hypotheses can change calls even when the p-values for previously tested barcodes remain exactly the same. The following calculation isolates that effect.

Choosing a Threshold and Comparing Alternatives

The value \(q\) is a policy choice about the desired expected false-discovery proportion, not a quantity estimated from the observed p-values. A smaller target is more stringent. Increasing \(q\) weakens the BH thresholds proportionally and cannot reduce the set of BH rejections for a fixed p-value family. The target should be reported along with the procedure and the universe so that the calls can be interpreted and reproduced.

BH is not the only multiplicity procedure. Bonferroni compares each p-value with \(q/m\) and controls the family-wise error rate, the probability of making at least one false rejection. This can be more conservative when many barcodes are tested, because it targets any false call rather than the expected fraction of false calls. Benjamini–Yekutieli adjusts BH to control FDR under arbitrary dependence using a more stringent threshold involving the harmonic sum \(1+1/2+\cdots+1/m\); it can be substantially more conservative. These alternatives address different error goals or dependence conditions, so the method should match the intended guarantee.

In barcode applications, dependence can arise because barcodes share an estimated ambient profile, library-level conditions, or other components of the testing procedure. BH is standard when its independence or positive-dependence conditions are defensible, but those conditions should not be presumed solely because every barcode has a p-value. Where dependence is concerning, investigators can examine the test construction and consider a dependence-robust procedure, while recognizing its loss of power. Neither BH nor a more conservative adjustment fixes a misspecified ambient null or p-values that are anti-conservative.

Diagnostics, Limitations, and Common Mistakes

Before interpreting calls, check that the analysis retained the exact p-value family and its size, the target \(q\), and the adjustment method. Re-sort the p-values and verify the largest passing rank; a rank-order or denominator error can change the call set. It is also useful to inspect the distribution of p-values and the resulting calls across relevant barcode ranges, but such displays are diagnostics, not a replacement for valid p-values or a defined testing universe.

• Uncalibrated null p-values: If the ambient model or the test’s selection conditions produce too many small p-values under true nulls, BH’s guarantee may not apply. Investigate calibration under the test’s null assumptions rather than assuming adjustment cures the problem.
• Unclear universe: Different eligibility filters produce different \(m\) and potentially different calls. Record which barcodes were tested and why excluded barcodes were ineligible.
• Overinterpretation of \(q\): The target is an expectation across repeated applications under assumptions; it is not a per-barcode error probability or a promise about the realized fraction in one experiment.
• Boundary and implementation errors: Use the largest passing rank and call all p-values up to that rank. Comparing every p-value only to \(q\), or using \(m\) after dropping inconvenient results, is not the BH procedure.

A common mistake is to say, “This barcode has \(p=0.03\), so it is called because \(0.03<0.05\).” That reasoning ignores multiplicity. In the eight-barcode example, the rank-6 p-value \(0.044\) is below 0.05 but does not pass its BH threshold of 0.0375; it is not called. The correction is to identify the complete test family, order its p-values, apply the rank-specific thresholds, and interpret the resulting set of calls at the chosen target.