Why Rare-Cell Features Can Be Missed
A feature-selection rule usually summarizes expression across all cells. That is useful when the goal is to represent broad variation, but a gene expressed mainly in a small cell population can have only a modest effect on a pooled variability score. The population contributes few observations, so a strong difference in that group may be diluted by the many cells in which the gene is absent or low.
This is not necessarily a flaw in a variability statistic. A global ranking answers a global question: which genes vary across the sampled cells on the chosen expression scale? Preserving rare-cell signals requires a complementary question: are there genes with a distinct, plausible detection pattern in a small population that a global ranking could underemphasize?
The goal is not to replace global ranking with a claim that every rare-population candidate is informative. Instead, use a transparent check to identify candidates for further assessment, while keeping the ranking and the check conceptually separate.
A Mixture Model for Pooled Variability
Suppose cells are sampled from the two groups, expression values are comparable on the chosen scale, and the group-specific means and variances are finite. The law of total variance separates pooled variability into within-group variability and variability between group means:
The first two terms are the group-weighted within-group variances. The final term is the between-group contribution. If the rare group has a substantially different mean, the squared difference can be large, but it is multiplied by \(q(1-q)\). When \(q\) is small, this multiplier is approximately \(q\). Thus, a strong group-specific signal can still contribute modestly to pooled variance. The same dilution can affect mean-adjusted ranking methods: adjusting for the overall mean does not, by itself, remove the influence of the small group fraction.
Deriving the Between-Group Contribution
Write \(E(X\mid G)\) for the conditional mean. The total-variance identity is \(\operatorname{Var}(X)=E[\operatorname{Var}(X\mid G)]+\operatorname{Var}(E[X\mid G])\). The first term is the weighted average of the group variances. For the second term, \(E(X\mid G)\) equals \(\mu_1\) with probability \(q\) and \(\mu_0\) with probability \(1-q\), so its mean is \(\mu=(1-q)\mu_0+q\mu_1\). Therefore,
Adding this term to the weighted within-group variances gives the stated mixture formula. It shows why the population fraction matters even when the difference between the group means is unchanged.
Worked Example: A Strong Difference With Modest Pooled Variance
Consider hypothetical normalized expression for a gene in 1,000 cells. Twenty cells belong to a rare population, so \(q=20/1000=0.02\). Suppose the gene has mean expression \(\mu_1=4.1\) and variance \(\sigma_1^2=1.00\) in that population, and mean \(\mu_0=0.1\) and variance \(\sigma_0^2=0.09\) in the other cells. The group mean difference is \(4.1-0.1=4.0\) expression units.
The pooled variance is \[ (1-0.02)(0.09)+(0.02)(1.00)+(0.02)(0.98)(4.0)^2. \] The within-group contribution is \(0.98(0.09)+0.02(1.00)=0.0882+0.0200=0.1082\). The between-group contribution is \(0.02(0.98)(16)=0.3136\). Thus, the pooled variance is \(0.1082+0.3136=0.4218\) expression units squared, with pooled standard deviation \(\sqrt{0.4218}\approx0.6495\) expression units.
The gene differs by 4.0 expression units between groups, yet the rare group contributes only 2% of cells. Its pooled variability is not determined by the group difference alone; it also depends on the group fraction and within-group variability. A global ranking can therefore place it below genes with more widespread variation.
Detection Patterns as a Complementary Check
For sparse counts, a useful companion summary is whether a gene is detected in each cell. Define \(Z=1\) when the gene has a positive observed count and \(Z=0\) otherwise. Let \(r_1=P(Z=1\mid G=1)\) and \(r_0=P(Z=1\mid G=0)\). The pooled detection rate is
This expression has the same mixture structure: even a high detection rate in a rare group may produce a low pooled detection rate. Comparing group-specific rates can reveal a concentrated pattern that a pooled summary obscures. Detection is not a perfect measure of biological expression: a zero count can reflect limited sequencing depth, and positive-count rates can differ across groups because of depth or technical conditions.
If population labels are available, they should be defined independently of the candidate gene where feasible. If provisional clusters are used to inspect detection, the interpretation is weaker when the same gene helped create those clusters. In either case, report the group sizes and detection counts, not only a rate. With few rare cells, one or two observations can substantially change the estimated rate.
Worked Example: Comparing Pooled and Group-Specific Detection
In a hypothetical set of 1,000 cells, 20 are assigned to a rare population and 980 to the remaining cells. A gene is detected in 16 of the 20 rare-population cells and in 10 of the other 980 cells.
The observed detection rates are \(16/20=0.800\) in the rare group and \(10/980\approx0.01020\) outside it. The pooled rate is \((16+10)/1000=26/1000=0.0260\), or 2.60%. Thus, the gene is detected in 80.0% of the rare group but only about 1.02% of the other cells. The ratio of these observed rates is \(0.800/0.01020\approx78.4\).
The pooled rate alone looks low, whereas the group-specific rates show strong enrichment in the rare group. This is a descriptive pattern, not proof of specificity: the rare group has only 20 cells, and detection may depend on sequencing depth, sample composition, and how group membership was assigned.
A Practical Retention Check
A detection-pattern check can be used alongside the global ranking produced by the selected variability method and threshold. One option is to keep the ranking as the primary feature set and create a small, clearly labeled reserve of candidates that meet a pre-specified population-specific criterion. For example, a workflow might flag genes with a rare-group detection rate above a chosen minimum and a substantial difference from the rate outside that group. The numerical cutoffs depend on the study and should not be presented as universal standards.
Record how the population was defined and which expression or count data are used for the global ranking. Define detection explicitly, such as a positive observed count.
For each candidate, tabulate detected and undetected cells by group. Examine group sizes and check whether sequencing depth or sample composition could explain the pattern.
Use a stated rule to flag plausible candidates that are not in the global feature set. Treat the result as a shortlist for evaluation, not as a replacement ranking or a test of biological truth.
State the global selection rule, the additional check, the number of candidates retained, and whether downstream conclusions change when the reserve is included.
Worked Example: Retaining a Candidate Outside the Global Cutoff
Suppose a hypothetical global ranking retains the top 300 genes. A candidate gene ranks 418th, so it does not meet that cutoff. In independently assigned cell groups, it is detected in 16 of 20 rare-population cells and 10 of 980 other cells, as in the preceding example. Before inspecting this candidate, the analyst specified a flag requiring at least 15 detected cells in the rare group, a rare-group detection rate of at least 0.50, and a rate ratio of at least 10 when the outside-group rate is nonzero.
The candidate has 16 detected rare-group cells, meeting the count requirement. Its rare-group rate is \(16/20=0.800\), meeting the 0.50 requirement. Its rate ratio is \(0.800/(10/980)\approx78.4\), meeting the threshold of 10. It is therefore flagged for a reserve set, although its global rank remains 418th. The conclusion is that it meets this stated descriptive rule, not that it is a validated marker or should automatically be used in every analysis.
Diagnostics, Alternatives, and Limitations
A detection-pattern check is most informative when its assumptions are examined. The group labels must correspond to a meaningful population for the question, and the expression measurements should be comparable across groups. In particular, compare library-size or sequencing-depth distributions. If the rare group has systematically greater depth, it may show more detected genes even without a population-specific biological difference. If samples differ in composition, a pooled group contrast can also reflect sample differences rather than cell identity. Inspect rates by sample where possible rather than relying only on an aggregate.
Several warning signs call for caution: the rare group contains very few cells; one sample supplies nearly all of its cells; detection changes sharply after accounting for depth; or the group definition used the candidate gene itself. A low outside-group rate can also make a rate ratio unstable, especially when the outside group has no detections. In that case, show the two rates and counts rather than adding an arbitrary constant and presenting the resulting ratio as definitive.
| Approach | What it emphasizes | Key limitation |
|---|---|---|
| Global variability ranking | Variation across all sampled cells on the selected expression scale | A small group contributes few observations and can have a diluted pooled signal |
| Group-specific detection check | Differences in the fraction of cells with observed expression | Rates depend on depth, group size, and how groups were defined |
| Group-specific expression summaries | Differences in expression levels within and between defined groups | Requires defensible group labels and does not by itself produce a universal feature ranking |
These approaches answer related but different questions. A global ranking is appropriate when broad variation across the dataset is the aim. A group-specific check is useful when retaining features associated with a small population is important. Neither should be treated as universally superior. A detection difference can be technically driven, and a gene selected by a group-aware check may not improve the intended downstream analysis. Keep the rule modest, report it separately, and assess whether substantive conclusions depend on the additional candidates.
Common Mistake
A related mistake is to assume that every gene outside the global cutoff should be rescued. That defeats the purpose of a selection threshold and can add many weak or technically driven features. Retention should follow a defined, limited check and be reported as a separate decision, rather than silently altering the interpretation of the global ranking.
Check Your Understanding
Try each question, then compare your reasoning with the solutions below.
- If \(q=0.01\), \(\mu_0=0\), \(\mu_1=5\), and both within-group variances are zero, what is the pooled variance?
- In 10 rare-population cells, a gene is detected in 8; in 990 other cells, it is detected in 20. Calculate both group rates and the pooled rate. Why can the pooled rate be misleading?
- What is one reason to define the rare population independently of a candidate gene before checking its detection pattern?
- A rare group has higher sequencing depth than the rest of the cells. Why might its detection rate be higher even if biological expression were similar?
- Does meeting a pre-specified detection flag prove that a gene is a validated marker? State what the flag supports and one further concern to examine.
Solutions. 1. The pooled variance is \(q(1-q)(\mu_1-\mu_0)^2=0.01(0.99)(25)=0.2475\) expression units squared. 2. The rates are \(8/10=0.800\) and \(20/990\approx0.02020\); the pooled rate is \(28/1000=0.0280\). The pooled rate is only 2.80% because most cells are outside the rare group, so it obscures the 80.0% rate within that group. 3. Independent definition reduces circularity: if the gene helps define the group, its apparent group-specific detection can partly reflect that construction. 4. Greater depth makes it more likely that low-abundance transcripts produce positive observed counts, raising detection without requiring a biological difference. 5. No. The flag supports that the observed data meet a stated descriptive rule; examine, for example, whether the pattern is consistent across samples and not explained by depth.