Why Filter Genes Before Testing?
An RNA-seq experiment can measure tens of thousands of genes, including genes with almost no reads in any sample. A test can still return a statistic for such a gene, but that does not mean the data contain useful information about its expression difference. With very small counts, random sampling can dominate the observed pattern: a change from zero reads to one read may look large on a relative scale, while providing little evidence about a reproducible difference between conditions.
A low-expression filter removes genes whose counts are too small or too sparse to support a useful test. The rule is based on expression information, not on whether a gene appears to have a large difference or a small p-value. This distinction matters: filtering can reduce the number of weakly informative tests, but choosing genes because their observed tests look significant is a different, outcome-dependent selection step.
A practical rule might retain a gene if its CPM is at least a chosen threshold in at least a chosen number of samples. For a two-group experiment with three biological replicates per group, one possible rule is “at least 1 CPM in at least three samples.” Requiring expression in at least the size of the smaller group can allow a gene expressed mainly in one condition to remain testable. The cutoff and required number of samples are choices, not universal constants; they should reflect sequencing depth, study design, and the level of expression at which the analysis can provide useful precision.
The counts are assumed to come from independent biological samples, with library-size normalization that is adequate for comparing samples. For understanding why low counts are problematic, suppose a gene’s counts follow a negative binomial model with mean \(\mu\) and dispersion \(\alpha\), so that $$ \operatorname{Var}(Y) = \mu + \alpha\mu^2. $$ This model describes count variability beyond the Poisson component. It is a useful explanation of the information problem; applying a filter does not itself establish that the model fits every gene.
Why Very Small Counts Are Unstable
The relative variability of a count is summarized by its coefficient of variation, the standard deviation divided by the mean. Under the negative binomial variance above, for \(\mu>0\), $$ \operatorname{CV}(Y)^2 = \frac{\operatorname{Var}(Y)}{\mu^2} = \frac{\mu+\alpha\mu^2}{\mu^2} = \frac{1}{\mu}+\alpha. $$ Thus, when the mean count is small, the term \(1/\mu\) is large. As the mean increases, the count becomes more precise relative to its mean, although the dispersion contribution \(\alpha\) remains. This is one reason a gene with almost no reads can have an imprecise estimated difference even when a fold-change calculation looks dramatic.
Worked Example: Relative Variability at Low and Moderate Counts
Suppose, for illustration, that two genes have expected counts \(\mu=0.5\) and \(\mu=50\) per sample, and that both have dispersion \(\alpha=0.2\). These are hypothetical values used to compare relative variability under the stated negative binomial model.
For the lower-mean gene, $$ \operatorname{CV}^2 = \frac{1}{0.5}+0.2=2+0.2=2.2, \qquad \operatorname{CV}=\sqrt{2.2}\approx 1.483. $$ Its standard deviation is about 1.483 times its mean. In fact, the model variance is \(0.5+0.2(0.5)^2=0.55\), giving standard deviation \(\sqrt{0.55}\approx0.7416\) counts.
For the higher-mean gene, $$ \operatorname{CV}^2 = \frac{1}{50}+0.2=0.02+0.2=0.22, \qquad \operatorname{CV}=\sqrt{0.22}\approx0.469. $$ Its model variance is \(50+0.2(50)^2=550\), so its standard deviation is about 23.45 counts. The absolute standard deviation is larger, but it is much smaller relative to the mean than for the low-count gene. This comparison explains why tiny counts tend to give unstable relative estimates; it does not imply that every higher-count gene is well measured or that dispersion can be ignored.
Applying a Count-Based Rule
The filter should be fully specified: the expression measure, the cutoff, and the number of samples that must meet the cutoff. A transparent rule makes it possible to reproduce the retained gene set and assess whether the rule is appropriate for the experiment. The following example uses equal effective library sizes to make the CPM arithmetic direct.
Worked Example: A CPM Filter in a Two-Group Experiment
Consider six hypothetical samples, three per group, each with an effective library size of 10,000,000 reads. Apply the rule “retain a gene if CPM is at least 1 in at least three of the six samples.” At this library size, 1 CPM corresponds to $$ 1 = 10^6\frac{Y_{ij}}{10{,}000{,}000} \quad\Longleftrightarrow\quad Y_{ij}=10 \text{ reads}. $$ So a sample meets the threshold exactly when its count is at least 10.
Gene A has counts \(0,1,0,2,0,1\). Its CPM values are \(0,0.1,0,0.2,0,0.1\). No sample reaches 1 CPM, so Gene A fails the rule.
Gene B has counts \(12,8,0,15,9,0\). Its CPM values are \(1.2,0.8,0,1.5,0.9,0\). Two samples reach 1 CPM, which is fewer than the required three, so Gene B also fails.
Gene C has counts \(20,18,22,0,16,14\). Its CPM values are \(2.0,1.8,2.2,0,1.6,1.4\). Five samples reach the threshold, so Gene C passes. The rule retains Gene C based on its expression pattern, without asking whether its groups differ.
A threshold can be set using CPM, normalized counts, or another prespecified expression measure. What matters is that the measure accounts for differences in library size and that the rule is appropriate for the experiment. A raw-count cutoff alone can behave inconsistently when library sizes vary: the same count represents a smaller fraction of a deep library than of a shallow one. CPM based on effective library sizes makes this exposure difference explicit.
Use a measure adjusted for library size, such as CPM calculated with effective library sizes.
State the minimum expression and how many samples must reach it. Consider the number of biological replicates and the expression level the study can reasonably assess.
Record which genes pass before selecting or ranking genes by their differential-expression statistics.
Summarize the number of genes retained and inspect the distribution of expression and sample patterns, including genes expressed mainly in one group.
Filtering Is Not Significance Selection
A filter asks whether there is enough expression information to justify testing. Significance selection asks whether the observed test result meets a criterion, such as a p-value below a cutoff. Those questions are not interchangeable. Filtering on observed p-values to decide which genes are “eligible” for testing uses the result of the test to define the set on which results will be reported.
An expression filter should be chosen independently of the gene’s observed contrast or test statistic, to the extent required by the testing procedure. A simple rule based on overall normalized expression is often used for this purpose, but independence is not automatic in every design or with every data-dependent rule. If filtering and the test statistic are related under the null, the filter can change the null distribution among retained genes. The validity of subsequent p-values and multiplicity adjustments depends on the filtering procedure’s properties and the analysis method. In particular, do not assume that any rule using counts is automatically harmless just because it does not explicitly mention p-values.
Worked Example: Expression Filtering Versus Selecting Small P-Values
Suppose an analysis uses six hypothetical samples with effective library sizes of 10,000,000 reads and the same rule as before: CPM at least 1 in at least three samples. Two genes have these counts and illustrative test results:
| Gene | Counts across six samples | Samples at or above 1 CPM | Illustrative p-value |
|---|---|---|---|
| D | 2, 0, 1, 0, 1, 0 | 0 | 0.012 |
| E | 15, 13, 14, 16, 12, 15 | 6 | 0.31 |
For each sample, 1 CPM is 10 reads because \(10^6(10)/10{,}000{,}000=1\). Gene D has a maximum count of 2, corresponding to \(10^6(2)/10{,}000{,}000=0.2\) CPM, and therefore fails the expression filter. Gene E has a minimum count of 12, corresponding to 1.2 CPM, so it passes in all six samples. Under this rule, D is filtered out and E remains eligible for testing, regardless of their illustrative p-values.
Keeping D because its p-value is 0.012 would replace the stated expression rule with significance-based selection. Conversely, removing E because its p-value is 0.31 would also be significance-based selection, not low-expression filtering. The p-values here are illustrative outputs, not calculations from a specified test model; the point is that the filter decision is made from the expression criterion, not from those outputs.
Diagnostics, Alternatives, and Limitations
After filtering, check the consequences rather than treating the retained gene count as proof that the rule was good. Useful checks include the number and proportion of genes removed, the distribution of CPM among retained genes, and whether genes expressed chiefly in one condition still pass. Inspect whether a few samples have unusually many genes failing the rule; that pattern may indicate a sample-quality or library-size issue rather than a reason to change the gene filter. Also verify that the effective library sizes used for CPM agree with the normalization used in the count analysis.
Several approaches are possible. A fixed CPM threshold is easy to explain and reproduce. A filter based on average normalized abundance uses information across all samples, while a prevalence rule explicitly requires enough samples above a threshold. Some analysis procedures provide an independent-filtering rule tailored to their testing framework. Alternatively, one can test all genes and accept that very low-count genes may contribute little power while remaining part of the testing family. These choices are not equivalent in every setting; follow the assumptions of the selected method and report the rule used.
Filtering has limits. A low-expression gene can be biologically important, and a gene with high counts can still be variable or poorly modeled. A filter cannot repair incorrect normalization, confounding, poor replication, or model misspecification. It does not prove that a removed gene is unexpressed, nor does passing the filter mean that a gene is differentially expressed. It defines which genes have enough observed expression to enter the chosen testing procedure. Report the filter rule and the number of genes removed so the scope of the reported results is clear.
Common Mistake
Check Your Understanding
Answer each question using the definitions and calculations in this tutorial. Brief solutions follow the questions.
- With an effective library size of 20,000,000 reads, how many reads correspond to 1 CPM?
- A gene reaches 1 CPM in two samples. Does it pass a rule requiring at least three samples at or above 1 CPM?
- Under the negative binomial variance model, why is relative count variability especially large when the mean is small?
- Why is retaining genes because their p-values are small not the same procedure as low-expression filtering?
- Name one useful diagnostic after applying an expression filter, and one limitation of that diagnostic.
Solutions. (1) Solve \(1=10^6Y/20{,}000{,}000\), giving \(Y=20\) reads. (2) No; two qualifying samples are fewer than the required three. (3) The squared coefficient of variation is \(1/\mu+\alpha\), so the \(1/\mu\) term is large when \(\mu\) is small. (4) A p-value is a test result, whereas an expression filter is defined from an expression threshold; selecting on observed significance makes eligibility depend on the test outcome. (5) One can inspect how many genes were removed or whether genes expressed mainly in one group remain. Such checks reveal consequences of the rule but cannot establish that the filter is biologically optimal or that the count model is correct.