Filtering Low-Information Features Before Integration

Why Filter Before Integration?

An omics block may contain thousands of features, but not every feature provides usable information for every analysis. Some are detected in very few samples, barely vary across the study, or have poor technical reproducibility. Retaining such features can add noise, increase computational burden, and make downstream summaries harder to interpret. Filtering is a way to exclude features that fail prespecified criteria for prevalence, variability, or measurement quality.

The goal is not to keep only features that appear biologically interesting. A feature can be low prevalence or low variance and still matter scientifically, particularly if it is relevant to a rare subgroup or a specific hypothesis. Filtering should therefore be based on the assay, study purpose, and data quality—not on an assumption that every retained feature is biologically important. This tutorial treats filtering separately within each omics block because detection and reliability have different meanings across assays.

Prevalence, Variance, and Measurement Quality

Detection prevalence is the fraction of eligible samples in which a feature is detected according to an assay-specific definition. For feature \(j\), let \(d_{ij}=1\) if feature \(j\) is detected in sample \(i\), and \(d_{ij}=0\) otherwise. Its prevalence is \(\hat{p}_j=(1/n)\sum_{i=1}^{n}d_{ij}\). A prevalence rule retains the feature when \(\hat{p}_j\) is at least a chosen threshold \(\tau\).

The detection definition must match the assay and the handling of zeros, limits of detection, and missing values. A recorded zero may represent a genuine zero, a nondetection, or a value substituted for a measurement below a detection limit. These cases are not interchangeable. In particular, a missing measurement should not automatically be counted as a nondetection unless that is justified by the assay and data-processing protocol. The earlier material on zeros and detection limits is relevant when deciding what the indicator \(d_{ij}\) means.

A variance filter retains features whose measured values vary enough across eligible samples to pass a cutoff. For a feature with \(m_j\) observed values \(x_{1j},\ldots,x_{m_jj}\), its sample variance is \(s_j^2=\sum_{i=1}^{m_j}(x_{ij}-\bar{x}_j)^2/(m_j-1)\), where \(\bar{x}_j\) is the mean of those observed values. This calculation is meaningful only when at least two values are observed. Variance depends on the measurement scale, so the transformation and scale on which it is calculated must be stated. A variance cutoff has squared units; it is not a universal constant applicable to every block.

A measurement-quality filter uses technical information to exclude unreliable measurements or features. Possible criteria include the fraction of failed quality-control checks, signal-to-noise criteria, or variability among technical replicates. The criterion must be appropriate to the instrument and assay. For example, a coefficient of variation (CV), defined as the standard deviation divided by the mean and expressed as a percentage, can summarize relative replicate variability when measurements are positive and the mean is meaningfully above zero. CV is not appropriate on every transformed scale.

Choosing and Applying a Rule

A common unsupervised rule is to retain feature \(j\) if its prevalence is at least \(\tau\), its variance is at least \(v_{\min}\), and its quality measure meets a specified limit. Requiring all conditions is an intersection rule: failure of any one condition excludes the feature. This is not the only reasonable rule, but it is easy to audit because each feature’s reason for exclusion can be recorded.

Prevalence thresholds have a discrete consequence. Since the number of detections is an integer, the requirement \(\hat{p}_j\geq\tau\) is equivalent to \(\sum_i d_{ij}\geq\lceil n\tau\rceil\), where \(\lceil a\rceil\) is the smallest integer at least \(a\). This follows by multiplying both sides of \((\sum_i d_{ij})/n\geq\tau\) by \(n\), then taking the smallest integer count that can satisfy the resulting inequality. Thus, a small change in \(\tau\) can change the required count by one and alter retention.

Apply the rule separately within each block, using criteria that make scientific and technical sense for that assay. If samples are paired across blocks, use the intended analysis sample set consistently or clearly state any block-specific exclusions. Do not calculate a variance across samples whose measurements are not comparable because of assay processing or an inappropriate mixture of conditions. Filtering is not a substitute for resolving a serious design or measurement problem.

Threshold Sensitivity, Diagnostics, and Alternatives

A filtering result should be checked, not treated as automatic. For each block, report the initial feature count, the count passing each criterion, the final retained count, and the number excluded for each reason. Compare the final feature set under a small range of plausible cutoffs. A retention table or plot can reveal whether a minor threshold adjustment causes a large change in the number of retained features. When many features lie just above or below a cutoff, membership may be sensitive even if the total count changes little.

Also inspect whether detection or exclusion is concentrated in particular groups, batches, or sample types. A feature may appear low-prevalence because it is detected only in one scientifically relevant subgroup. Conversely, high overall prevalence can conceal failed detection in a subset of samples. If detection is associated with a technical batch, a prevalence filter may remove features because of batch-specific measurement failure rather than biological irrelevance. These patterns are diagnostic signals to investigate, not reasons to change a threshold after seeing a desired result.

Several alternatives answer different questions. A prevalence filter is useful when measurements are too sparse for reliable downstream use; a variance filter removes near-constant features on the specified scale; a quality filter addresses technical reliability. These criteria can be combined, but none is a substitute for the others. Selecting features by association with an outcome is supervised feature selection, not an unsupervised low-information filter. If prediction performance is evaluated by cross-validation, any outcome-informed selection—and any data-derived filtering rule—must be estimated using the training data in each split and then applied to the held-out data. Otherwise, information from the held-out observations can leak into model development.

Filtering has limits. A fixed variance threshold is scale-dependent, can be strongly influenced by outliers, and may discard useful features with small but reproducible changes. A prevalence threshold can exclude rare features relevant to a targeted biological question. Quality-control replicates may not represent all sources of variation in study samples. No single cutoff is universally optimal, and filtering cannot correct poor measurement, inappropriate sample matching, or confounding. Preserve a record of the criteria and consider sensitivity to reasonable alternatives.

Common Pitfall

A common mistake is to treat every recorded zero as a failed detection and then apply one prevalence cutoff across all blocks. The reasoning is that zero means “not measured,” so low nonzero frequency must indicate an uninformative feature. The correction is to establish what zero represents for each assay before defining detection. A zero may be a genuine measured value, a censored value, or a placeholder for missingness; these meanings imply different prevalence calculations. Likewise, a cutoff appropriate for one assay does not automatically transfer to another.

Another mistake is to describe a feature that fails a filter as biologically irrelevant. Filtering establishes only that the feature did not meet the specified operational criterion in the analyzed data. It does not prove the feature has no biological role or no association with an outcome.