Why Missingness Needs Its Own Description
A cell in an omics table may be blank, coded as NA, or assigned a special value. Those entries can arise for quite different reasons. A feature might not be detected by the assay, a measurement might fail, or the study design might mean that the feature was never measured for that sample. Treating all three situations as the same kind of absence obscures what the table represents.
This tutorial concerns missing entries within one assay table. It describes where the entries occur and distinguishes several plausible causes. The earlier material on biological zeros, detection limits, and censored signals is relevant: an observed zero or a signal known only to be below a limit is not automatically an ordinary missing value. The recorded value and its measurement status must be interpreted together.
Here, \(i=1,\ldots,n\) indexes samples and \(j=1,\ldots,p\) indexes features. A feature is the assay-specific measurement target represented by one column; a sample is the biological or experimental unit represented by one row. This convention describes a sample-by-feature matrix, not the physical orientation of a spreadsheet. To use these indicators correctly, the table must have a consistent definition of “missing,” and entries treated as observed must actually be measurements rather than placeholders such as -999 or a text code that software interprets as a value.
Describing Where Missing Cells Occur
A feature-level pattern asks whether one or more features are frequently missing across samples. For example, a particular metabolite feature may be missing in most samples. A sample-level pattern asks whether one or more samples have many missing features. A sample with few usable measurements across the assay may reflect an issue that affects that sample broadly. These are descriptions of the table, not conclusions about the cause.
Missing cells can also be scattered, with isolated gaps across the table, or clustered into blocks. A block might involve a group of features in a group of samples. Such a shape can be informative when compared with run, plate, batch, or design metadata, but the shape alone cannot establish a technical or biological explanation. In particular, a block is not proof of a batch effect, and a feature missing in many samples is not proof that the feature is biologically absent.
Each rate is a proportion of cells in its stated denominator. Multiplying by 100 expresses it as a percentage. A rate does not express the size of a measured signal or the probability that a missing value has any particular value. It is a descriptive summary of the recorded table. The overall rate can conceal important structure: the same overall percentage may come from a few nearly empty samples, a few rarely detected features, or a scattered pattern.
Worked Example: Comparing Sample-Level and Feature-Level Rates
Suppose a hypothetical assay records four features for three samples. The entry “NA” marks a missing measurement; all numbers, including zero, are recorded measurements.
| Sample | Feature A | Feature B | Feature C | Feature D |
|---|---|---|---|---|
| S1 | 2.1 | NA | 0 | 4.0 |
| S2 | NA | NA | 1.2 | 3.1 |
| S3 | 2.4 | 0.8 | NA | 3.8 |
For S1, one of four cells is missing, so \(r_1^{(\mathrm{sample})}=1/4=0.25\), or 25%. For S2, two of four are missing, so \(r_2^{(\mathrm{sample})}=2/4=0.50\), or 50%. For S3, one of four is missing, so \(r_3^{(\mathrm{sample})}=1/4=0.25\), or 25%.
Feature A is missing once among three samples, giving \(r_A^{(\mathrm{feature})}=1/3=0.333\), or 33.3%. Feature B is missing twice, giving \(2/3=66.7\%\). Feature C is missing once, giving \(1/3=33.3\%\), and Feature D has no missing cells, giving \(0/3=0\%\).
There are four missing cells among the \(3\times4=12\) total cells. Therefore, the overall rate is \(4/12=0.333\), or 33.3%. The table shows both a sample-level concentration in S2 and a feature-level concentration in Feature B. The two zeros are recorded values, so they do not contribute to any missingness count under the stated encoding.
The overall rate is a weighted average of the sample-level rates when every sample has the same number \(p\) of feature cells:
The equality follows by grouping the same sum of missing indicators first by sample and then dividing by the number of cells. With a complete rectangular table, each sample contributes the same number of cells. If some cells were never part of the assay design, they must be handled explicitly rather than silently counted in a denominator as though they were eligible measurements.
Three Different Reasons for an Unavailable Cell
A useful first description separates missing entries by their recorded or documented status. These categories are operational: they refer to why the measurement is unavailable, not to the numerical value the measurement might have taken.
- Technical non-detection: the assay was attempted, but the target did not produce a signal that met the detection or reporting criteria. For example, a mass-spectrometry feature may be below a platform’s detection threshold. Depending on the recording convention, the data may contain a missing code, a censored value, or another explicit below-limit status. Do not assume these encodings are interchangeable.
- Failed measurement: a measurement was intended, but the attempt did not yield a usable result because of an operational failure, such as a failed library, instrument interruption, poor sample quality, or failed assay control. A failure code or laboratory record may distinguish this from a successful attempt with no detected signal.
- Not recorded by design: the feature was not measured for that sample because of the protocol, panel, allocation, or another planned feature of data collection. This is not the same as attempting a measurement and failing to detect the target. The cell may be better represented as “not applicable” or “not assayed” than as an ordinary missing measurement.
A fourth practical possibility is unknown provenance: a cell is blank, but the available table and records do not establish which of these explanations applies. It is more accurate to preserve that uncertainty than to assign a cause based only on the visual pattern. In particular, “missing” describes the table state; “non-detected,” “failed,” and “not assayed” are claims about how that state arose.
Worked Example: Separating Cause Labels From Cell Status
A hypothetical targeted assay contains 10 samples and 5 planned features, so there are 50 intended sample-feature measurements. Laboratory records show that two cells were not assayed because a panel was not run for those samples; three attempted measurements failed quality control; and four attempted measurements had no reportable signal. The remaining cells have usable measurements.
If all three statuses were stored as NA, the table has \(2+3+4=9\) missing cells, and its overall missingness rate relative to all intended measurements is \(9/50=0.18\), or 18.0%. But the cause breakdown is different: design-based omissions are \(2/50=4.0\%\), failed measurements are \(3/50=6.0\%\), and technical non-detections are \(4/50=8.0\%\).
The arithmetic is consistent because the three documented categories are mutually exclusive in this example and together account for all nine missing cells. If the two not-assayed cells were instead excluded from the denominator, the rate among the 48 cells that were actually assayed would be \(7/48=14.6\%\). Both rates can be meaningful, but they answer different questions: 18.0% of intended cells were unavailable, while 14.6% of attempted cells did not yield usable measurements. A report should state which denominator is used.
Patterns Do Not Prove Causes
The same visible pattern can arise from different processes. A feature missing in many samples might be difficult to detect, absent from a subset of panels, or affected by a problem in recording or processing. A sample with many missing entries might have poor overall measurement quality, or the table may combine features that were not all planned for that sample. Without records about assay attempts and design, the matrix alone cannot distinguish these explanations.
This is why the cause categories should not be confused with statistical missing-data mechanisms. In standard terminology, MCAR means that the probability a value is missing does not depend on observed or unobserved data; MAR means that it may depend on observed data but, conditional on those data, not on the missing value itself; MNAR means that missingness can still depend on the unobserved value after conditioning on observed data. These labels describe probability relationships, not laboratory events. For example, “technical non-detection” does not by itself establish MNAR, and “failed measurement” does not by itself establish MCAR. Establishing a mechanism requires more information and assumptions than a missingness map supplies.
Worked Example: A Design-Based Block
A hypothetical assay table has six samples and four features. Features A and B were measured for all six samples. Features C and D were included only for samples S1 through S3; the protocol specifies that C and D were not assayed for S4 through S6.
The table contains \(6\times4=24\) cells, of which the six cells for C and D in S4 through S6 are not assayed by design. If those are all coded NA, a naive table-wide rate is \(6/24=25.0\%\). But relative to the 18 measurements planned under the design, the design-based omission rate is \(0/18=0\%\): all planned measurements were obtained. The 25.0% figure describes unavailable cells in a rectangular display, not assay failure.
If the six not-assayed cells are excluded from the eligible denominator, the usable-measurement rate among planned cells is \(18/18=100\%\). The correct description should retain both facts: the displayed matrix has six unavailable cells because the design did not call for those measurements, and no planned measurement is missing. Treating the six entries as failed assays would misrepresent the protocol.
Practical Comparisons, Checks, and Limitations
There are several reasonable ways to summarize missingness, but they are not substitutes for one another. An overall percentage is compact, yet can hide clustering. Sample-level rates highlight samples with many unavailable entries, while feature-level rates reveal features with sparse measurement coverage. A binary missingness map displays location and blocks; it does not by itself summarize severity or identify causes. Cause-labelled counts add context when records support those labels, but they depend on consistent documentation. For a rectangular table with a common set of planned cells, the summaries can be reported together. For incomplete or varying panels, define the eligible denominator for each rate.
A basic audit should check that missing-value codes are interpreted consistently, that zeros and below-limit statuses are not automatically converted to ordinary missing values, and that design-based cells are identified. Counts should agree: the sum of missing cells by sample must equal the sum by feature, and both must equal the total number of cells classified as missing. These checks catch bookkeeping errors; they do not determine the underlying missingness mechanism. Detailed diagnosis of feature-table patterns is a separate task.
Several limitations remain. A summary rate does not describe the distribution of observed measurements, quantify uncertainty about a missing value, or tell how a later analysis should handle it. A documented status can still be incomplete or inaccurate. Also, the same feature-level or sample-level pattern may have multiple explanations. Therefore, report patterns and known causes separately, use “unknown” when appropriate, and avoid causal claims based solely on the table’s shape.
Common Mistake: Treating Every Blank as the Same Event
Incorrect reasoning: “Every blank is a non-detection, so I can count all blanks as evidence that the feature was below the detection limit.”
Correction: A blank may indicate non-detection, a failed attempt, a measurement not planned by design, or unknown provenance. Check the data dictionary and assay records before assigning a cause. A recorded zero or an explicitly censored below-limit value also should not be recoded as an ordinary missing cell without justification.
Check Your Understanding
Answer each question, then compare with the short solutions.
- A table has 8 samples and 5 features, with 10 cells coded missing. What is the overall missingness rate?
- A feature is blank in 7 of 10 samples. What does its feature-level missingness rate establish, and what does it not establish?
- For one sample, a feature was never included in the planned panel. Should this automatically be called a failed measurement? Explain.
- An observed zero is recorded for a cell, and the data dictionary defines zero as a measured value. Does it contribute to the ordinary missingness count?
- Explain why “technical non-detection” and “MNAR” are not interchangeable labels.
Solutions. (1) There are \(8\times5=40\) cells, so the rate is \(10/40=0.25\), or 25.0%. (2) Its feature-level rate is \(7/10=70.0\%\). This establishes how often that feature is missing in the table, not why it is missing or what its unobserved values would have been. (3) No. It was not assayed by design, rather than attempted and failed; its status and denominator should reflect the plan. (4) No. Under the stated encoding it is an observed value, so its missingness indicator is zero. (5) Technical non-detection describes an assay event; MNAR describes a probability relationship between missingness and an unobserved value after conditioning on observed data. The event label alone does not establish that relationship.