Why Missingness Needs Its Own Notation
A dataset with blanks or special codes does not directly show the values that were not recorded. It is useful to distinguish two things: the values that conceptually belong to the units being studied, and the information about which of those values is actually available. This distinction is foundational in missing-data work. It lets us describe a dataset precisely without treating an unrecorded value as though it were zero or as though it did not exist.
The notation in this tutorial is deliberately descriptive. A response indicator tells us whether a value is observed; by itself, it does not explain why that value is missing or what statistical assumptions about missingness are appropriate. Those questions require additional concepts.
“Complete data” is a conceptual reference: it describes the values for all units and variables in the analysis, including values that are unavailable in the recorded dataset. The values need not be stored in a file, and a researcher must not pretend to know an unobserved value. The notation simply provides a way to distinguish the value of interest from whether it was observed.
An indicator is a status label, not a measurement of the variable itself. For one variable, \(R_{ij}=1\) means that its value is available in the recorded data, while \(R_{ij}=0\) means it is unavailable. The actual recorded dataset therefore contains values where \(R_{ij}=1\), together with the status information that indicates which cells are missing.
From a Data Table to Indicators
Consider the following hypothetical dataset. A dot indicates a missing entry; it is a display convention here, not a data value.
| Unit \(i\) | Age \(Y_{i1}\) (years) | Lab value \(Y_{i2}\) (units) | Response status for lab \(R_{i2}\) |
|---|---|---|---|
| 1 | 34 | 5.2 | 1 |
| 2 | 51 | . | 0 |
| 3 | . | 4.8 | 1 |
| 4 | 42 | . | 0 |
There are four units and two variables. In the complete-data notation, \(Y_{22}\) denotes the lab value for unit 2, and \(Y_{31}\) denotes the age for unit 3, even though neither value is shown in the recorded table. The indicators make the status of each cell explicit: \(R_{22}=0\), \(R_{32}=1\), and \(R_{31}=0\). The table also shows that unit 1 has \(R_{11}=1\) and unit 4 has \(R_{41}=1\). A dot in the table is not a substitute for an indicator; it is a visual way to show that the indicator is zero.
Worked Example: Marking Missing Cells in a Small Dataset
In a hypothetical dataset, the complete-data variables are age and a symptom score for three participants. The recorded rows are: participant 1, age 28 and score 6; participant 2, age 35 and score missing; participant 3, age missing and score 4. Use \(Y_{ij}\) and \(R_{ij}\) to describe the status of the cells.
Let \(j=1\) denote age and \(j=2\) denote symptom score. For participant 1, both values are recorded, so \(R_{11}=1\) and \(R_{12}=1\). For participant 2, age is recorded but the score is missing, so \(R_{21}=1\) and \(R_{22}=0\). For participant 3, age is missing and the score is recorded, so \(R_{31}=0\) and \(R_{32}=1\).
In complete-data notation, \(Y_{22}\) is still the complete-data symptom score for participant 2, and \(Y_{31}\) is still the complete-data age for participant 3. Their numerical values are not supplied by the recorded data. The indicators express the observation status, not the values themselves.
Observed and Missing Parts of the Data
The indicators can be used to define which index pairs correspond to observed cells and which correspond to missing cells. An index pair \((i,j)\) identifies one unit-variable cell. The observed and missing index sets are
These sets divide the \(np\) cells into two non-overlapping groups: every cell is either observed or missing. Their sizes follow directly from the indicator values. For variable \(j\), the number observed is the sum of its indicators across units, because each observed cell contributes one and each missing cell contributes zero:
The second equality holds because there are \(n\) cells for variable \(j\), and each is either observed or missing. Across all variables, the total numbers are \(\sum_{i=1}^{n}\sum_{j=1}^{p}R_{ij}\) observed cells and \(np-\sum_{i=1}^{n}\sum_{j=1}^{p}R_{ij}\) missing cells. These are counts, not assumptions about how the missing values arose.
Worked Example: Counting Observed Values by Variable
Use the four-unit table above. For age, the indicators are \(R_{11}=1\), \(R_{21}=1\), \(R_{31}=0\), and \(R_{41}=1\). For the lab value, they are \(R_{12}=1\), \(R_{22}=0\), \(R_{32}=1\), and \(R_{42}=0\). Calculate the observed and missing counts for each variable.
For age, the observed count is \(n_{\mathrm{obs},1}=1+1+0+1=3\). There are four age cells, so \(n_{\mathrm{mis},1}=4-3=1\). Thus age is observed for three units and missing for one unit.
For the lab value, the observed count is \(n_{\mathrm{obs},2}=1+0+1+0=2\). The missing count is \(n_{\mathrm{mis},2}=4-2=2\). There are five observed cells in total and three missing cells in total across the two variables: \(3+2=5\) and \(1+2=3\). Together these counts account for all \(4\times2=8\) cells.
The missing fraction for the lab variable in this dataset is \(2/4=0.50\), or 50%. This is a description of these four records, not a claim about a larger population.
Row Patterns and Cell-Level Status
For multiple variables, one unit can have several observation patterns. In the four-unit table, the lab indicator vector is \(R_2=(1,0)\) for unit 2: age is observed and the lab value is missing. For unit 3, \(R_3=(0,1)\): age is missing and the lab value is observed. A row-level label such as “incomplete” can summarize whether a unit has at least one missing cell, but it loses the detail carried by the cell-level indicators.
Worked Example: Distinguishing a Row Summary from a Cell Pattern
Suppose a hypothetical study records blood pressure, \(Y_{i1}\), and a questionnaire score, \(Y_{i2}\), for two participants. Participant A has both values, while participant B has blood pressure recorded but the questionnaire score is missing. Write their indicator vectors and explain what a single complete-case flag would retain.
For participant A, both cells are observed, so \(R_A=(1,1)\). For participant B, the first cell is observed and the second is missing, so \(R_B=(1,0)\). A row-level complete-case flag could be defined as 1 when every variable is observed and 0 otherwise. It would therefore be 1 for A and 0 for B.
That row-level flag identifies which participants have complete rows, but it does not itself say which variable is missing for B. The vector \(R_B=(1,0)\) preserves that cell-level information. When different variables have different missingness patterns, the vector is the more informative description.
Assumptions, Checks, and Common Errors
The notation relies on a few basic conditions. First, the analysis specifies the units and variables clearly, so each \(Y_{ij}\) refers to one particular cell. Second, each variable value is meaningful for the unit under consideration. Third, the observed status is recorded correctly: a value counted as available can actually be read or used, and a value counted as missing is not silently treated as observed. Under these conditions, \(R_{ij}\) is a faithful binary description of availability.
A value that is not defined for a unit should not automatically be called missing. For example, a question about current employment may not have a meaningful response for a person outside the question’s target scope, depending on how the study defines its variables and population. “Not applicable” may require separate coding or a refined definition of the analysis variables. The \(R_{ij}=0\) convention describes a missing value in the specified data structure; it does not settle whether every blank in a file is a missing value of the same kind.
- Confusing missing with zero. If \(R_{ij}=0\), the recorded value is unavailable. This does not imply \(Y_{ij}=0\).
- Treating a placeholder as a measurement. Codes such as 999 or “.” may mark missingness in a file, but they are not automatically legitimate values of \(Y_{ij}\). Verify how the data are coded before analysis.
- Using one row indicator when cells differ. A complete-case flag can identify rows with no missing values, but it cannot describe every variable-specific pattern.
- Reading an indicator as an explanation. \(R_{ij}=0\) says that a value was not observed. It does not say why, or what the distribution of that value is.
- Assuming the complete-data value is known. Writing \(Y_{ij}\) for a missing cell is notation for the value in the complete-data representation, not a way to recover that value from the observed records.
A practical check is to compare the indicators with the actual data representation. For each variable, count the cells with \(R_{ij}=1\) and verify that this matches the number of usable recorded values; then confirm that the missing count completes the total of \(n\). For a multivariable dataset, inspect patterns across the indicator vectors rather than relying only on an overall missing percentage.
Choosing a Representation
A table with blank cells or software-specific missing-value markers is convenient for storage and display. Indicator notation is more explicit for mathematical descriptions, because it records status as a binary quantity and applies consistently to every unit-variable pair. A row-level complete-case indicator is a compact summary when the question concerns whether a whole row is complete, but it is not a replacement for the cell-level indicators when the pattern matters.
These representations serve different purposes. The observed-data table shows what is available; the \(R_{ij}\) values formalize availability; and the complete-data notation names the values that would form the full set if every cell were observed. None of these representations, on its own, explains the process that led to missingness. Keeping these roles separate prevents a coding convention from being mistaken for a statistical conclusion.
Check Your Understanding
Answer each question using complete-data notation and the binary response-indicator convention.
- A dataset has five units and one variable. Its indicators are \(1,1,0,1,0\). How many observations are recorded and how many are missing?
- For two variables, a unit has indicator vector \(R_i=(0,1)\). State what is observed and what is missing.
- Explain why a missing value should not be replaced by zero merely because its indicator is zero.
- A dataset has \(n=6\) units. For variable 2, \(\sum_{i=1}^{6}R_{i2}=4\). Find the observed and missing counts for variable 2.
- Why can a row-level complete-case flag be insufficient for describing the missingness pattern in a dataset with several variables?
Solutions. (1) The sum of the indicators is \(1+1+0+1+0=3\), so there are 3 observed values. The missing count is \(5-3=2\). (2) The first variable is missing because \(R_{i1}=0\); the second is observed because \(R_{i2}=1\). (3) \(R_{ij}=0\) records that \(Y_{ij}\) is unavailable, not that its value equals zero. Replacing it with zero would assert a measurement that was not recorded. (4) There are 4 observed values and \(6-4=2\) missing values. (5) The flag reports only whether a row is entirely observed. For an incomplete row, it does not identify which variable or variables are missing; the cell-level vector \(R_i\) does.