Why Map Missingness?
Before fitting a regression model, it helps to know where the gaps in the data occur. A total such as “12% of values are missing” can hide important differences: perhaps most gaps are in one predictor, perhaps outcomes and predictors are missing together, or perhaps gaps occur in several distinct combinations. A missingness pattern table makes these arrangements visible and countable.
The purpose here is descriptive: to summarize which variables are affected and which values are absent together in the same records. A pattern table does not, by itself, tell us why values are missing or establish which missing-data model is appropriate. Those are separate questions.
For example, if the variables are ordered as outcome \(Y\), predictor \(X_1\), and predictor \(X_2\), the pattern \(010\) means that \(Y\) is observed, \(X_1\) is missing, and \(X_2\) is observed. The pattern \(000\) means all three values are observed. State the order in a table heading or note: without it, a string such as \(010\) cannot be interpreted reliably.
From Patterns to Variable-Level Summaries
Suppose there are \(n\) records and \(p\) variables. Let \(c_r\) be the number of records with pattern \(r\), where each pattern is a length-\(p\) sequence of zeros and ones. The pattern counts must add to \(n\), because each record contributes to exactly one pattern.
For variable \(j\), its missing count is the sum of the counts for every pattern whose \(j\)th indicator is 1. Its missing percentage uses all \(n\) records as the denominator. Thus the percentage describes how many records have that variable missing, not the proportion of the nonmissing values that are absent.
Here \(R_j(r)\) is the indicator in position \(j\) of pattern \(r\). Because the patterns partition the records, this sum counts each record with a missing value for variable \(j\) once, and counts every other record zero times.
Worked Example: Variable Counts From a Four-Variable Pattern Table
A hypothetical regression data set contains \(n=50\) records. The variables are ordered \(Y, X_1, X_2, X_3\), where \(Y\) is the outcome and the \(X\)'s are predictors. A 1 marks a missing value. The observed pattern frequencies are:
| Pattern \(Y,X_1,X_2,X_3\) | Number of records |
|---|---|
| 0000 | 30 |
| 1000 | 4 |
| 0100 | 4 |
| 0010 | 3 |
| 0001 | 2 |
| 1100 | 2 |
| 1010 | 1 |
| 0110 | 2 |
| 0101 | 1 |
| 1001 | 1 |
First check the total: \(30+4+4+3+2+2+1+2+1+1=50\), so the listed patterns account for all records.
For \(Y\), add the frequencies whose first indicator is 1: patterns 1000, 1100, 1010, and 1001. The missing count is \(4+2+1+1=8\), so the missing percentage is \(100(8/50)=16.0\%\).
For \(X_1\), add the frequencies whose second indicator is 1: patterns 0100, 1100, 0110, and 0101. The missing count is \(4+2+2+1=9\), and the missing percentage is \(100(9/50)=18.0\%\).
For \(X_2\), the relevant patterns are 0010, 1010, and 0110. The missing count is \(3+1+2=6\), giving \(100(6/50)=12.0\%\). For \(X_3\), the relevant patterns are 0001, 0101, and 1001. The count is \(2+1+1=4\), giving \(100(4/50)=8.0\%\).
Thus \(X_1\) has the largest missing percentage among these four variables, at 18.0%, followed by \(Y\) at 16.0%, \(X_2\) at 12.0%, and \(X_3\) at 8.0%. These are descriptive summaries for this hypothetical data set; they do not explain why any values are missing.
Co-Occurrence: Which Variables Are Missing Together?
A variable-level count answers how often one variable is missing. It does not say which other variables are missing in those same records. To summarize a pair, count records whose pattern has a 1 in both variables’ positions. Divide by \(n\) to express that count as a percentage of all records, if a percentage is useful.
The product \(R_j(r)R_k(r)\) equals 1 only when both indicators are 1. It is 0 if either value is observed. Therefore, the sum counts records with both variables missing, not records with either variable missing. This distinction matters: “both” and “at least one” describe different sets of records.
The pairwise co-occurrence count is not generally the sum of the two marginal missing counts. A record in which both variables are missing contributes once to each marginal count and once to their co-occurrence count. A record in which only one is missing contributes to just that variable’s marginal count.
Worked Example: Finding Pairs Missing Together
Use the four-variable table above, with order \(Y, X_1, X_2, X_3\). To count records where both \(Y\) and \(X_1\) are missing, look for patterns beginning \(11\): 1100 has frequency 2, and none of the other listed patterns begins \(11\). Therefore, \(m_{Y,X_1}=2\), or \(100(2/50)=4.0\%\) of all records.
For \(X_1\) and \(X_2\), the pattern must have a 1 in the second and third positions. Pattern 0110 has frequency 2, and no other listed pattern has both indicators equal to 1. Thus \(m_{X_1,X_2}=2\), again \(4.0\%\) of all records.
For \(X_1\) and \(X_3\), pattern 0101 has frequency 1, so \(m_{X_1,X_3}=1\), or \(100(1/50)=2.0\%\). For \(X_2\) and \(X_3\), no listed pattern has both the third and fourth indicators equal to 1, so the co-occurrence count is 0.
A compact table can collect all pairwise counts. The diagonal entries show each variable’s marginal missing count; off-diagonal entries show the number of records with both variables missing.
| Both missing | \(Y\) | \(X_1\) | \(X_2\) | \(X_3\) |
|---|---|---|---|---|
| \(Y\) | 8 | 2 | 1 | 1 |
| \(X_1\) | 2 | 9 | 2 | 1 |
| \(X_2\) | 1 | 2 | 6 | 0 |
| \(X_3\) | 1 | 1 | 0 | 4 |
The table shows, for example, that \(X_1\) is missing in 9 records, but is missing alongside \(Y\) in only 2 of them. It also shows that no record in this data set has both \(X_2\) and \(X_3\) missing. The pairwise table does not retain the full pattern structure: different sets of records can produce the same pairwise counts while differing in how three or more missing values combine.
Reading Full Patterns and Pairwise Tables Together
A full pattern table and a pairwise table serve related but different purposes. The pattern table preserves the entire combination of missing and observed values for every distinct pattern. The pairwise table summarizes one pair at a time. A useful first pass is to check the pattern frequencies, calculate each variable’s marginal count and percentage, and then examine co-occurrence for pairs relevant to the regression model.
The all-zero pattern is especially easy to locate: it counts records for which all listed variables are observed. Other patterns can identify combinations that a marginal summary conceals. For instance, two variables can have similar missing percentages even though the missing values occur in quite different records.
When there are many variables, a pattern table may contain many possible combinations, though only patterns that actually occur need to be listed. A long table can be summarized or sorted by frequency, but the variable order and the meaning of 0 and 1 must remain explicit. A useful report also states the total number of records and the denominator used for percentages.
Worked Example: Identifying Affected Variables and Combinations
Consider a second hypothetical regression data set with \(n=20\) records. Its variables are ordered as outcome \(Y\), predictor \(A\), and predictor \(B\). The pattern frequencies are:
| Pattern \(Y,A,B\) | Number of records |
|---|---|
| 000 | 11 |
| 100 | 3 |
| 010 | 2 |
| 001 | 1 |
| 110 | 1 |
| 101 | 1 |
| 011 | 1 |
The frequencies sum to \(11+3+2+1+1+1+1=20\). For \(Y\), patterns 100, 110, and 101 contribute, giving \(3+1+1=5\) missing outcomes and \(100(5/20)=25.0\%\). For \(A\), patterns 010, 110, and 011 contribute, giving \(2+1+1=4\) missing values and \(100(4/20)=20.0\%\). For \(B\), patterns 001, 101, and 011 contribute, giving \(1+1+1=3\) missing values and \(100(3/20)=15.0\%\).
The pair \(Y,A\) is missing together in pattern 110, which occurs once. The pair \(Y,B\) occurs together in pattern 101, also once; \(A,B\) occurs together in pattern 011, also once. Pattern 111 does not occur, so no record has all three variables missing. The table therefore identifies that every variable is affected, and that each pair co-occurs once, while no record has all three missing.
This is a map of the recorded gaps, not a conclusion about how to handle them. It also does not establish that one variable’s missing values caused another variable’s values to be missing. Those interpretations require information beyond the pattern counts.
A Short Justification for the Counting Rules
Each record belongs to exactly one pattern because its indicators have one definite value, 0 or 1, in every variable position. The patterns are therefore mutually exclusive and collectively include all records. For a fixed variable \(j\), consider its indicator \(R_{ij}\) in each record \(i\). Adding these indicators gives 1 for every record where variable \(j\) is missing and 0 for every record where it is observed. Thus the total of these indicators is exactly the missing count.
The first sum adds record-level indicators. The second groups records by their patterns: every record in pattern \(r\) has the same indicator \(R_j(r)\), and there are \(c_r\) such records. Dividing \(m_j\) by \(n\) gives the fraction of all records with the variable missing; multiplying by 100 expresses that fraction as a percentage.
For a pair, the product \(R_{ij}R_{ik}\) is 1 precisely when both values are missing for record \(i\). Summing those products therefore counts the intersection of the two missingness sets. Grouping by pattern gives the equivalent frequency calculation:
These identities explain why the pattern table can be used to calculate both marginal and pairwise summaries without returning to the original records, provided the pattern frequencies and variable order are correct.
Common Mistakes and Practical Checks
- Reversing the indicator convention. Here 1 means missing and 0 means observed. If a data system uses the reverse convention, recode or state it clearly before interpreting patterns.
- Using the wrong variable order. A pattern is meaningful only relative to its stated order. Confirm the order before adding frequencies for any variable.
- Leaving out a pattern when calculating a marginal count. Include every pattern with a 1 in that variable’s position, not just patterns where that is the only missing value.
- Confusing “both missing” with “either missing.” The product of two indicators counts records with both missing. It does not count every record with at least one missing.
- Using the wrong percentage denominator. For the missing percentage in a variable, divide by the number of records in the data set being described. State that denominator.
- Treating a pattern table as an explanation. It describes which recorded values are absent together. Counts alone do not identify the reason for missingness or prove that the pattern is harmless.
Check Your Understanding
Use the definitions and counting rules to answer these questions.
- With variables ordered \(Y,X_1,X_2\), what does pattern 101 mean?
- A pattern table lists 000 with frequency 18, 100 with frequency 4, 010 with frequency 3, and 110 with frequency 2. How many records are represented, and how many have \(X_1\) missing?
- Using the same table, calculate the missing percentage for \(Y\) using all represented records as the denominator.
- In that table, how many records have both \(Y\) and \(X_1\) missing? Which pattern contributes to that count?
- Why does a pairwise co-occurrence table not necessarily give the same detail as the full pattern table?