Why Visualize Missingness Patterns?
A count of missing values in each variable is useful, but it does not show whether the same records are missing several values together. Two data sets can have identical variable-level missing totals while having very different patterns: missing values may be scattered one at a time, or they may repeatedly occur together in particular fields. A pattern display makes that distinction visible.
For predictive modeling, recurring patterns matter because they show which combinations of fields are often unavailable on the same records. A display is a descriptive tool: it helps identify and communicate the structure of the observed gaps. By itself, it does not establish why values are missing or what method should be used to handle them.
A useful pattern summary lists each distinct sequence and the number of records with that sequence. A pattern such as 1000 is an exact pattern: only the first variable is missing. By contrast, “missing both the first and second variables” is an inclusive condition; it includes any pattern with 1 in both of those positions, even if additional variables are missing.
What a Pattern Display Shows
Two complementary displays are especially useful. A missingness matrix shows records as rows and variables as columns, marking each cell as observed or missing. It retains record-level detail and can reveal blocks or repeated combinations, particularly when records with the same pattern are placed together. A pattern-frequency display summarizes how many records have each distinct pattern, often with a bar for each pattern. It makes common and rare combinations easier to compare.
A pattern-frequency display should not be confused with a variable-level count display. Pattern counts answer “how many records have this exact combination?” Variable-level counts answer “how many records are missing this variable, regardless of the other variables?” Both views are useful, but they answer different questions. The variable ordering and the meaning of the missing-value mark should be clear in any matrix; the pattern labels and their frequencies should be legible in a summary plot.
| Display | What each row or mark represents | Best used to inspect |
|---|---|---|
| Missingness matrix | A record-by-variable cell marked observed or missing | Record-level blocks, repeated combinations, and isolated gaps |
| Pattern-frequency display | One distinct missingness pattern and its record count | Which combinations recur and how common they are |
| Variable-level summary | A variable and its missing count or proportion | Which individual fields have the most missing values |
Relating Pattern Counts to Missing-Cell Totals
Pattern counts can be checked against the missing totals for each variable. Let \(m=(m_1,\ldots,m_p)\) be a pattern, where each \(m_j\) is 0 or 1, and let \(n_m\) be the number of records with that pattern. The number of missing cells contributed by each record with pattern \(m\) is the sum of its binary entries.
The first expression adds the number of missing variables within each pattern, multiplied by how many records have that pattern. To see why it equals the second expression, expand the first sum: each product \(n_m m_j\) contributes once for every variable \(j\) marked missing in pattern \(m\). Regrouping those same terms by variable gives the second expression, the sum of the missing counts for each variable. This is a useful arithmetic check on a pattern table.
The number of records with at least one missing value is different from the total number of missing cells. A record missing three fields contributes one to the former and three to the latter. Confusing those quantities can make a pattern summary appear inconsistent even when its counts are correct.
Worked Example: Separating Isolated and Recurring Gaps
A hypothetical portfolio extract records four fields in this order: driver age (A), claim type (B), vehicle age (C), and annual mileage (D). The pattern summary is 0000: 3 records, 1000: 2, 0100: 1, 1100: 3, 1010: 2, and 0011: 1. Find the number of incomplete records, the missing count for each field, and the total number of missing cells.
Method. Read each binary pattern in A, B, C, D order. First add the pattern frequencies to get the number of records. Then count the 1s in each position, weighting each pattern by its frequency.
There are \(3+2+1+3+2+1=12\) records. The 0000 pattern accounts for three complete records. The remaining pattern counts sum to \(2+1+3+2+1=9\), so nine records have at least one missing value.
For A, the patterns with a 1 in the first position are 1000, 1100, and 1010. Thus A is missing for \(2+3+2=7\) records. For B, the relevant patterns are 0100 and 1100, giving \(1+3=4\). For C, the relevant patterns are 1010 and 0011, giving \(2+1=3\). For D, only 0011 has a 1 in the fourth position, giving 1.
The total missing-cell count from the variable margins is \(7+4+3+1=15\). Check it by pattern: 1000 contributes \(2\times1=2\), 0100 contributes \(1\times1=1\), 1100 contributes \(3\times2=6\), 1010 contributes \(2\times2=4\), and 0011 contributes \(1\times2=2\). Their sum is \(2+1+6+4+2=15\). The checks agree. The 1100 pattern specifically means three records are missing both A and B and have observed values for C and D; it does not count every record missing A and B if other patterns allowed additional missing fields.
Reading Frequencies Alongside Marginal Counts
A common display-reading task is to identify whether missing values tend to occur alone or in combination. Consider a pattern table with three variables, in the order A, B, C. A pattern-frequency display can show the number of records in each bar, while a separate variable-level summary can show the marginal missing counts. A large bar for a pattern such as 110 indicates repeated co-occurrence of missing A and B, but it does not by itself say how many records are missing A in total: other patterns may also have A marked missing.
Worked Example: Comparing Exact Patterns With Variable Totals
A hypothetical application system records address (A), occupation (B), and requested coverage (C) for 20 applicants. The pattern counts are 000: 8, 100: 4, 010: 3, 001: 2, 110: 2, and 011: 1. Interpret the most frequent incomplete pattern, calculate missing proportions by variable, and calculate the overall incomplete-record proportion.
The most frequent incomplete pattern is 100, with 4 records. It means address alone is missing for those four records. The next most frequent incomplete pattern is 010, with 3 records; the 110 pattern occurs for 2 records and represents address and occupation both missing, with coverage observed.
Address is missing in patterns 100 and 110, so its count is \(4+2=6\), and its proportion is \(6/20=0.30\), or 30%. Occupation is missing in 010, 110, and 011, so its count is \(3+2+1=6\), also 30%. Coverage is missing in 001 and 011, so its count is \(2+1=3\), or \(3/20=0.15\), which is 15%.
There are \(4+3+2+2+1=12\) incomplete records, out of 20, so the incomplete-record proportion is \(12/20=0.60\), or 60%. The total missing-cell count is \(6+6+3=15\). The pattern calculation gives the same result: \(4+3+2+2(2)+1(2)=15\), where the first three terms are the one-missing-variable patterns and the last two terms account for patterns 110 and 011, each with two missing variables. The arithmetic confirms that the pattern summary and the margins are consistent.
Building a Pattern Summary in R
In R, the key step is to apply is.na to the variables of interest, then summarize the resulting row patterns. For a data frame called dat, the selected column order should be explicit. For example, use vars <- c("A", "B", "C"), then M <- is.na(dat[vars]). Each row of M is a set of TRUE/FALSE indicators, with TRUE meaning missing. Convert those indicators to 1 and 0 and join them in the stated column order to create pattern labels. A frequency table of those labels gives the number of records for each exact pattern.
A simple base-R construction is patterns <- apply(M, 1, function(z) paste(as.integer(z), collapse = "")), followed by table(patterns). The function converts TRUE to 1 and FALSE to 0. The result might list labels such as 000, 100, or 011; those labels only make sense when the variable order is also reported. A missingness matrix can be built from the same logical matrix: its rows are records and its columns are variables.
For a readable display, order patterns consistently, show counts rather than relying only on bar height, and distinguish “no missing values” from the other patterns. If there are many distinct patterns, a long tail of rare patterns can make a plot crowded. A compact frequency table alongside the plot may help, but rare patterns should not silently be combined with common ones when their combinations are relevant to interpretation.
Worked Example: Checking an R Pattern Summary by Hand
A data frame contains eight hypothetical records and three ordered fields, A, B, and C. Their missingness patterns, in record order, are 000, 100, 100, 010, 110, 110, 001, and 011. State the frequency summary, the variable-level missing counts, and the total missing-cell count. Then explain what an R pattern table should report.
Method. Tally identical strings for the exact-pattern summary, then count the 1s in each position for the variable margins.
The exact pattern counts are 000: 1, 100: 2, 010: 1, 110: 2, 001: 1, and 011: 1. They sum to \(1+2+1+2+1+1=8\) records. Only the 000 record is complete, so \(8-1=7\) records are incomplete.
For A, patterns 100 and 110 contribute \(2+2=4\) missing values. For B, patterns 010, 110, and 011 contribute \(1+2+1=4\). For C, patterns 001 and 011 contribute \(1+1=2\). The margins therefore give \(4+4+2=10\) missing cells.
The pattern check gives 100: \(2\times1=2\), 010: \(1\times1=1\), 110: \(2\times2=4\), 001: \(1\times1=1\), and 011: \(1\times2=2\). Thus \(2+1+4+1+2=10\), agreeing with the margins. In R, the pattern table should report those six labels and frequencies, subject only to its display order. A result that includes a seventh record as complete, or reports a different total frequency than eight, would indicate that the selected columns, row handling, or pattern construction needs checking.
Common Interpretation Traps
- Confusing an exact pattern with an inclusive condition. Pattern 110 in three variables means A and B are missing and C is observed. “Both A and B are missing” also includes patterns such as 111.
- Adding pattern frequencies as if they were missing cells. A pattern with two 1s contributes two missing cells per record, not one. Use the number of 1s in the pattern as its weight.
- Reading a marginal count from one bar. The total missing count for a variable is obtained from every pattern with a 1 in that variable’s position.
- Omitting the variable order. The string 101 has no stable interpretation unless the order of its three variables is known.
- Treating a visual pattern as an explanation. A display describes which fields are missing together. It does not establish the process or reason behind that co-occurrence.
These distinctions support careful data exploration and are aligned with the SOA Exam PA (Predictive Analytics) syllabus. A clear analysis states the variable order, separates record counts from missing-cell totals, and checks pattern-derived totals against the variable margins. The display then provides evidence about recurring combinations without claiming more than the observed data show.
Check Your Understanding
For each problem, use the variable order stated and distinguish incomplete records from missing cells.
- A three-field form has patterns 000: 5, 100: 2, 010: 1, 110: 2, and 001: 3, in A, B, C order. How many records are incomplete, and what is the total missing-cell count? (Answer: 8 incomplete records; 10 missing cells.)
- For four ordered fields A, B, C, D, the pattern counts are 0000: 4, 1000: 3, 0100: 2, 0010: 1, 0001: 2, 1100: 2, and 0111: 1. How many records are incomplete, and how many missing cells occur in total? (Answer: 11 incomplete records; 15 missing cells.)
- For A, B, C, the pattern counts are 000: 6, 100: 2, 010: 2, 001: 1, and 101: 2. How many records have both A and C missing, and what is the total missing-cell count? (Answer: 2 records; 9 missing cells.)
- Ten records have patterns 0000: 2, 1000: 2, 0100: 1, 1100: 3, and 0011: 2, in A, B, C, D order. How many records are missing both A and B, and what is the total missing-cell count? (Answer: 3 records; 13 missing cells.)
- A 16-record file has three-variable pattern counts 000: 8, 100: 4, 010: 2, and 111: 2. What proportion of records are incomplete, and what proportion of records have the exact pattern 111? (Answer: 0.5000 incomplete; 0.1250 have pattern 111.)