Why Compare Missingness by Outcome and Subgroup?
An overall missingness rate can conceal important differences. If income is missing for 15% of a portfolio, that single percentage does not reveal whether missing income is more common among policyholders who renewed, among applicants in a particular region, or among a particular outcome-and-region combination. Comparing rates across observed outcomes and subgroups makes these patterns visible and helps direct data-quality questions.
These comparisons are descriptive diagnostics. A difference may be consistent with a relationship between missingness and an observed variable, but it does not establish why values are missing. In particular, observed rates alone cannot distinguish all possible missingness mechanisms. The interpretation must stay within what the recorded data support.
The denominator is essential. A group’s missingness rate is not its share of all missing records. It is the proportion of that group’s records for which the variable is missing. For example, a large group can contain more missing values simply because it contains more records, even when its missingness rate is lower.
Calculate the Rate, Then Compare Like with Like
Suppose the outcome is observed for a record and takes values such as “renewed” and “not renewed.” To compare missingness of a predictor by outcome, calculate the predictor’s missingness rate separately within each observed outcome category. If the outcome itself is missing, it cannot identify an observed outcome category; the analyst should state how such records are handled rather than silently assigning them to a category.
The rate can be reported as a proportion or percentage. A useful direct comparison is the difference in rates, measured in percentage points. For groups \(g_1\) and \(g_2\), that difference is \(r_{g_1}-r_{g_2}\). A difference of 0.10 is a difference of 10 percentage points, not necessarily a 10% relative increase.
This formula follows directly from the definition of a sample proportion. Each record contributes either 1, if \(X\) is missing, or 0, if it is observed. Adding those indicators counts the missing values. Dividing by the number of eligible records gives the fraction of the group with missing \(X\). The formula applies when the group membership and eligibility rules are specified and the denominator is positive.
Worked Example: Comparing Missing Income by Renewal Outcome
Hypothetical scenario. A review team studies 120 policy records. Renewal outcome is observed on every record. Income is missing for 18 of the 72 records that renewed and for 6 of the 48 records that did not renew. Compare the income-missingness rates by outcome.
Method. Use each outcome group’s own record count as its denominator.
For renewed records, the rate is \(18/72=0.25\), or 25.00%. For records that did not renew, the rate is \(6/48=0.125\), or 12.50%.
The difference is \(25.00\%-12.50\%=12.50\) percentage points. The rate among renewed records is twice the rate among non-renewed records, since \(0.25/0.125=2\).
Check. The groups contain \(72+48=120\) records, and the missing counts total \(18+6=24\). The overall rate is \(24/120=20.00\%\), between the two group rates as expected. The comparison describes an association in these records; it does not show that missing income caused renewal or identify the process that led to missing income.
Common mistake avoided. Dividing 18 by 120 gives 15%, but that is the share of all records that are both renewed and missing income. It is not the missingness rate among renewed records; the correct denominator for that rate is 72.
Look Within Subgroups Before Interpreting a Pooled Difference
A pooled comparison by outcome can reflect differences in subgroup composition. For instance, if outcome groups contain different proportions of regions, and regions have different missingness rates, the overall outcome-specific rates may not describe the pattern within any particular region. Therefore, when a relevant observed subgroup is available, compare rates within that subgroup as well as in the pooled data.
This is a descriptive check, not a guarantee that all relevant differences have been accounted for. The chosen subgroup may not capture every feature related to missingness, and small subgroup counts can make rates unstable. Always show the group counts alongside the rates so the reader can judge how much information each percentage represents.
Worked Example: A Pooled Pattern Changes Across Regions
Hypothetical scenario. An analyst compares missing income by renewal outcome in two regions. In the North, 12 of 40 renewed records and 6 of 60 non-renewed records have missing income. In the South, 2 of 20 renewed records and 16 of 80 non-renewed records have missing income. Calculate the regional rates and the pooled rates.
Within the North: among renewed records, \(12/40=30.00\%\) have missing income. Among non-renewed records, \(6/60=10.00\%\) have missing income. The within-region difference is 20.00 percentage points, higher among renewed records.
Within the South: among renewed records, \(2/20=10.00\%\) have missing income. Among non-renewed records, \(16/80=20.00\%\) have missing income. The within-region difference is \(-10.00\) percentage points, so the rate is higher among non-renewed records.
Pooled renewed records: there are \(40+20=60\) records and \(12+2=14\) with missing income. The pooled rate is \(14/60=23.33\%\). Pooled non-renewed records: there are \(60+80=140\) records and \(6+16=22\) with missing income. The pooled rate is \(22/140=15.71\%\). The pooled difference is \(23.33\%-15.71\%=7.62\) percentage points.
The pooled rate is higher among renewed records, but the within-region comparisons point in opposite directions. This example shows why the pooled comparison alone can obscure subgroup patterns. The arithmetic checks: the pooled group counts are 60 and 140, and the missing counts are 14 and 22, matching the sums across regions.
Common mistake avoided. Reporting only the pooled 7.62-point difference could suggest a consistent pattern across regions. The regional calculations show that this is not the case. Neither the pooled nor the within-region differences, by themselves, prove a missingness mechanism.
Grouped Missingness Summaries in R
In R, is.na() returns TRUE for a missing value and FALSE for an observed value. Because TRUE and FALSE behave as 1 and 0 in a numeric mean, the mean of is.na(x) within a group is the missingness rate for \(x\) in that group. Grouped summaries should report both the number of records and the rate, so denominators are visible.
The following pattern uses dplyr-style grouped operations. Here, outcome is the outcome variable, region is an observed subgroup variable, and income is the predictor being audited. Filtering to observed outcome and region values makes the target population explicit: records with missing outcome or region are not included in these particular group comparisons. The filter does not imply that those records should always be discarded for every analysis.
R code: audit <- df %>% filter(!is.na(outcome), !is.na(region)) %>% group_by(outcome, region) %>% summarise(n = n(), n_missing = sum(is.na(income)), rate = mean(is.na(income)), .groups = "drop").
Since income is not used to filter records, a missing income value remains in its outcome-and-region group and contributes TRUE to is.na(income). That is what the rate calculation requires. If the analysis instead removed records with missing income before calculating the rate, the denominator would exclude the very records being counted and the result would be wrong.
Worked Example: Verify a Grouped R Summary
Hypothetical scenario. A file contains 200 records with observed outcome and product category. For product A, 9 of 30 renewed records and 7 of 70 non-renewed records have missing income. For product B, 2 of 20 renewed records and 16 of 80 non-renewed records have missing income. Calculate the grouped rates and show how the R summary can be checked.
Method. For each product-and-outcome group, divide its missing-income count by its own record count. Product A’s renewed rate is \(9/30=30.00\%\), and its non-renewed rate is \(7/70=10.00\%\). Product B’s renewed rate is \(2/20=10.00\%\), and its non-renewed rate is \(16/80=20.00\%\).
The grouped summary should therefore contain four rows: A/renewed has \(n=30\), \(n_{\text{missing}}=9\), and rate 0.30; A/non-renewed has \(n=70\), \(n_{\text{missing}}=7\), and rate 0.10; B/renewed has \(n=20\), \(n_{\text{missing}}=2\), and rate 0.10; B/non-renewed has \(n=80\), \(n_{\text{missing}}=16\), and rate 0.20.
For this data, the code pattern is: df %>% group_by(outcome, product) %>% summarise(n = n(), n_missing = sum(is.na(income)), rate = mean(is.na(income)), .groups = "drop"). If the variables are factors with unused levels, those levels do not represent observed records; check the output and do not interpret an empty group’s rate as a meaningful estimate.
Check. The group counts total \(30+70+20+80=200\), and the missing counts total \(9+7+2+16=34\). Thus, the overall rate is \(34/200=17.00\%\). The code’s grouped counts and rates should reconcile to these totals. Common mistake avoided: calculating a product’s missing count as a proportion of all 200 records would answer a different question from the missingness rate within a product-and-outcome group.
What Rate Differences Do—and Do Not—Tell You
The missingness assumptions provide a useful vocabulary for describing what a pattern might suggest. Under MCAR, missingness is independent of both observed and unobserved data. Under MAR, missingness may depend on observed data but, conditional on those observed data, does not depend on the missing value itself. Under MNAR, missingness depends on the unobserved value even after conditioning on observed data. These are assumptions about a data-generation process, not labels that can be established merely by comparing observed percentages.
If a predictor’s missingness rate differs across observed outcome categories, that difference is evidence of an observed association in the records analyzed. It is inconsistent with a simple claim that missingness is wholly unrelated to the observed outcome in that population, subject to the study’s data and selection rules. It does not determine whether the cause is an observed variable, an unobserved value, a data-collection process, or some combination. A lack of a visible difference also does not prove MCAR: a comparison may be imprecise, or the relevant relationship may involve variables not examined.
Use measured language in reporting. “Income missingness was 25% among renewed records and 12.5% among non-renewed records” is a descriptive statement. “Income was missing because renewed policyholders had higher incomes” is not justified by those rates alone. The latter claims a cause and refers to values that may not be observed.
Name the predictor whose missingness is being measured, the outcome categories, any subgroup variable, and which records are eligible for each rate.
For every group, record its total eligible count and its count with the predictor missing.
Divide the missing count by the group total; report rates, group sizes, and, where useful, differences in percentage points.
Compare pooled and within-subgroup rates, then describe the observed pattern without presenting it as proof of a missingness mechanism.
This approach is aligned with the SOA Exam PA (Predictive Analytics) syllabus: the objective is to interpret data-quality evidence carefully, keep denominators and comparisons clear, and avoid conclusions that the observed data cannot support.
Check Your Understanding
For each problem, calculate the requested rates and state what the comparison supports.
- A hypothetical portfolio has 80 records with observed outcome “active,” of which 12 have missing tenure, and 120 records with outcome “closed,” of which 12 have missing tenure. Calculate both rates and their difference in percentage points. (Answer: Active 15.00%; closed 10.00%; difference 5.00 percentage points.)
- In a hypothetical service study, 9 of 30 urban records and 6 of 40 rural records have missing service time. Calculate each subgroup rate and the overall rate. (Answer: Urban 30.00%; rural 15.00%; overall \(15/70=21.43\%\).)
- A hypothetical sample has 50 records with outcome “paid,” with 10 missing scores, and 150 with outcome “not paid,” with 15 missing scores. Calculate both outcome-specific rates and explain why comparing missing counts alone would mislead. (Answer: Paid 20.00%; not paid 10.00%; missing counts alone mislead because the group sizes differ.)
- For a hypothetical product group, 8 of 40 records with outcome A and 12 of 60 records with outcome B have missing values. Calculate both rates and the percentage-point difference. What does equality of these rates establish? (Answer: Both rates are 20.00%; difference 0.00 percentage points; this comparison shows no observed rate difference in this group, but does not prove MCAR.)
- A hypothetical two-region review finds the following missing-amount counts: in East, 6 of 20 outcome-A records and 4 of 40 outcome-B records; in West, 2 of 40 outcome-A records and 8 of 40 outcome-B records. Calculate the within-region rates and pooled outcome rates. (Answer: East A 30.00%, B 10.00%; West A 5.00%, B 20.00%; pooled A \(8/60=13.33\%\), B \(12/80=15.00\%\).)