Why Missingness Patterns Matter
A model does not receive the information that was never recorded. When a field becomes missing more often, or when different fields begin to be missing together, the model’s inputs may change even if the underlying people or risks have not. The effect depends on how the prediction process handles missing values: a record might be excluded, imputed, assigned a missing-value category, or processed by a model that handles missingness directly.
For monitoring, it is not enough to report that “some data are missing.” A useful summary identifies which fields are missing, how often, for which records, and in what combinations. Comparing those summaries over time can reveal a stable limitation or a change that merits investigation. A change is a signal, not by itself proof of its cause or proof that predictive accuracy has changed.
The denominator matters. A field may be expected only for certain records. For example, a supplementary application question might be shown only when an applicant selects a particular product. If the question is not applicable to other applicants, treating every unasked question as an ordinary missing value can create a misleading rate. Define the eligible population for each field before comparing periods, and keep that definition consistent.
Summarizing and Comparing Missingness
A monitoring summary should usually include both counts and rates. Counts show the operational volume; rates allow comparison when cohort sizes differ. For a field with \(m\) missing values among \(n\) eligible records, the rate is \(m/n\). When comparing two periods, report the change in percentage points as well as any relative change that is useful. A rise from 8% to 15% is 7 percentage points; it is an 87.5% relative increase because \((15-8)/8=0.875\). Those are different descriptions of the same change.
Field-level rates do not reveal whether fields are missing on the same records. A joint-pattern table adds that information. With two fields, for example, records can have neither missing, only the first missing, only the second missing, or both missing. The share with at least one missing field is not generally the sum of the two field-level rates, because records missing both fields would be counted twice in that sum.
Comparisons should use equivalent populations and definitions. If the mix of channels, products, or eligibility groups changes, an overall rate can move even when rates within each group are stable. It is therefore useful to examine overall summaries alongside selected, relevant group summaries. This is a diagnostic comparison, not a substitute for defining a meaningful target population.
Worked Example: Comparing Field Rates Across Two Periods
A hypothetical insurer monitors two application fields, annual income and employment status. In Period A, 500 eligible applications include 40 missing income values, 25 missing employment values, and 10 applications missing both. In Period B, 600 eligible applications include 90 missing income values, 30 missing employment values, and 24 missing both. Calculate field rates, joint missingness, and the changes that should be highlighted.
Method. Divide each missing count by the eligible applications in its period. For income, Period A has \(40/500=0.08=8\%\), while Period B has \(90/600=0.15=15\%\). The change is \(15\%-8\%=7\) percentage points. Relative to the original rate, the increase is \(7/8=0.875=87.5\%\).
For employment status, Period A has \(25/500=5\%\), and Period B has \(30/600=5\%\). The count rises, but the rate is unchanged. Reporting only the counts would obscure that distinction.
Joint patterns. In Period A, the number missing at least one field is \(40+25-10=55\), so the rate is \(55/500=11\%\). The subtraction avoids counting the 10 records missing both fields twice. In Period B, the corresponding count is \(90+30-24=96\), so the rate is \(96/600=16\%\). The both-missing rates are \(10/500=2\%\) and \(24/600=4\%\).
Check. The Period A pattern counts are 445 missing neither, 30 missing income only, 15 missing employment only, and 10 missing both; they sum to 500. Period B has 504 missing neither, 66 missing income only, 6 missing employment only, and 24 missing both; they sum to 600. The income field and joint missingness both merit attention; the employment rate alone is stable. This calculation avoids the common mistake of interpreting a larger count as a higher rate.
From Missingness to Prediction Effects
The operational handling of missing inputs determines how a missingness change can affect predictions. If records with missing inputs are excluded, the scored population changes. If a field is imputed, the value supplied by that procedure becomes part of the model input. If a model includes a missingness indicator or uses a missing category, the indicator or category can itself affect the prediction. A monitoring note should therefore state the scoring rule rather than imply that all missing values have the same effect.
A simple calculation shows how a change in the share of missing records can shift an average prediction. Suppose that, under a fixed scoring policy, the mean prediction among records with an observed field is \(\mu_O\), and the mean prediction among records with that field missing is \(\mu_M\). If the missing share is \(p\), and those two group means remain fixed, the overall mean prediction is the weighted average below.
The formula follows by splitting the cohort into two non-overlapping groups. A proportion \(1-p\) contributes mean prediction \(\mu_O\), and a proportion \(p\) contributes mean prediction \(\mu_M\). Adding their weighted contributions gives the overall mean. Between two periods with missing shares \(p_1\) and \(p_2\), if the group means truly remain fixed, the change is
This is an accounting identity under the stated fixed-group-means condition, not a causal claim. If the types of records within either group change, or if the scoring policy changes, the group means may not stay fixed. A changed average prediction also does not establish changed realized accuracy; evaluating accuracy requires outcome information and an appropriate comparison.
Worked Example: Estimating the Effect of a Higher Missing Share
A hypothetical service team scores 1,000 applications in each of two periods. Under its unchanged scoring policy, the mean predicted event probability is 0.10 for applications with an observed verification field and 0.16 for applications where that field is missing. The field is missing for 10% of applications in Period A and 25% in Period B. Estimate the average predicted probability in each period and the difference in expected event counts.
Method. Apply the weighted-average formula using \(p_A=0.10\), \(\mu_O=0.10\), and \(\mu_M=0.16\):
In Period B:
The average predicted probability rises by \(0.115-0.106=0.009\), or 0.9 percentage points. For 1,000 applications, the sums of predicted probabilities are \(1{,}000(0.106)=106\) and \(1{,}000(0.115)=115\). These are expected event counts under the predictions, not guaranteed observed counts; their difference is 9.
Check. The missing-share change is \(0.25-0.10=0.15\), and the difference between the group means is \(0.16-0.10=0.06\). Their product is \(0.15(0.06)=0.009\), matching the difference in average predicted probabilities. This calculation avoids the mistake of treating a change in predictions as evidence that actual event frequency changed.
Stable Patterns, New Signals, and Limits
Missingness can be informative about the data-collection process, the population, or both. A stable rate may be a known limitation that still affects which records can be scored or how their predictions are formed. A sudden or concentrated change may indicate a new operational problem, but a summary alone cannot identify its cause. The appropriate conclusion is that the pattern changed and should be investigated, not that a particular system or group caused it.
It is also important to distinguish observed patterns from assumptions about why values are missing. In standard statistical terminology:
- Missing completely at random (MCAR) means that missingness is independent of both observed and unobserved data relevant to the analysis.
- Missing at random (MAR) means that, conditional on observed information, missingness does not depend on the unobserved value itself.
- Missing not at random (MNAR) means that missingness can depend on the unobserved value, even after conditioning on observed information.
A change in a rate or joint pattern does not, by itself, establish which mechanism applies. In particular, the observed data cannot generally confirm that missingness is unrelated to the values that were not recorded. Treat these labels as assumptions about the data-generation process, not conclusions obtained merely from a monitoring table.
Worked Example: Detecting a Change in Joint Missingness
A hypothetical administrator monitors contact preference and account tenure for 200 eligible records in each of two periods. In Period A, 150 records have both fields observed, 20 are missing contact preference only, 20 are missing tenure only, and 10 are missing both. In Period B, the corresponding counts are 120, 20, 20, and 40. Calculate each field’s missing rate, the share with any missingness, and the change in the both-missing pattern.
Period A. Contact preference is missing on \(20+10=30\) records, giving \(30/200=15\%\). Tenure is also missing on \(20+10=30\) records, giving \(15\%\). Any missingness occurs on \(20+20+10=50\) records, so the rate is \(50/200=25\%\). Both fields are missing on \(10/200=5\%\).
Period B. Each field is missing on \(20+40=60\) records, giving \(60/200=30\%\). Any missingness occurs on \(20+20+40=80\) records, so the rate is \(80/200=40\%\). Both fields are missing on \(40/200=20\%\).
The both-missing share increased by \(20\%-5\%=15\) percentage points; it quadrupled because \(20\%/5\%=4\). The pattern counts check: each period’s four cells sum to 200. This is more than a field-level rate change: the share missing both fields rose substantially. The result supports investigating the affected collection and scoring process, but does not establish why the change occurred or whether predictions became less accurate.
A Practical Monitoring Sequence
A concise review can separate measurement, interpretation, and follow-up. Keep the comparison tied to a defined cohort and the actual scoring policy. When rates change, ask whether the shift is concentrated in a field, a joint pattern, or a relevant group. Then state what the evidence supports and what remains uncertain.
Specify eligible records, fields, time window, and whether “missing” includes blank, unknown, or not-applicable values. Preserve the same definitions across periods.
Show the denominator as well as missing counts and percentages. Add joint-pattern summaries where co-missingness could matter.
Check whether an overall change is also present within relevant segments. A changed mix of groups can alter an overall rate even when their within-group rates are stable.
Record whether missing inputs lead to exclusion, imputation, a missing category, or another specified handling rule. Estimate prediction impact only under clearly stated assumptions.
Distinguish the observed change from a hypothesized cause. Identify what must be checked and whether the missingness affects coverage, predictions, or both.
A common exam trap is to add field-level missingness rates and call the result the rate of records with any missing data. If 15% of records are missing field A and 15% are missing field B, the share missing at least one field could range from 15% (if exactly the same records are missing both) to 30% (if the missing groups do not overlap). The joint pattern is needed to calculate it. Another trap is to state that stable missingness is harmless: stability describes the pattern over the comparison period, not its impact on the model or the suitability of its assumptions.
Check Your Understanding
For each problem, calculate using the stated eligible population and distinguish a rate change from a count change.
- A portfolio has 400 eligible records in Month 1, with 32 missing occupation values. In Month 2, it has 500 eligible records, with 45 missing occupation values. What are the two rates and the change in percentage points? (Answer: 8% and 9%; increase of 1 percentage point.)
- Among 300 records, 24 are missing field A, 18 are missing field B, and 6 are missing both. What share has at least one field missing? (Answer: \(36/300=12\%\).)
- A model’s mean prediction is 0.08 for records with an observed field and 0.14 for records with that field missing. The missing share rises from 20% to 30%, while those group means remain fixed. By how much does the overall mean prediction change? (Answer: \(0.10(0.14-0.08)=0.006\), an increase of 0.6 percentage points.)
- In a cohort of 250 records, 200 have neither of two fields missing, 15 have only the first missing, 20 have only the second missing, and 15 have both missing. What is the rate for the first field, the rate for the second field, and the rate with any missingness? (Answer: 12%, 14%, and 20%, respectively.)
- A monitored field is missing for 10% of records overall in each of two periods. In Period 1, two equally sized groups have missing rates of 4% and 16%; in Period 2, the groups have rates of 8% and 12%. What is the overall rate in each period, and what can the equal overall rates alone establish about subgroup patterns? (Answer: 10% in each period; the equal overall rates do not establish stable subgroup patterns, since both group-specific rates changed.)