What Missingness Diagnostics Can Establish

What Can a Missingness Diagnostic Tell You?

A missingness diagnostic asks whether the records with missing values look different, in some observable way, from the records with observed values. A plot or subgroup summary may reveal that missingness is more common for older participants, for a particular clinic, or among people with lower recorded scores. That is useful evidence: it identifies structure that a purely random-missingness account would not predict. But the check only uses information that was recorded. It cannot directly show how missing values themselves relate to the chance of being missing.

This distinction matters when choosing how to interpret regression results. Finding that missingness varies with an observed variable can make a missing-completely-at-random explanation less plausible. It does not, by itself, establish that missingness is missing at random rather than missing not at random. The latter distinction concerns information that is unavailable for the missing records.

Definition: For a variable \(Y\), define its response indicator \(R_Y\) by \(R_Y=1\) when \(Y\) is observed and \(R_Y=0\) when \(Y\) is missing. A missingness diagnostic compares \(R_Y\), or the observed values of \(Y\), with other recorded information. MCAR means missingness does not depend on the data values; MAR means it may depend on observed data but not on the missing values after conditioning on the observed data; MNAR means it still depends on missing values after conditioning on observed data.

For example, if age is recorded for everyone but a laboratory measurement \(Y\) is sometimes missing, the missingness rate can be compared across age groups. If that rate differs substantially, the data provide evidence that \(R_Y\) is associated with age. Because age is observed, this pattern is compatible with MAR: missingness may depend on age. It does not rule out MNAR, because missingness might also depend on the unrecorded value of \(Y\), even after accounting for age.

Plots and Subgroup Summaries

A useful first step is to choose an observed variable that could plausibly be related to data collection or to the variable with missing values. Then examine the missingness rate across meaningful values or groups of that variable. For a quantitative variable, a scatterplot or a plot of missingness rate across bins can reveal a trend. For a categorical variable, a bar chart of missingness rates by group is often easier to interpret. A table of counts and rates is a simple, auditable version of the same comparison.

The response rate in a subgroup is the proportion of its records for which the target variable is observed. If subgroup \(g\) contains \(n_g\) records and \(m_g\) have missing \(Y\), its missingness rate is \(m_g/n_g\); its response rate is \(1-m_g/n_g\). Always show the subgroup counts as well as the percentages. A rate based on a small group can look dramatic while being imprecise.

Worked Example: Missing Biomarker Values by Age Group

Hypothetical data: A clinic recorded age for 80 people, but a biomarker measurement was missing for some. Among 40 people younger than 50, the measurement was missing for 4. Among 40 people aged 50 or older, it was missing for 16. Compare the missingness rates and state what can be concluded.

For the younger group, the missingness rate is \(4/40=0.10\), or 10%. For the older group, it is \(16/40=0.40\), or 40%. The difference in missingness rates, older minus younger, is

$$ 0.40-0.10=0.30 $$

Thus, in this hypothetical sample, the biomarker is missing 30 percentage points more often in the older group. A bar chart of the two rates would make this pattern visible. Since age is observed for everyone, the difference is evidence that biomarker missingness is associated with recorded age; it is not what MCAR would predict in the population. It is consistent with missingness depending on age, but it does not show whether the unrecorded biomarker values also affect missingness. The result therefore does not establish MAR rather than MNAR.

Plots are most useful when they show both the pattern and the amount of data behind it. For instance, plotting each observed predictor against \(R_Y\), or comparing predictor distributions for \(R_Y=1\) and \(R_Y=0\), can help identify trends that a single overall missingness percentage would hide. When there are many predictors or many possible subgroups, treat striking patterns as leads for investigation, not as definitive discoveries: searching many comparisons makes it easier to find a difference by chance.

Response-Indicator Comparisons

Another diagnostic compares observed values of a predictor between records where a target variable is observed and records where it is missing. In this comparison, the response indicator divides the records into two groups. The predictor must actually be available for both groups; it would not be possible to compare the missing target values with the observed target values as if the former had been measured.

A difference between the groups can flag systematic missingness related to the observed predictor. A lack of a visible difference is more limited: it says that this comparison did not reveal an association, not that the response indicator is independent of every observed variable or of the unavailable values. Graphs can show distributional differences that a comparison of means would miss, such as different spreads or unusually large values.

Worked Example: Prior Scores and Missing Follow-Up Outcomes

Hypothetical data: A program has a recorded prior score for every participant. A follow-up outcome is observed for 32 participants, whose mean prior score is 72 points, and missing for 28 participants, whose mean prior score is 61 points. Compare the groups using the response indicator for the follow-up outcome.

Let \(R_Y=1\) denote an observed follow-up outcome and \(R_Y=0\) a missing follow-up outcome. The observed difference in mean prior score, observed-outcome group minus missing-outcome group, is

\bar{x}_{R_Y=1}-\bar{x}_{R_Y=0}=72-61=11\text{ points}

The observed-outcome group has a mean prior score 11 points higher than the missing-outcome group. A boxplot or dot plot of prior score by \(R_Y\) would help show whether this difference reflects a broad shift or a few extreme scores. The comparison suggests that missingness is associated with an observed predictor, so an MCAR account may be questionable. It does not tell us whether follow-up outcomes would have been higher or lower for the participants with missing outcomes. In particular, it cannot decide whether the missingness is MAR or MNAR.

A response-indicator comparison is descriptive unless a formal inferential procedure is specified. Even a small p-value from a test of association would indicate incompatibility with a particular independence model under that test's assumptions; it would not give the probability that missingness is MAR, nor establish the mechanism. Conversely, a large p-value is not proof of independence. Small samples, noisy measurements, and weak associations can all make real patterns hard to detect.

Why Observed Data Cannot Separate MAR from MNAR

The limitation is not merely that a particular plot might be too crude. The distinction can be impossible to resolve from observed data alone. Records with \(R_Y=0\) reveal that \(Y\) is missing, but they do not reveal the value of \(Y\) that was not recorded. Different assumptions about those unseen values can produce exactly the same observed records.

Worked Example: Two Mechanisms, the Same Observed Records

Hypothetical data-generating illustration: Suppose the observed data have the following proportions: 40% of all records have \(Y=1\) observed, 40% have \(Y=0\) observed, and 20% have \(Y\) missing. The observed records therefore have equal numbers of \(Y=1\) and \(Y=0\). Consider two possible explanations for the missing records.

Explanation A: MCAR (and therefore MAR). Suppose the full population has \(P(Y=1)=0.50\), and each record has probability 0.80 of having \(Y\) observed, independently of \(Y\). Then

$$ P(R_Y=1,Y=1)=0.80(0.50)=0.40 $$

Likewise, \(P(R_Y=1,Y=0)=0.80(0.50)=0.40\), leaving 0.20 missing. This produces the stated observed proportions.

Explanation B: MNAR. Instead, suppose the full population has \(P(Y=1)=0.58\), with 0.18 of all records having \(Y=1\) missing and 0.02 having \(Y=0\) missing. The observed proportions are still 0.40 for each value, and 0.20 missing. In this explanation, the probability of observing \(Y\) differs by its value:

P(R_Y=1\mid Y=1)=\frac{0.40}{0.58}\approx 0.690, \qquad P(R_Y=1\mid Y=0)=\frac{0.40}{0.42}\approx 0.952

Here the chance of observing \(Y\) depends on \(Y\) itself, so the mechanism is MNAR. Yet both explanations produce the same observed proportions: \(Y=1\) observed for 40% of records, \(Y=0\) observed for 40%, and \(Y\) missing for 20%. No diagnostic based only on these observed data can choose between the explanations.

This example gives a short justification for a general caution: observed-data patterns can reveal associations among recorded variables and response indicators, but the missing values are not available to check whether response depends on them. The same limitation applies when the regression data contain several variables with different missingness patterns. A rich set of plots can describe observed structure in detail without identifying how unseen values are distributed.

Interpreting Diagnostics Carefully

A practical diagnostic report should say exactly what was compared, how much data were involved, and what the pattern supports. For example: “The follow-up outcome was missing more often among older participants in this sample; this is evidence of an association with recorded age, but does not distinguish MAR from MNAR.” This is more informative and more defensible than labelling the data “MAR” because one comparison showed a difference.

Use counts and rates together. Percentages without denominators can conceal how little information supports a subgroup rate.
Look beyond averages. Plots can reveal skew, spread, or unusual observations even when group means are similar.
Keep the target clear. State which variable's response indicator is being compared; different variables may have different missingness patterns.
Separate evidence from assumptions. An observed association can challenge MCAR, but MAR and MNAR concern how missingness relates to values that may not be observed.
Do not treat “not detected” as “absent.” A quiet plot or non-significant test cannot prove the absence of systematic missingness.

Key takeaway: Plots and response-indicator comparisons can find observable patterns in missingness and can make a simple MCAR explanation less credible. Because the missing values themselves are unavailable, these checks cannot establish that missingness is MAR rather than MNAR.

Check Your Understanding

Answer each question using the distinction between observed patterns and unobserved values.

A laboratory result is missing more often in one recorded age group than another. What does this pattern suggest, and what does it not establish?
Why should a subgroup missingness table show counts as well as percentages?
A predictor has the same sample mean among records with observed and missing outcomes. Does that establish MCAR? Explain.
In the worked illustration, what information about \(Y\) is unavailable for records with \(R_Y=0\), and why does that matter for distinguishing MAR from MNAR?
Write one cautious sentence reporting a diagnostic that found an association between an observed predictor and a response indicator.