Comparing Values Within Missingness Patterns

Why Compare Within Missingness Patterns?

Observed and imputed values can come from people with different patterns of missing data. A pooled comparison may therefore mix two things: differences between observed and imputed values among comparable pattern groups, and differences in how heavily each group contributes to the two sets. Separating these features helps make the diagnostic more interpretable.

Here, a missingness pattern describes which variables are observed and which are missing for a case. The focus is not on splitting cases by a substantive predictor such as age or treatment group. It is on asking whether observed and imputed values look similar among cases with the same observed-versus-missing configuration for the other variables. This is a descriptive check; it does not by itself establish that the imputation model is valid.

Definition: Let \(Y\) be the variable being diagnosed, and let \(R_{ij}=1\) if variable \(j\) is observed for case \(i\), and \(R_{ij}=0\) if it is missing. A reduced missingness pattern \(g\) for \(Y\) is a particular configuration of the indicators for variables other than \(Y\), written \(R_{i,-Y}=g\). Within that reduced pattern, compare observed \(Y\) values from cases with \(R_{iY}=1\) with imputed \(Y\) values from cases with \(R_{iY}=0\).

The word reduced matters. If a group were defined using the full pattern including \(Y\), cases with observed \(Y\) and cases missing \(Y\) could not belong to the same group: their \(Y\)-status differs by definition. Instead, match on the pattern of the other variables, then distinguish observed-\(Y\) cases from missing-\(Y\) cases within that group. A reduced pattern is still only a coarse description of the cases; two cases can share it while differing in their actual observed values.

What to Compare, and Under What Conditions

For a continuous \(Y\), useful comparisons within each reduced pattern include location, spread, quantiles, tails, and the overall shape of distribution plots. For a binary, nominal, ordinal, or count variable, use summaries appropriate to that variable’s scale. These comparisons should use the same coding, units, and plotting choices for observed and imputed values. Earlier material on distribution plots, sparse observed values, and variable-specific diagnostics provides the relevant plot and summary choices; the additional step here is to carry them out within missingness-pattern groups.

The diagnostic is most useful when a pattern has enough observed and missing-\(Y\) cases to make both distributions visible, and when there is meaningful overlap in reduced patterns. If a reduced pattern contains only observed \(Y\), or only missing \(Y\), there is no within-pattern observed–imputed comparison to make. If it contains only a few observed values, a discrepancy or apparent match may be unstable. A pattern-based match also does not mean cases are identical on the variables they observed.

Interpretation requires restraint. Similar distributions within a reduced pattern are reassuring as a descriptive check, but do not prove that imputations reproduce the unobserved values. Conversely, a difference can reflect a real difference between cases with observed and missing \(Y\), not necessarily an error in the imputation model. Pattern membership alone does not make observed and missing cases exchangeable. The diagnostic therefore supplements, rather than replaces, consideration of the imputation model and the assumptions used to justify it.

Choose the target variable.
Identify \(Y\), the variable whose observed and imputed values will be compared.

Define reduced patterns.
Group cases by the observed-versus-missing configuration of variables other than \(Y\). State which variables define the pattern so that the comparison is reproducible.

Check whether comparisons are possible.
For each pattern, count cases with observed \(Y\) and cases with missing \(Y\). Flag small or one-sided groups rather than implying that they provide a reliable comparison.

Compare within patterns and then inspect composition.
Use suitable summaries or plots within each usable group. Also examine how much each pattern contributes to the observed and imputed sets before interpreting a pooled comparison.

Report limitations.
Describe sparse groups, absent overlap, and the fact that pattern-based agreement is diagnostic evidence, not proof of accuracy.

Why Pooled Differences Can Be Misleading

The arithmetic of a pooled mean makes the role of group composition explicit. Suppose there are \(G\) reduced patterns. Let \(n_{Og}\) and \(\bar{Y}_{Og}\) be the number and mean of observed \(Y\) values in pattern \(g\), and let \(n_{Ig}\) and \(\bar{Y}_{Ig}\) be the number and mean of imputed \(Y\) values there. The pooled means are weighted by the pattern shares in their respective sets:

\bar{Y}_{O}=\sum_{g=1}^{G}\frac{n_{Og}}{n_O}\bar{Y}_{Og}, \qquad \bar{Y}_{I}=\sum_{g=1}^{G}\frac{n_{Ig}}{n_I}\bar{Y}_{Ig},

where \(n_O=\sum_g n_{Og}\) and \(n_I=\sum_g n_{Ig}\). Thus even if \(\bar{Y}_{Og}=\bar{Y}_{Ig}\) in every pattern, the pooled means can differ when the pattern weights differ. The same compositional issue can affect distribution plots and other summaries, not only means.

This identity is a decomposition of the displayed means, not a causal explanation. It shows that pattern composition is one possible source of a pooled gap; it does not establish why cases have different patterns or whether the imputations are appropriate.

Worked Example: A Pooled Gap Despite Within-Pattern Agreement

Suppose a hypothetical continuous outcome is measured in units. In a low-valued pattern, 90 observed cases have mean 10 units and 10 cases with missing outcomes have imputed mean 10 units. In a high-valued pattern, 10 observed cases have mean 20 units and 90 cases with missing outcomes have imputed mean 20 units. There are no within-pattern mean differences.

The pooled observed mean is

\bar{Y}_{O} =\frac{90(10)+10(20)}{90+10} =\frac{900+200}{100} =11\text{ units}.

The pooled imputed mean is

\bar{Y}_{I} =\frac{10(10)+90(20)}{10+90} =\frac{100+1800}{100} =19\text{ units}.

The pooled gap is \(19-11=8\) units, even though the observed and imputed means agree exactly in both patterns. The observed set is mostly from the low-valued pattern, while the imputed set is mostly from the high-valued pattern. The pooled comparison alone would conceal this composition difference. Pattern-specific agreement is informative, but it is not proof of accuracy: the cases with missing outcomes could still differ from the observed cases within a pattern in ways not captured by the pattern label.

Worked Pattern Comparisons

Worked Example: Matching on the Other Variables’ Pattern

In hypothetical clinic data, \(Y\) is a laboratory measure in milligrams per litre and \(X\) is a predictor. The reduced pattern is whether \(X\) is observed. Among cases with observed \(X\), 20 observed-\(Y\) cases have mean 10.0 mg/L and 8 missing-\(Y\) cases have imputed mean 10.8 mg/L. Among cases with missing \(X\), 6 observed-\(Y\) cases have mean 14.0 mg/L and 12 missing-\(Y\) cases have imputed mean 13.5 mg/L.

Within the observed-\(X\) pattern, the mean difference is

10.8-10.0=0.8\text{ mg/L}.

Within the missing-\(X\) pattern, it is

13.5-14.0=-0.5\text{ mg/L}.

The directions differ: imputed values have the higher mean in the first pattern and the lower mean in the second. This cautions against describing the imputation as uniformly high or low based on either group alone. The observed and imputed pooled means are also different:

\bar{Y}_{O} =\frac{20(10.0)+6(14.0)}{26} =\frac{284}{26} =10.92\text{ mg/L}, \qquad \bar{Y}_{I} =\frac{8(10.8)+12(13.5)}{20} =\frac{248.4}{20} =12.42\text{ mg/L}.

The pooled difference is \(12.42-10.92=1.50\) mg/L. The group calculations show that this pooled gap should not be read as a common within-pattern shift. The groups also differ in size, and the observed counts of 6 and 20 are modest; distribution plots and uncertainty about the missing values would still matter. This example checks pattern-specific location only, not the complete distribution.

Worked Example: Sparse Patterns and Distribution Shape

Consider hypothetical measurements in two reduced patterns. In Pattern A, observed values are 8, 10, and 12 units; imputed values are 9 and 11 units. Both means are 10 units. The observed sample standard deviation is

s_O=\sqrt{\frac{(8-10)^2+(10-10)^2+(12-10)^2}{3-1}} =\sqrt{\frac{8}{2}} =2.00\text{ units}.

The imputed sample standard deviation is

s_I=\sqrt{\frac{(9-10)^2+(11-10)^2}{2-1}} =\sqrt{2} =1.41\text{ units}.

The matching means do not imply matching spreads, although the imputed group has only two values. In Pattern B, the observed values are 18 and 22 units, while the imputed values are 15, 17, 18, 19, 21, 22, 23, and 25 units. Both means are 20 units. The observed sample standard deviation is

s_O=\sqrt{\frac{(18-20)^2+(22-20)^2}{2-1}} =\sqrt{8} =2.83\text{ units}.

For the imputed values, the squared deviations from 20 sum to 78, so

s_I=\sqrt{\frac{78}{8-1}} =\sqrt{\frac{78}{7}} =3.34\text{ units}.

Pattern B’s observed spread estimate rests on just two observations, so it is especially unstable. This comparison illustrates why counts and distribution shape should accompany location summaries. A plot can reveal features that a mean and standard deviation miss, but it cannot overcome a lack of observed information in a sparse pattern.

Multiple Imputations, Diagnostics, and Alternatives

With multiple imputation, each missing \(Y\) value has an imputed value in each completed data set. A diagnostic can display the observed values alongside the imputed values from each completed data set, using the same reduced patterns, or summarize how the pattern-specific comparisons vary across imputations. Do not concatenate all imputed values from all completed data sets and treat them as independent observations: the same cases recur across imputations, and the resulting display can give an exaggerated impression of the amount of information. State whether a plot shows one completed data set or several, and inspect whether the main pattern-specific features are stable across them.

Warning signs include a consistent location or spread discrepancy within a well-populated pattern, implausible values, or a discrepancy that appears in only some imputations. These signals merit investigation of the imputation model, coding, and pattern definitions. A single outlying case or a dramatic-looking difference in a tiny group is weaker evidence. If a group has no observed-\(Y\) cases, there is no within-group observed distribution against which to check imputations; report that limitation rather than borrowing a comparison from a different pattern without justification.

A pooled observed–imputed comparison is simpler but can hide the composition effect shown above. Stratifying by substantive predictors addresses a different question: whether the comparison varies across predictor values. Reduced-pattern stratification instead asks whether it varies across configurations of missingness. Both kinds of stratification can be useful, but neither alone makes cases exchangeable or supplies missing ground truth. When there is little overlap, a pooled display may still describe the overall data, but it cannot substitute for a within-pattern comparison that the data do not support. Combining patterns is defensible only when the grouping remains meaningful for the diagnostic; merging simply to obtain larger groups can conceal relevant differences.

Common Mistakes

Using the full pattern, including \(Y\), as the matching group. This separates observed-\(Y\) cases from missing-\(Y\) cases automatically. Match on the other variables’ pattern, then distinguish \(Y\)-status within that group.
Calling a pooled difference an imputation shift. Different pattern weights can produce a pooled gap even when pattern-specific means agree. Inspect within-pattern comparisons and group counts.
Treating a pattern-only match as complete adjustment. A shared missingness configuration does not mean cases share the same observed covariate values. Interpret the comparison as descriptive.
Overinterpreting sparse groups. A few observed values cannot provide a stable picture of a distribution. Mark such groups as limited or not meaningfully comparable.
Counting each imputation as a new case. Repeated imputations represent alternative completed data sets, not additional independent people. Compare across imputations without inflating the apparent sample size.

Key takeaway: Compare observed and imputed values within reduced missingness patterns when both groups are represented. Then examine pattern composition before interpreting pooled differences, and treat every result as a diagnostic rather than proof of imputation accuracy.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Why can cases with observed \(Y\) and cases missing \(Y\) not share a group defined by the full missingness pattern?
In Pattern A, observed and imputed means are both 5 units. In Pattern B, both are 15 units. If observed cases are mostly in A and imputed cases mostly in B, can pooled means differ?
A reduced pattern has 30 missing-\(Y\) cases but no observed-\(Y\) cases. What within-pattern comparison is available?
Why should imputed values from several completed data sets not be concatenated and treated as independent observations?

Solutions. (1) The full pattern includes whether \(Y\) is observed, so the two groups necessarily have different patterns. Define groups using the other variables’ indicators instead. (2) Yes. The pattern-specific means match, but different group weights can make pooled means differ. (3) None against observed \(Y\) within that pattern; report the lack of overlap rather than claiming a match. (4) The same cases recur across completed data sets, so the imputations are alternative completed versions, not new independent cases.