Pattern-Specific Missingness In Multivariate Data

Why Missingness Patterns Need Separate Attention

With several variables, a record can be incomplete in more than one way. A person might have income missing but health observed; another might have health missing but income observed; a third might have both missing. These are distinct missingness patterns. The probability of each pattern can respond to different information: the probability of missing income might depend on observed health, while the probability of missing both could depend on an observed baseline characteristic.

A single overall missing-data percentage cannot capture these differences. To assess the Missing At Random (MAR) condition, ask separately for each pattern whether its probability depends only on variables observed in that pattern. MAR does not require pattern probabilities to be equal across people or across patterns. The essential restriction is which values each probability may depend on.

Definition: Let \(Y=(Y_1,\ldots,Y_p)\) denote the full data vector for one unit. Define \(R=(R_1,\ldots,R_p)\), where \(R_j=1\) if \(Y_j\) is observed and \(R_j=0\) if it is missing. A missingness pattern is a possible value \(r\) of \(R\). Let \(O(r)=\{j:r_j=1\}\) be the indices observed in pattern \(r\), and let \(M(r)=\{j:r_j=0\}\) be the indices missing. The pattern probability given the full data is \(\pi_r(y)=P(R=r\mid Y=y)\). The missingness mechanism is MAR if, for every pattern \(r\), \(\pi_r(y)\) depends on \(y\) only through the observed components \(y_{O(r)}\). Equivalently, \(\pi_r(y)=\pi_r(y')\) whenever \(y_{O(r)}=y'_{O(r)}\), for full-data values in the relevant support.

This is a pattern-by-pattern condition. If \(r\) records that \(Y_1\) is observed and \(Y_2\) is missing, then \(Y_1\) is available to explain that pattern's probability but \(Y_2\) is not. If a pattern has no observed variables, its probability must not depend on any component of \(Y\) under MAR. If every variable is observed, the condition for that pattern places no restriction on dependence upon \(Y\), because all components are observed.

A Direct Check of the Condition

For a proposed mechanism, write down the probability of each pattern given the full data. For each pattern in turn, hold its observed values fixed and ask whether changing any missing value changes that pattern's probability. If it does, the MAR condition fails for that pattern. If it does not, that pattern satisfies its part of the MAR condition.

Worked Example: Checking MAR for Two Binary Variables

Suppose the full data are two binary variables \(X\) and \(Z\), each taking values 0 or 1. The four patterns are \(r=(1,1)\) (both observed), \(r=(1,0)\) (\(X\) observed, \(Z\) missing), \(r=(0,1)\) (\(X\) missing, \(Z\) observed), and \(r=(0,0)\) (both missing). Consider this hypothetical mechanism:

\begin{aligned} \pi_{10}(x,z)&=0.10+0.10x,\\ \pi_{01}(x,z)&=0.15+0.10z,\\ \pi_{00}(x,z)&=0.05,\\ \pi_{11}(x,z)&=1-\pi_{10}(x,z)-\pi_{01}(x,z)-\pi_{00}(x,z)\\ &=0.70-0.10x-0.10z. \end{aligned}

Check the probabilities. For \(x,z\in\{0,1\}\), every expression is nonnegative. For example, the smallest value of \(\pi_{11}\) is \(0.50\), at \(x=z=1\). The four probabilities sum to 1 by construction, so they define a valid pattern distribution.

Check each pattern. For \(r=(1,0)\), the observed value is \(X\), and \(\pi_{10}\) depends only on \(x\), not on missing \(z\). For \(r=(0,1)\), the observed value is \(Z\), and \(\pi_{01}\) depends only on \(z\). For \(r=(0,0)\), neither variable is observed and \(\pi_{00}\) is constant. For \(r=(1,1)\), both variables are observed, so dependence on both \(x\) and \(z\) is permitted. Thus each pattern satisfies its MAR condition, and the mechanism is MAR.

The response probabilities are not identical. For \(x=z=0\), the probabilities of patterns \(10\), \(01\), \(00\), and \(11\) are \(0.10\), \(0.15\), \(0.05\), and \(0.70\). For \(x=z=1\), they are \(0.20\), \(0.25\), \(0.05\), and \(0.50\). This variation is compatible with MAR because, for each pattern, the probability depends only on values observed under that pattern.

Why the Condition Is Pattern-Specific

The formal condition follows directly from what MAR means for a given pattern. Once the pattern \(r\) is fixed, the observed information about \(Y\) consists of \(y_{O(r)}\). A probability that is a function of these observed values has the same value for any two full data vectors agreeing on them. Conversely, if the probability is unchanged whenever the observed components are held fixed, it cannot depend on the missing components over the relevant support. This gives the equivalent formulation in the definition.

This reasoning also explains two useful edge cases. For a pattern with no observed values, all full-data vectors agree on the empty set of observed components, so its probability must be constant in \(y\) under MAR. For a complete-data pattern, two vectors agreeing on all observed components agree on all of \(y\); the condition therefore imposes no additional restriction on that pattern's probability.

Worked Example: Finding a Violation in One Pattern

Again let \(X\) and \(Z\) be binary, and consider a different hypothetical mechanism:

\begin{aligned} \pi_{10}(x,z)&=0.10+0.20z,\\ \pi_{01}(x,z)&=0.10,\\ \pi_{00}(x,z)&=0.05,\\ \pi_{11}(x,z)&=0.75-0.20z. \end{aligned}

The probabilities sum to \(1\). When \(z=0\), they are \(0.10,0.10,0.05,0.75\) for patterns \(10,01,00,11\); when \(z=1\), they are \(0.30,0.10,0.05,0.55\). All are nonnegative.

For pattern \(10\), \(X\) is observed and \(Z\) is missing. Hold \(X=0\) fixed and compare two full-data values: \((x,z)=(0,0)\) gives \(\pi_{10}=0.10+0.20(0)=0.10\), while \((x,z)=(0,1)\) gives \(\pi_{10}=0.10+0.20(1)=0.30\). The probability changes when only the missing value \(Z\) changes. Therefore MAR fails for pattern \(10\). The fact that \(\pi_{01}\) and \(\pi_{00}\) are constant does not repair this failure: MAR requires the condition to hold for every pattern. Pattern \(11\) satisfies its condition because both variables are observed.

Pattern Frequencies in Observed Data

Pattern frequencies can be summarized within groups defined by variables observed for everyone. Such summaries show whether the distribution of patterns varies across those observed groups. They do not reveal how pattern probabilities depend on values that are missing for some records. Consequently, observed frequencies may be consistent with a MAR explanation without establishing that MAR is true.

Worked Example: Comparing Patterns Across Age Groups

In a hypothetical dataset of 100 people, age group is recorded for everyone, while two measurements \(Y\) and \(Z\) may be missing. The table gives pattern counts within each age group.

Age group	Both observed	Only \(Y\) observed	Only \(Z\) observed	Neither observed	Total
Younger	30	10	5	5	50
Older	20	10	10	10	50

Within the younger group, the empirical pattern proportions are \(30/50=0.600\) for both observed, \(10/50=0.200\) for only \(Y\) observed, \(5/50=0.100\) for only \(Z\) observed, and \(5/50=0.100\) for neither observed. Within the older group they are \(20/50=0.400\), \(10/50=0.200\), \(10/50=0.200\), and \(10/50=0.200\), respectively. Each row sums to 1.

These empirical frequencies differ by age group for three patterns. Because age is observed for everyone, such variation could be compatible with MAR: each pattern probability may depend on age, as well as on any \(Y\) or \(Z\) values observed in that pattern. However, the table cannot tell whether, for example, the probability of observing only \(Y\) also depends on the unobserved value of \(Z\). The observed data therefore do not verify MAR. The proportions are estimates from this hypothetical sample, not known population probabilities.

How MAR Differs From Nearby Mechanism Descriptions

MCAR, MAR, and MNAR describe restrictions on how pattern probabilities may relate to the full data. The distinction is easiest to state for a fixed pattern: MCAR allows no dependence on the data values; MAR allows dependence on values observed in that pattern; MNAR involves dependence on one or more values missing in that pattern after the observed values are held fixed. MCAR is a special case of MAR. A response rate that differs across patterns is not, by itself, evidence of MNAR; the relevant question is whether a pattern's probability depends on values missing under that pattern.

Description	Restriction for a pattern \(r\)
MCAR	\(\pi_r(y)\) is constant in all components of \(y\).
MAR	\(\pi_r(y)\) may depend on \(y_{O(r)}\), but not on \(y_{M(r)}\) once \(y_{O(r)}\) is fixed.
MNAR	For at least one pattern, its probability depends on a component in \(y_{M(r)}\) after the observed components are fixed.

A practical diagnostic is to tabulate or visualize pattern frequencies against fully observed covariates and compare patterns on variables observed for the relevant records. Such checks can identify variation worth describing, but they cannot rule out dependence on values that are missing. The MAR condition concerns the full-data mechanism, including those unavailable values. This limitation is why observed pattern associations should not be treated as a test that proves MAR.

Common Mistake

Wrong reasoning: “The overall missingness rate is 25%, so the mechanism is MAR,” or “The rates differ across patterns, so the mechanism is MNAR.” Neither conclusion follows. An overall rate combines distinct patterns and discards information about which variables are observed. Unequal pattern probabilities may arise solely from differences in observed covariates, which is compatible with MAR. Conversely, apparently similar observed rates do not exclude a dependence on missing values.

Correction: Specify the pattern, identify its observed and missing variables, and compare its probability across full-data values that agree on the observed variables. If the probability changes as a missing value changes, that pattern violates MAR. Observed data alone generally cannot carry out this comparison for the missing values.

Key takeaway: MAR is a collection of restrictions, one for each missingness pattern. For each pattern, its probability may vary with values observed in that pattern, but must remain unchanged when only values missing in that pattern are changed.

Check Your Understanding

Answer each question by identifying the relevant pattern and the values observed under it.

For variables \(A,B,C\), pattern \(r=(1,0,1)\) has probability \(0.2+0.1A\). Does this pattern satisfy MAR?
For the same pattern, suppose its probability is \(0.2+0.1B\). Does it satisfy MAR?
Under MAR, must the probability of a pattern with no observed variables be the same for everyone?
In a sample, the proportion with both outcomes observed is lower in one age group than another. Does this alone establish MNAR?
For a complete-data pattern, what does the pattern-specific MAR condition require?

Solutions. (1) Yes. The pattern observes \(A\) and \(C\), and this probability depends only on \(A\), an observed value. (2) No. \(B\) is missing under \(r=(1,0,1)\), so changing \(B\) while holding \(A\) and \(C\) fixed changes the pattern probability. (3) Yes. With no observed variables, MAR permits no dependence on any data value, so the pattern probability must be constant in \(y\). (4) No. The difference may be explained by observed age, which is compatible with MAR; the pattern probabilities' dependence on missing outcomes is not determined by this comparison. (5) Nothing beyond being a valid probability mechanism: all variables are observed in that pattern, so the condition places no restriction on dependence on their values.