Missingness Diagnostics and Study-Design Implications

Why Diagnose Missingness Before Choosing a Remedy?

A multivariate analysis can be numerically successful yet depend on very thin data support. A covariance estimate may rely on only a few people observed on a particular pair of variables; an inverse-probability analysis may give enormous weight to a small number of respondents. Diagnostics make these weaknesses visible before they are mistaken for evidence about the population.

The purpose is not to identify the missingness mechanism with certainty from the observed data. Instead, diagnostics describe which measurements are available together, assess whether observed response probabilities are small, and separate findings that can challenge simple assumptions from assumptions that require sensitivity analysis or better design.

Definition: For unit $i$ and variable $j$, let $R_{ij}=1$ if the value is observed and $R_{ij}=0$ if it is missing. The response pattern for unit $i$ is the vector $R_i=(R_{i1},\ldots,R_{ip})$. For a pattern $r$, let $n_r$ be the number of units with $R_i=r$. The variables observed under pattern $r$ are denoted $Y_{\mathrm{obs}}(r)$, and those missing are denoted $Y_{\mathrm{mis}}(r)$.

A pattern table answers a basic but important question: how many units contribute to each combination of observed variables? Marginal response rates alone do not answer it. If two variables are each observed often, that does not guarantee that many of the same units have both values.

Summarizing Patterns and Co-Observation

A useful diagnostic report gives the number and proportion of units in each observed response pattern, the observation rate for each variable, and the number observed jointly for each important pair. For variables $j$ and $k$, define the pairwise co-observation count $n_{jk}=\sum_{i=1}^{n}R_{ij}R_{ik}$. The proportion $n_{jk}/n$ describes how much of the sample contains both values. For larger $p$, a co-observation matrix or graph can make weakly supported pairs easier to spot.

These counts are descriptive, not an identification result. A nonzero count says that some paired observations exist; it does not guarantee adequate precision or that the covariance is identifiable under a chosen model. A zero count means the data contain no direct paired observations for that pair, so any estimate involving their joint distribution must depend on additional modeling structure.

Worked Example: Reading a Response-Pattern Table

Eight participants have measurements on $Y_1,Y_2,Y_3$. In the pattern codes below, 1 means observed and 0 means missing. The observed patterns are $111,110,101,011,110,100,010,001$. Summarize the pattern counts and co-observation support.

The pattern $110$ occurs twice. Each of $111,101,011,100,010,001$ occurs once. The counts sum to $2+1+1+1+1+1+1=8$, as required.

For $Y_1$, the observed patterns are $111,110,101,100$, giving $1+2+1+1=5$ observed values and an observation rate $5/8=0.625$. For $Y_2$, the patterns are $111,110,011,010$, also giving $5/8=0.625$. For $Y_3$, the patterns are $111,101,011,001$, giving $4/8=0.5$.

For the pair $(Y_1,Y_2)$, only $111$ and $110$ contribute, so $n_{12}=1+2=3$. For $(Y_1,Y_3)$, the contributing patterns are $111$ and $101$, so $n_{13}=1+1=2$. For $(Y_2,Y_3)$, they are $111$ and $011$, so $n_{23}=1+1=2$. Only one participant has all three measurements. Thus, the marginal rates conceal that joint support is sparse, especially for analyses requiring all three variables.

Response Probabilities, Overlap, and Practical Positivity

When a response indicator is modeled using fully observed covariates $X$, the response probability is $\pi(x)=P(R=1\mid X=x)$. A fitted model gives estimated probabilities $\hat{\pi}_i$ for individual units. Reviewing their distribution and comparing them across scientifically relevant groups can reveal regions with little representation among respondents. A model-based estimate is not a direct observation of the true probability, so its adequacy depends on the model and the data available to fit it.

For a single response indicator, positivity means that $\pi(x)>0$ in the covariate regions relevant to the target population. In practice, probabilities that are positive but extremely small can still cause difficulty: inverse-probability weights $1/\hat{\pi}_i$ become large, and estimates can be dominated by a few observations. Diagnostics should therefore report the range or distribution of fitted probabilities, the number of observations in low-probability regions, and whether conclusions change under reasonable modeling choices. Truncating weights may stabilize a calculation, but changes the procedure and can introduce bias; it does not repair absent support.

Worked Example: Comparing Estimated Response Probabilities

Suppose a fully observed baseline risk group contains 10 low-risk and 10 high-risk participants. Eight low-risk participants and two high-risk participants provide a follow-up measurement. Calculate the group-specific estimated response probabilities and interpret the contrast.

For the low-risk group, $\hat{\pi}_{L}=8/10=0.80$. For the high-risk group, $\hat{\pi}_{H}=2/10=0.20$. The absolute difference is $0.80-0.20=0.60$, and the ratio is $0.80/0.20=4$. Thus, the estimated response probability in the low-risk group is four times that in the high-risk group, not five times.

The corresponding inverse-probability weights are $1/0.80=1.25$ and $1/0.20=5$. A high-risk respondent therefore receives four times the weight of a low-risk respondent under this simple weighting scheme. The contrast is a warning about potential instability and limited representation; with only 10 people in each group, it is also imprecisely estimated. It does not establish why response differs or prove that a particular missingness assumption holds.

For multivariate outcomes, response probabilities may concern an entire pattern rather than one variable. Pattern probabilities can be estimated conditionally on observed predictors, but the number of distinct patterns may be large. It is often more useful to diagnose support for the scientific estimand: for example, whether important covariate strata have respondents with a particular outcome, or whether enough people have a key pair of measurements.

Worked Example: Marginal Rates Do Not Determine Joint Availability

In a study of 100 participants, measurement $A$ is observed for 70 and measurement $B$ for 60. Both are observed for 40 participants. Find the four pattern counts, compare the observed joint rate with the product of marginal rates, and state the design implication.

The $A$-only count is $70-40=30$, and the $B$-only count is $60-40=20$. The count with neither measurement is $100-(40+30+20)=10$. The four counts are therefore 40 with both, 30 with $A$ only, 20 with $B$ only, and 10 with neither; they sum to 100.

The marginal rates are $0.70$ and $0.60$, while the observed joint rate is $40/100=0.40$. Multiplying the marginal rates gives $0.70(0.60)=0.42$, which is not the observed joint rate. That product would be the joint probability only under an independence condition on the response indicators; the marginal rates by themselves do not supply that condition.

If the scientific objective requires relating $A$ to $B$, 40 paired records may be the limiting resource even though each marginal rate looks moderate. A design response could be to schedule both measurements together for a targeted subset, monitor paired completion during data collection, and reserve resources for follow-up of participants missing either measurement.

What Observed Data Can Say About Missingness

Definition: Missing completely at random (MCAR) means $P(R=r\mid Y=y)=P(R=r)$ for every relevant pattern $r$ and full data value $y$. Missing at random (MAR) means $P(R=r\mid Y=y)=g_r\{Y_{\mathrm{obs}}(r)\}$ for some function $g_r$: conditional on the values observed under pattern $r$, the pattern probability does not depend on the missing values. Missing not at random (MNAR) refers to dependence on missing values that remains after conditioning on the observed values.

MCAR has observable implications. For example, if a variable is observed in multiple response groups, its observed distribution can be compared across those groups. Systematic differences can be evidence against MCAR, subject to sampling variability and the comparison being meaningful. Formal tests of MCAR also rely on modeling and regularity assumptions; a nonsignificant result is not proof of MCAR, particularly when patterns are sparse.

MAR is different. It describes how response depends on values that are not all observed for the people with a given pattern. In general, the observed-data distribution alone cannot distinguish MAR from suitable MNAR mechanisms. Finding that response is predictable from observed baseline variables can be evidence against MCAR, and can motivate adjustment, but does not verify MAR. Conversely, failing to predict response from recorded variables does not show that response is independent of unobserved values.

Under MAR, there is a useful conditional-distribution implication. It is a precise statement of the assumption, but it is not usually a direct empirical test: for a given pattern, the missing values are unavailable precisely for the units selected into that pattern. When the same quantities are observed under other patterns, comparisons may probe restrictions under additional modeling assumptions, but they do not turn MAR into a generally testable assumption.

Result: Suppose the full data have a joint density, the pattern probability under MAR is $g_r(y_{\mathrm{obs}})$, and the relevant conditional densities exist. At observed values with positive pattern probability, $$ f(y_{\mathrm{mis}}\mid y_{\mathrm{obs}},R=r) = f(y_{\mathrm{mis}}\mid y_{\mathrm{obs}}). $$ To see this, the conditional density on the left is proportional to the joint density of the full data times the pattern probability: $$ f(y_{\mathrm{mis}}\mid y_{\mathrm{obs}},R=r) = \frac{f(y_{\mathrm{mis}},y_{\mathrm{obs}})g_r(y_{\mathrm{obs}})} {\int f(u,y_{\mathrm{obs}})g_r(y_{\mathrm{obs}})\,du}. $$ Because $g_r(y_{\mathrm{obs}})$ is constant with respect to the integration variable $u$, it cancels. The result is $$ \frac{f(y_{\mathrm{mis}},y_{\mathrm{obs}})} {\int f(u,y_{\mathrm{obs}})\,du} = f(y_{\mathrm{mis}}\mid y_{\mathrm{obs}}). $$ The positivity condition ensures that conditioning on $R=r$ at these observed values is meaningful. The equality follows from MAR; it does not make the missing conditional distribution directly observable for that pattern. $\square$

Why Small Probabilities Matter for Estimation

The next derivation shows why probability diagnostics are not merely presentational. Consider a scalar outcome $Y$, response indicator $R$, and fully observed covariates $X$. Suppose response is conditionally independent of $Y$ given $X$, with $\pi(X)=P(R=1\mid X)>0$. For any integrable function $h(X)Y$, inverse-probability weighting has the following population identity.

Result: If $E[|h(X)Y|]<\infty$, $0<\pi(X)$, and $P(R=1\mid X,Y)=\pi(X)$, then $$ E\left[\frac{R h(X)Y}{\pi(X)}\right]=E[h(X)Y]. $$ Indeed, conditional expectation given $X,Y$ yields $$ E\left[\left.\frac{R h(X)Y}{\pi(X)}\right|X,Y\right] = \frac{h(X)Y}{\pi(X)}E[R\mid X,Y] = \frac{h(X)Y}{\pi(X)}\pi(X) = h(X)Y. $$ Taking expectations of both sides proves the identity. For $h(X)=1$, it recovers the population mean of $Y$ from respondents in expectation. If $\pi(X)$ is very small, observed respondents in that region receive large weights, so finite-sample estimates can be unstable even when the identity holds. $\square$

This result assumes the response probability is correctly specified and the conditional response assumption is appropriate. Replacing $\pi$ with an estimated probability adds model-estimation uncertainty. A fitted probability close to zero is therefore a warning to investigate overlap, model specification, and the target population, not a signal that weighting should automatically be applied.

Turning Diagnostics Into Design Decisions

Diagnostics are most useful when tied to a planned action. The right remedy depends on whether the weakness is low overall response, poor joint measurement, or limited representation in particular covariate regions.

Locate the weak support.
Use pattern counts, variable-specific rates, pairwise co-observation, and response-probability summaries to identify the measurements or groups that contribute least to the target analysis.

Identify a feasible design change.
Consider synchronized measurement visits, reminders, alternate collection modes, translated materials, transport support, or targeted follow-up in groups with low response. If complete measurement on everyone is impractical, a planned validation or intensive-follow-up subsample can be designed to measure key outcomes together.

Reassess support and document limits.
After the design change, recalculate pattern counts and estimated probabilities. Record remaining sparse regions and state which conclusions depend on MAR, model extrapolation, or sensitivity assumptions.

A design remedy cannot retroactively reveal unobserved values, but prospective changes can reduce dependence on extrapolation. Stratified recruitment or targeted follow-up can improve representation in low-response groups. Collecting measurements together can improve joint support. Recording auxiliary variables that predict both response and outcomes can make a MAR analysis more plausible and informative, although it cannot prove MAR. When no design change is possible, report the limitation and examine how conclusions vary under plausible MNAR assumptions.

Common Diagnostic Mistakes

Treating marginal response rates as joint support. High rates for two variables do not imply that many of the same participants have both measurements. Check pairwise and pattern counts directly.
Calling a response model a test of MAR. Predictability of response from observed covariates can challenge MCAR and help construct an adjustment model. It cannot establish that response is independent of missing values conditional on the observed data.
Interpreting a non-significant test as proof. Sparse patterns and small samples can make tests weak. Report what was compared and the limits of the evidence.
Ignoring near-zero fitted probabilities. Positive estimates may still produce extreme weights and unstable estimates. Assess overlap and reconsider the target, design, or model rather than relying on a numerical fit alone.
Assuming response indicators are independent. The joint response rate is not generally the product of marginal rates. That equality requires an independence assumption that should not be silently imposed.

Key takeaway: Missingness diagnostics describe the observed support and expose fragile regions; they do not reveal the unobserved values or certify MAR. Use pattern counts and estimated probabilities to guide design, model checking, and sensitivity analysis, and state clearly which conclusions rely on assumptions that the observed data cannot verify.

Check Your Understanding

Use these questions to check whether you can interpret the diagnostics and their limits.

Why can two variables with high marginal observation rates still have weak co-observation?
In a response-probability model, what is the practical concern when some fitted probabilities are close to zero?
What kind of observed-data evidence can challenge MCAR, and why does it not establish MAR?
Under the stated MAR condition, what equality relates the distribution of missing values among pattern-$r$ units to the full-data conditional distribution?
Name one study-design change that could improve paired measurement support, and one limitation that remains even after making the change.