Posterior Predictive Checks for Candidate MNAR Models

What a Posterior Predictive Check Can Assess

A candidate MNAR model connects outcomes to the process that makes them missing. Its assumptions therefore affect both the distribution of observed data and the model’s predictions for outcomes that were not observed. Posterior predictive checks ask a focused question: if this fitted model had generated a study like the one observed, would the observable features of the replicated study look like the actual data?

The check is useful for finding observable features that a candidate model reproduces poorly. Examples include the number of participants who drop out, the distribution of observed outcomes at each visit, and an association between an observed earlier outcome and later dropout. A check that looks satisfactory is not evidence that the model has recovered the distribution of missing outcomes. Different MNAR assumptions can agree on the observable data and still extrapolate differently beyond it.

Definition: For a fixed candidate model \(M\) with parameter \(\theta\), a posterior predictive replicate is a simulated data set generated from the model after drawing \(\theta\) from its posterior distribution given the observed data. A posterior predictive check compares a summary \(T\) of the observed data with the same summary calculated from such replicated data.

For a dropout study, write \(Y\) for the outcome data, \(R\) for the observation indicators, and \(X\) for fixed design variables and covariates. For example, \(R_{it}=1\) can indicate that participant \(i\)'s outcome at time \(t\) is observed. A selection model may specify a joint distribution as \(p(Y,R\mid X,\theta,M)=p(Y\mid X,\theta,M)p(R\mid Y,X,\theta,M)\). The dependence of \(R\) on outcomes, including unobserved outcomes in the model, is where the MNAR assumption enters. Other candidate formulations may express the same joint distribution differently; each must specify how outcomes and observation indicators are generated.

The posterior \(p(\theta\mid O,X,M)\) is based on the observed data \(O\), which include the observed outcomes and missingness indicators. It incorporates the candidate model’s likelihood and parameter uncertainty. The check then generates complete replicated outcomes and indicators under that model and applies the missingness indicators to obtain the replicated observed data. It is these observable replicas—not the hidden parts of the actual study—that are compared with the actual observations.

Constructing the Check

The design variables or covariates \(X\) are usually held at their observed values when the goal is to check the model’s fit for this study design. For each simulation, draw a parameter value from the posterior, generate outcomes and missingness under the candidate model, and calculate summaries from the resulting observed portion of the replicate. The summary should be chosen before interpreting its result and should target a meaningful feature of the observed data.

\theta^{(b)}\sim p(\theta\mid O,X,M), \qquad (Y^{\mathrm{rep},(b)},R^{\mathrm{rep},(b)}) \sim p(Y,R\mid X,\theta^{(b)},M)

Let \(O^{\mathrm{rep},(b)}\) denote the observed portion of the \(b\)th simulated study: the outcomes generated for which \(R^{\mathrm{rep},(b)}=1\), together with the simulated observation indicators. Calculate \(T(O^{\mathrm{rep},(b)})\) and compare it with \(T(O)\). The posterior predictive distribution of the summary is

p\{T(O^{\mathrm{rep}})\mid O,X,M\} = \int p\{T(O^{\mathrm{rep}})\mid X,\theta,M\} \,p(\theta\mid O,X,M)\,d\theta .

This integral averages over both parameter uncertainty and the model’s sampling variation. In practice, simulations approximate it. For example, the fraction of replicated dropout counts at or below the observed count estimates a lower-tail posterior predictive probability. A very small or very large tail fraction can flag a discrepancy. Such a fraction is a diagnostic summary, not a frequentist p-value with a guaranteed type I error rate, and it is not the probability that the model is true.

Practical rule: Compare like with like. If \(T\) is calculated among observed outcomes in the actual data, calculate it among observed outcomes in each replicate too. If it is a dropout summary, calculate it from the replicated observation indicators. State the population, visit, subgroup, and conditioning used to define each summary.

Worked Checks for Observable Features

Worked Example: Checking the Overall Dropout Count

Hypothetical data and check. A two-visit study enrolled 100 participants, and 26 had dropped out before the second visit. Under one fitted MNAR candidate, 500 posterior predictive replicas are generated, each with the same 100-participant design. Suppose their replicated dropout counts have mean 26.4, standard deviation 4.8, and a central 90% interval from 18 to 34. In 231 of the 500 replicas, the count is 26 or less.

Calculation. The estimated lower-tail probability is \(\hat{q}=231/500=0.462\), or 46.2%. The observed count, 26, also lies within the central 90% predictive interval, 18 to 34.

Interpretation and assumptions. The observed dropout count is not unusual relative to these simulated counts, so this particular summary does not reveal a clear mismatch. This conclusion assumes the replicas were generated from the specified candidate, with posterior draws based on the observed data and the study design held fixed as intended. It does not show that the candidate’s assumptions about the unobserved outcomes are correct: the count alone does not identify those outcomes.

Worked Example: Checking an Observed-Outcome Distribution

Hypothetical data and check. At a scheduled visit, 60 of 80 participants have an observed outcome measured in points. Their observed mean is 10.4 points. A candidate model generates 1,000 posterior predictive replicas; in each, the mean is calculated over the outcomes that are observed in that replica. Suppose the simulated means have median 10.1 points and a central 90% interval from 9.35 to 10.82 points. The observed mean is at the 72nd percentile of the replicated means.

Calculation. The observed value lies inside the interval because \(9.35<10.4<10.82\). If 720 of the 1,000 simulated means are at or below 10.4, the estimated percentile is \(720/1000=0.720\), or 72.0%.

Interpretation and assumptions. The candidate can plausibly reproduce this observed mean in these simulations; the 72nd percentile is not an extreme position. This check is meaningful only if the summary is defined consistently—for example, the same visit and the observed outcomes in both the actual and replicated data. The mean alone can hide distributional problems: the model might miss skewness, tails, or subgroup differences while matching the average. Further summaries can target those features.

Worked Example: Checking a Joint Outcome-Dropout Feature

Hypothetical data and check. In a two-visit study, all 100 participants have a baseline outcome. Let the observed summary be the mean baseline outcome among participants who drop out before visit 2 minus the mean baseline outcome among those who remain. In the actual data, the difference is 1.8 points. For 500 replicas from a fitted MNAR candidate, calculate the same difference using the replicated baseline outcomes and replicated dropout indicators. Suppose the replicated median is 0.5 points, its central 90% interval is from \(-0.7\) to 1.7 points, and 14 replicas have a difference of at least 1.8 points.

Calculation. The estimated upper-tail fraction is \(14/500=0.028\), or 2.8%. The observed difference, 1.8 points, is just above the stated central interval’s upper endpoint of 1.7 points.

Interpretation and assumptions. This is a potentially meaningful mismatch in an observable relationship: under the candidate, a difference at least this large occurred rarely. It is a diagnostic signal to examine the model’s representation of baseline outcomes and dropout, not proof of a particular alternative mechanism. The comparison is appropriate because baseline outcomes are observed for everyone and the same difference is calculated in the actual and replicated data. The tail fraction depends on the chosen summary and is not a calibrated hypothesis test.

Why the Check Works—and What It Cannot Establish

The simulation has a direct justification. Under the candidate model, the observed-data likelihood determines a posterior distribution for \(\theta\). Conditional on a posterior draw, the model specifies a distribution for a new study’s outcomes and missingness. Integrating over posterior uncertainty gives the distribution of a summary that the model predicts for replicated observable data. If the observed summary falls far into a tail of that distribution, the candidate has difficulty reproducing that feature. The conclusion is about the candidate’s predictive performance for that summary, not about every aspect of the model.

A central limitation is that posterior predictive checks use summaries of observable data. The actual missing outcomes are not available to compare against predictions. If two candidates make similar predictions for observed outcome distributions and dropout counts, their different predictions for missing outcomes are not adjudicated by those summaries. In particular, a model can reproduce the frequency of dropout while assigning quite different outcome distributions to participants after dropout. Agreement on observed summaries cannot establish which extrapolation is right.

There is also a built-in conservatism: the same observed data are used to form the posterior and to define the observed summary being checked. As a result, a candidate may reproduce some summaries reasonably well even when it is not a good description of the data-generating process. This makes it important to use several summaries chosen for substantive reasons, rather than treating a single non-extreme simulated comparison as validation.

Diagnostics, Alternatives, and Limitations

Useful observable summaries include dropout counts at each visit, the distribution of observed outcomes by visit, quantiles or tail proportions, and relationships between observed outcomes or covariates and later dropout. For longitudinal data, distinguish overall summaries from those conditional on visit, prior history, or subgroup. An overall mean can be misleading when the composition of observed participants changes over time. If the number of participants at risk differs across visits, define the denominator for a dropout frequency explicitly.

A check can be going wrong when its data-generating step does not match its stated target. For example, simulating outcomes but retaining the actual missingness indicators cannot check whether the model reproduces dropout frequencies. Conversely, generating missingness but calculating a replicated outcome mean using values that would be unobserved in practice does not check the observed-outcome mean. Also inspect the replicated summaries themselves: implausible ranges, almost no variability, or a pattern inconsistent with the model’s stated process can signal coding or model-fitting problems.

Posterior predictive checks complement, rather than replace, other comparisons. Observed-data residual checks examine discrepancies in a more residual-focused way; posterior predictive checks compare chosen statistics against simulated data distributions while propagating posterior uncertainty. Comparing candidate MNAR models can also show whether a conclusion changes across assumptions. None of these observable-data checks alone validates the missing-outcome extrapolation. A check is only as informative as its summary, and a satisfactory result for one summary does not imply satisfactory fit for all observable features.

Common pitfall: “The observed dropout count lies in the 90% posterior predictive interval, so the MNAR mechanism is correct.” The interval only says the count is not unusual under that fitted model’s predictive distribution. Many mechanisms can yield similar dropout counts while implying different distributions for the missing outcomes. State the check’s observable target and keep its conclusion within that scope.

A Careful Workflow

Choose a targeted observable summary.
Specify exactly which outcomes, visit, subgroup, and dropout definition enter \(T(O)\), and why that feature matters.

Generate posterior predictive replicas.
Draw parameters from the fitted candidate’s posterior, simulate outcomes and missingness under that candidate, and form the replicated observed data.

Compare distributions, not just point summaries.
Inspect intervals, plots, or tail fractions for \(T(O^{\mathrm{rep}})\), and locate \(T(O)\) relative to the replicated distribution.

Report the scope of the conclusion.
Describe which observable feature the check supports or challenges, and state that it does not verify predictions for missing outcomes.

Check Your Understanding

Answer each question, then compare with the short solutions below.

A candidate produces a posterior predictive interval for the dropout count that contains the observed count. What does this establish, and what does it not establish?
Why should the observed-outcome mean in each replica be calculated only among outcomes that would be observed in that replica?
Of 400 replicas, 12 have a summary at least as large as the observed summary. Calculate and interpret the upper-tail fraction.
Can a good check of dropout frequency establish that the model’s missing-outcome distribution is correct? Explain.

Solutions. (1) It indicates that the observed count is not outside the chosen predictive interval under that fitted candidate. It does not verify the MNAR mechanism or the distribution of missing outcomes. (2) The target is the distribution of an observable summary; applying each replicate’s missingness indicators makes its summary comparable to the actual observed-data summary. (3) The fraction is \(12/400=0.030\), or 3.0%. The observed summary is relatively far into the upper tail, which flags a possible discrepancy, but this fraction is not a calibrated test p-value. (4) No. Different assumptions about missing outcomes can produce similar dropout frequencies, so that observable check cannot validate the unobserved-outcome extrapolation.