Comparing MNAR Models at the Level of Observed-Data Fit

What an Observed-Data Fit Comparison Can Tell You

An MNAR model specifies how outcomes and missingness are related, including features of outcomes that are not observed. A fit comparison, however, is calculated from the records that are available. Its direct evidence concerns how well candidate models represent those records: the observed outcomes, covariates, and missingness indicators included in the likelihood. A better observed-data fit does not by itself establish that a model correctly describes outcomes after dropout or other unobserved values.

This distinction matters because different assumptions about missing outcomes can sometimes produce the same probability distribution for everything recorded. Such models have the same likelihood on the observed data, even if their implied full-data distributions differ. Conversely, a model can fit the observed records poorly, which is evidence against the restrictions it imposes on that observed distribution. Neither result alone verifies an MNAR assumption.

The Likelihood Uses What Was Recorded

Let \(Y_i\) denote a subject’s outcome, \(R_i\) indicate whether it is observed, and \(X_i\) denote observed covariates. For a simple outcome, take \(R_i=1\) when \(Y_i\) is observed and \(R_i=0\) otherwise. A selection model represents the full-data distribution using an outcome model and a response model, for example:

Here \(\theta\) and \(\psi\) index the outcome and response models. When \(R_i=1\), the observed-data contribution is the joint probability or density of the observed outcome and response indicator. When \(R_i=0\), the outcome is unknown, so its possible values must be summed or integrated out:

For independent subjects, the observed-data likelihood is \(L(\theta,\psi)=\prod_i L_i(\theta,\psi)\), and the log-likelihood is \(\ell(\theta,\psi)=\sum_i\log L_i(\theta,\psi)\). For discrete outcomes, the integral is replaced by a sum. This construction explains both the value and the limit of the comparison: the likelihood accounts for the missingness pattern, but it does not use the unobserved outcome as if it had been recorded.

A comparison may assess fit to observed outcomes, the frequency of missingness patterns, or their joint distribution, depending on how the likelihood is specified. If an analysis conditions on a missingness pattern rather than modeling its probability, it does not assess the marginal fit of that pattern’s frequency. It is therefore important to state what data enter the likelihood before interpreting a fit statistic.

Why Observed Fit Does Not Verify the Missing-Outcome Assumption

The limitation can be shown directly with a binary outcome. Let \(Y=1\) indicate one outcome category, and \(R=1\) indicate that the outcome is observed. Suppose the observed-data probabilities are

The last probability records that the outcome is missing but does not say which value it would have taken. Let \(p=P(Y=1)\), \(q_1=P(R=1\mid Y=1)\), and \(q_0=P(R=1\mid Y=0)\). Any values satisfying

give the same three observed-data probabilities, provided the resulting response probabilities lie between zero and one. For example, choose \(p=0.50\). Then \(q_1=0.24/0.50=0.48\) and \(q_0=0.36/0.50=0.72\). Alternatively, choose \(p=0.60\). Then \(q_1=0.24/0.60=0.40\) and \(q_0=0.36/0.40=0.90\). Both specifications make response depend on \(Y\), and hence are MNAR in this simple setup; both assign the same probabilities to all recorded-data categories.

This example is a finite illustration of a general identification issue: observing the response indicator and observed outcomes does not automatically identify how response depends on an outcome that is itself missing. Additional restrictions, external information, or sensitivity assumptions can distinguish full-data models, but a comparison of observed-data likelihoods cannot supply that information by itself.

Choosing a Comparison and Checking Its Conditions

A likelihood-ratio test is appropriate when one model is nested within another, both are fitted to the same observed records using compatible likelihoods, and the null model is a regular restriction of the larger model. Under the null, the statistic \(2(\ell_{\text{large}}-\ell_{\text{small}})\) has an approximate chi-square distribution, with degrees of freedom equal to the difference in the number of free parameters. This approximation relies on conditions such as identifiability, an interior null parameter value, and adequate sample size. It is not automatically valid just because two likelihood values are available.

For nonnested candidates, an information criterion such as AIC can summarize a trade-off between maximized likelihood and model complexity. For AIC, \( \mathrm{AIC}=-2\ell+2k \), where \(k\) is the number of estimated parameters. It can be compared across models fitted to the same data with likelihoods for the same observed records. A smaller AIC is a relative selection result among the candidates considered, not a test that a model is true and not evidence that its missing-outcome assumptions are correct.

The following distinctions help prevent overinterpretation.

Diagnostics, Failure Modes, and Practical Limits

Before interpreting a comparison, verify that the candidate models use the same subjects and the same observed-data likelihood. Check which covariates, outcomes, and missingness patterns are modeled, and whether any records were excluded. A raw log-likelihood from a model conditional on response patterns is not automatically comparable with a likelihood that also models pattern frequencies. Likewise, comparisons can be misleading if one likelihood includes constants or data contributions that the other omits inconsistently.

Inspect the fitted observed-data probabilities and residual discrepancies, not only a single summary statistic. In sparse samples, near-zero fitted probabilities or very small expected counts can undermine chi-square approximations and make estimates unstable. Boundary estimates, nonidentifiable parameters, or a null model that is not a regular interior restriction are warning signs that the usual likelihood-ratio reference distribution may fail. A parametric bootstrap or an appropriate specialized method may be needed to calibrate a test, but it still cannot recover information absent from the observed records without additional assumptions.

A poor observed-data fit is consequential: it indicates that a model fails to reproduce some feature of the data it was asked to represent. A good fit is more limited. Several models may fit equally well while making materially different assumptions about unobserved outcomes. Therefore, the fit comparison should be reported alongside the exact restriction or criterion, the records included, and the scope of the conclusion.

Common Mistake

The same caution applies when two models have indistinguishable fit statistics. That does not prove their full-data implications are equivalent. It may instead mean the recorded data do not distinguish their assumptions. The substantive consequence is that fit-based ranking alone is insufficient for choosing among such MNAR specifications.