Common Pitfalls in Ranking MNAR Models

Why Ranking MNAR Models Requires Care

Comparing missing-not-at-random (MNAR) models can help show how conclusions depend on assumptions about unobserved outcomes. The comparison becomes misleading, however, if a single fit statistic is treated as a verdict about which model is true. Different candidates can fit the observed data similarly while implying different distributions for outcomes that are missing. A numerical ranking is meaningful only after specifying what is being compared, on which data, and for what inferential purpose.

The focus here is on common errors in ranking candidates, not on choosing one universal ranking procedure. Information criteria, Bayesian model evidence, cross-validation, predictive checks, and simulation address different questions. Their results cannot automatically be substituted for one another.

What Observed-Data Fit Can and Cannot Establish

Let \(Y\) denote an outcome, \(R\) indicate whether it is observed (\(R=1\)) or missing (\(R=0\)), and \(X\) denote observed covariates. In a selection model, a joint model may be written as an outcome distribution \(f(Y\mid X;\theta)\) and a response probability \(P(R=1\mid Y,X;\psi)\). For a single outcome, the observed-data contributions have the form

The first contribution involves an observed outcome; the second averages over outcomes that were not observed. Consequently, the observed-data likelihood evaluates the joint model through the information available in the observed records, while some parts of the model concern values that the data do not reveal directly. Distinct assumptions about those unobserved values can therefore lead to different extrapolations even when the candidates describe the observed data comparably well.

This is why “the model with the best observed-data fit is the correct MNAR model” is not a valid conclusion. A fit criterion can be useful for detecting poor representation of observed data or for comparing candidates on a common likelihood target. It cannot, on its own, verify an untestable MNAR assumption. The introductory material on information criteria, Bayesian model evidence, cross-validation, and predictive checks provides the standard tools; the key here is to interpret each tool within its scope.

First Pitfall: Ranking by a Number Without Checking Its Target

Fit measures are not interchangeable. For example, the Akaike information criterion is \(AIC=-2\ell+2k\), where \(\ell\) is the maximized log-likelihood for the specified data likelihood and \(k\) is the number of estimated parameters. Lower AIC is preferred among candidates whose likelihoods describe the same observed data and a comparable likelihood target. The penalty accounts for model complexity in that comparison; it does not turn the criterion into a test that the missingness mechanism is correct.

A common error is to compare maximized likelihoods from different observed-data targets, such as one calculated on all available records and another calculated only on complete cases, then declare the larger likelihood superior. The values need not be comparable: the records and likelihood being scored differ. The remedy is to define a common comparison set and likelihood target, or to state clearly that the analyses answer different questions and not rank them as if they were a single competition.

Second Pitfall: Comparing Nonaligned Analyses

A model comparison can be numerically tidy and still be scientifically misaligned. Candidate analyses may differ in their target population, outcome definition, follow-up time, covariate adjustment, or treatment-effect estimand. A treatment contrast for participants with observed outcomes at a visit is not automatically the same quantity as a contrast for all randomized participants at that visit. Comparing their estimates as though they were competing estimates of one common parameter confounds model assumptions with a change in the question.

Before interpreting a ranking, check that the models use the same outcome scale, visit or time horizon, population, and estimand. Also check that they are fitted to the same relevant observed information. If a candidate changes several of these at once, a ranking cannot isolate which change explains the result. A useful comparison can still be reported, but it must be described as a comparison of different analyses rather than a ranking of interchangeable models.

Third Pitfall: Treating Predictive Checks as Mechanism Tests

A predictive check asks whether data generated under a fitted model reproduce selected features of the data that can be checked. It is a diagnostic for model implications. It is not, by itself, a test that confirms an MNAR mechanism. In particular, a check based on observed outcomes and observed response patterns cannot reveal the actual values of outcomes that were never recorded.

A predictive check can also be uninformative if it targets a feature that all candidates reproduce by construction, or if its summary is insensitive to the differences between the candidates. A good diagnostic therefore states the replicated statistic, explains why it matters, and reports how the candidates perform on the same check. A striking discrepancy is evidence of a problem with the model's implications for that feature, not a direct measurement of the missing values.

A Practical Comparison Workflow

The following sequence helps prevent overinterpretation. It is not a mechanical rule for selecting a single winner; it makes explicit what each comparison can support.

The main alternatives are complementary, not interchangeable. Information criteria compare fit and complexity under a shared likelihood target. Cross-validation evaluates prediction for held-out data under the chosen validation design; it may favor prediction of observed records without resolving extrapolation to missing outcomes. Bayesian model evidence compares marginal likelihoods, which depend on the specified models and their prior distributions. Simulation with a known generating mechanism can assess performance in that simulated setting, but its conclusion depends on the mechanisms represented in the simulation. Sensitivity analysis across defensible MNAR assumptions describes how conclusions change; it does not select the true assumption merely because one result appears more convenient.

Diagnostics, Limitations, and a Common Mistake

Several warning signs should prompt a recheck of a reported ranking:

• The candidates are scored on different records, outcomes, time points, or likelihood targets.
• The reported “best” model changes when the scoring rule changes, but the result is presented as an unqualified winner.
• A predictive check uses an observable summary but is described as proof about unobserved outcomes or the missingness mechanism.
• Model-specific estimates target different populations or estimands, yet their numerical difference is interpreted as pure model sensitivity.
• The analysis reports one preferred candidate but omits materially different estimates from other plausible MNAR specifications.

Common mistake: “Candidate A has the best observed-data fit, so it is the correct MNAR model and its estimate should be reported as the answer.” The correction is that the fit criterion supports a relative statement only for its stated comparison target. It does not verify the model's assumptions about unobserved outcomes. Report the target and scoring rule, examine the candidates' implications, and show how the substantive conclusion varies across plausible specifications.

Even a careful comparison has limits. No ranking can recover information that the observed data do not contain without assumptions. A ranking can be sensitive to the set of candidates, the chosen diagnostics, validation design, or prior distributions. The appropriate conclusion is therefore usually conditional: under these specified candidates and comparison targets, this model scores better by this criterion, while the substantive estimate changes by this amount across the considered MNAR assumptions.