Worked Comparison of MNAR Models in a Longitudinal Trial

Why a Worked Comparison Needs Alignment

Several MNAR models can describe the same longitudinal trial while making different assumptions about how dropout depends on outcomes that were not observed. Comparing their treatment-effect estimates is useful only if the models are fitted to a common analysis target. Otherwise, a difference might reflect changed participants, visits, or estimands rather than the missingness assumptions themselves.

This tutorial uses a hypothetical randomized trial with a symptom score measured at baseline and weeks 4, 8, and 12. Lower scores indicate fewer symptoms. The primary contrast is the week-12 mean score in the treatment arm minus the week-12 mean score in the control arm, for all randomized participants. Thus a negative contrast favors treatment on this score. The comparison concerns how that contrast changes across plausible MNAR assumptions; it does not establish which assumption is true.

Set Up the Candidate Models Before Fitting

For participant \(i\), let \(Y_i\) denote the vector of scheduled symptom scores, \(R_i\) the corresponding vector of observation indicators, and \(X_i\) treatment assignment and baseline covariates. An entry of \(R_i\) is 1 when the corresponding score is observed and 0 when it is missing. A selection model has an outcome distribution \(f_\theta(Y_i\mid X_i)\) and a missingness distribution \(P_\psi(R_i\mid Y_i,X_i)\).

The models in a comparison should be specified in enough detail to identify which outcomes enter the selection part, whether the outcome-dependence parameters are shared across treatment arms, and which baseline covariates are included. For example, three candidate MNAR models might use: (A) current outcome in the dropout model with a common outcome-dependence slope across arms; (B) current outcome with separate slopes by treatment arm; and (C) current and previous outcomes, allowing dependence on recent outcome history. The models need not be nested, but each must define a coherent joint model for the outcomes and missingness.

Important assumptions include a correctly specified outcome distribution, a correctly specified selection probability for each candidate, and a consistent treatment-effect estimand across fits. The selection dependence on unseen outcomes is not identified by the observed data alone. Consequently, the MNAR terms are assumptions to examine through sensitivity analysis, not quantities that observed-data fit can verify directly.

For each candidate, the observed-data likelihood integrates over missing outcome values. If \(Y_{i,\mathrm{obs}}\) and \(Y_{i,\mathrm{mis}}\) denote the observed and missing components, respectively, the contribution is

For discrete missing outcomes, the integral is replaced by a sum. The likelihood includes the observed outcomes and missingness indicators; it is not the likelihood from a complete-case analysis. The models can then be compared on the same observed records, provided the likelihoods are evaluated consistently and the same data contribution is used for every model.

Fit and Compare on a Common Basis

Before inspecting rankings, check that each candidate converged, that the likelihood is evaluated on the same sample, and that parameter counts include the outcome and selection components. A useful basic criterion is the Akaike information criterion, \(AIC=-2\ell+k\), where \(\ell\) is the maximized observed-data log likelihood and \(k\) is the number of estimated parameters. Its penalty discourages selecting a model only because it has more flexibility. It is a relative criterion among the models considered, not a test of whether any candidate is the true MNAR mechanism.

The comparison should also use diagnostics tied to what can be checked. Examples include observed-data residual checks, calibration of predicted dropout frequencies, and predictive checks for outcomes that are observed. These checks can reveal contradictions between a model and observed information. Agreement with observed data cannot, by itself, confirm how participants with missing outcomes would have behaved.

If likelihood-based criteria are used, a useful justification for alignment is direct: AIC compares \(-2\ell\) plus a parameter penalty, so a difference in \(-2\ell\) has an interpretable model-comparison role only when the \(\ell\) values refer to the same observed data and likelihood basis. If one fit omits participants or visits, its likelihood value is not directly comparable in this way. If models are Bayesian, evidence comparisons additionally depend on the specified priors; Bayesian model evidence and prior sensitivity are relevant considerations, not interchangeable with AIC.

Inspect Diagnostics, Not Just a Ranking

A diagnostic display should make the comparison visible. For instance, plot observed dropout proportions against model-predicted dropout probabilities across groups defined using information available for all participants, or display observed-data residual summaries by visit and treatment arm. Label the axes, show uncertainty where feasible, and identify which summaries are observed and which are model-predicted. A striking mismatch is a reason to investigate specification, coding, or fit; a close match is not evidence that the unobserved outcome dependence is correct.

Check the computational diagnostics too. Warning signs include nonconvergence, boundary estimates, very large standard errors for outcome-dependence parameters, sensitivity to starting values, and implausible predicted trajectories or dropout probabilities. These can indicate weak information, a poorly identified or over-flexible specification, or implementation problems. If different candidates are fit with different integration accuracy or convergence tolerances, apparent fit differences may be numerical rather than substantive.

Compare Treatment Contrasts in Context

For every candidate, report the same estimand in the same units and with uncertainty. Do not compare coefficients that represent different visits, populations, or conditioning sets and call their difference sensitivity to MNAR assumptions. A treatment-effect contrast and its standard error are model-based: their interpretation is conditional on the candidate model and its assumptions.

A cautious synthesis separates three statements: how well candidates reproduce observable features, how their treatment estimates vary, and what assumptions generate that variation. If estimates materially disagree, report the disagreement and which assumptions produce it. Model envelopes and treatment-effect robustness summaries can communicate that sensitivity, but neither removes the untestable component of MNAR modeling.

Common Mistakes and Limitations

• Ranking as truth. The wrong reasoning is “the model with the smallest AIC is the true dropout model.” AIC ranks relative expected predictive performance under its conditions; it cannot identify dependence on outcomes that were never observed.
• Comparing unlike fits. A likelihood from a reduced sample or a different set of visits cannot be treated as directly comparable to one using all scheduled observations. Refit on a common analysis basis or explain why the comparison is not valid.
• Calling an observed-data check an MNAR test. Diagnostics can reveal poor agreement with observed information, but cannot directly check the missing outcomes or verify the selection assumption.
• Ignoring estimation uncertainty. A change in point estimates across models is a sensitivity finding. It is not automatically a statistically significant difference between effects, and overlapping confidence intervals are not a formal test of model equivalence.

A further limitation is that candidate choice can depend on the scientific rationale for each selection model, not only on numerical criteria. The outcome and dropout models may be misspecified in several ways at once, and flexible selection terms can be weakly identified. The earlier material on information criteria, Bayesian model evidence, observed-data residual checks, and treatment-effect robustness provides methods for related parts of this assessment; none makes the MNAR assumption empirically testable from observed data alone.