Cross-Validation for MNAR Models and Its Limits

What Cross-Validation Can Compare

Cross-validation estimates how well a fitted model predicts data that were not used to fit it. For missing-not-at-random (MNAR) models, the phrase “predicts the data” needs care: the held-out outcomes that can be directly checked are the outcomes that were actually observed. A model can predict those well without correctly describing outcomes that were missing.

The comparison is meaningful only after the predictive target is specified. One target is the joint observed record, including the missingness pattern and all observed outcomes. Another is an outcome conditional on its having been observed, such as \(Y\mid R=1,X\), where \(R=1\) indicates observation and \(X\) denotes covariates. These targets answer different predictive questions. A joint score evaluates predictions of both the observation pattern and the observed values; a conditional outcome score evaluates observed values among records with the specified observation pattern.

For example, let \(Y_i^{\mathrm{obs}}\) denote the observed components of subject \(i\)'s outcome vector, and let \(R_i\) encode which components are observed. With covariates \(X_i\) and training data \(D_{-k}\), a joint log predictive score for subject \(i\) in held-out fold \(k\) is

Here \(M\) labels the candidate model, and \(n\) is the number of held-out observational units. Larger average log scores are better: a model assigns greater predictive probability to the observed records. The probability in this expression is an observed-data predictive probability. In a likelihood-based MNAR model, it accounts for unobserved outcome components by integrating them out, rather than pretending that they were observed. If the goal is instead to score outcomes conditional on observation, use \(\log p_M(Y_i^{\mathrm{obs}}\mid R_i,X_i,D_{-k})\) and state that target explicitly.

Why the Log Score Is Useful

The log score rewards predictive distributions that put high probability on what happens and penalizes confident predictions that assign it very low probability. Its justification can be seen by considering a discrete observable record \(Z\), with true distribution \(q(z)\) and a candidate predictive distribution \(p(z)\). The expected log score under the true distribution is \(\sum_z q(z)\log p(z)\). Subtracting the score of the true distribution gives

The final inequality follows from nonnegativity of Kullback–Leibler divergence, with equality when \(p=q\) on the support of \(q\). Thus, in expectation, the log score favors the correct distribution for the target being scored. It does not say that a finite cross-validation sample will rank models correctly, nor that a model with the best score is true. Most importantly, the result concerns the distribution of the scored observable records, not an unobserved-outcome distribution that the score never tests.

For continuous outcomes, the analogous score uses a predictive density evaluated at the held-out value. Density scores depend on the measurement scale, so comparisons require a consistent outcome representation and density convention across candidates. Scores from different targets or incompatible scales should not be compared as if they measured the same prediction task.

Designing Folds and Fitting Each Candidate

The split must match the unit to which predictions are intended to generalize. If each subject contributes repeated visits, randomly placing visits from one subject in both training and test sets can let information about that subject leak into the fit. For prediction to new subjects, put all visits from a subject in the same fold. Likewise, use site-level or cluster-level folds when the target is prediction at new sites or clusters. The split should reflect the scientific use of the prediction, not merely make the folds convenient.

Within each fold, fit the candidate using only the training data. Any data-dependent preprocessing, variable selection, tuning, or imputation used to construct predictions must also be learned from that training data alone. Otherwise, held-out information influences the prediction before it is scored. For a Bayesian candidate, the held-out predictive distribution averages over parameter uncertainty given the training data. For a frequentist candidate, the prediction may instead use fitted parameter estimates; whichever convention is used should be consistent across candidates.

An MNAR model generally specifies both an outcome distribution and a model for observation or dropout, with dependence between them. For a joint observed-record score, the held-out probability must be computed under the candidate’s observed-data distribution: latent missing components are integrated out, and the probability of the recorded observation pattern is included. A conditional outcome score has a different interpretation and may not assess whether the model predicts the observation pattern. Neither choice makes the missing outcomes in the held-out records observable.

Reading Score Differences and Diagnosing Failure

Cross-validation scores are estimates with sampling variability. Since candidates are evaluated on the same held-out units, retain paired score differences, rather than comparing unrelated averages. A few unusually influential records can dominate a log score, especially when one model assigns an observed record a very small probability. Inspect fold-specific and record-specific contributions, and check that all scored probabilities or densities are finite and computed for the same target.

Other useful diagnostics include checking whether fold sizes and missingness patterns are reasonably represented, whether a candidate’s score depends heavily on one fold, and whether a model failed to fit in some training splits. A model that produces extreme predictions or numerical integration failures may have unstable predictive behavior even if its overall average looks favorable. Report the split unit, the scored target, the aggregation rule, and the variability of paired differences.

Cross-validation complements, rather than replaces, other checks. Artificial-masking calibration evaluates predictions for outcomes deliberately hidden from records where those outcomes were observed; it provides a different and limited check because the artificially hidden cases may not resemble genuinely missing cases. Residual and posterior predictive checks examine aspects of the observed data under the fitted model. Cross-validation evaluates held-out prediction for its chosen observable target. None of these procedures directly verifies an MNAR assumption about outcomes that are never observed.

Alternatives and Limits

A single held-out split is simple but can depend strongly on which records were selected for validation. Repeated or \(K\)-fold cross-validation uses the data more broadly, though overlapping training sets mean fold scores are not automatically independent replicates. Leave-one-out cross-validation reduces the size of each held-out set but can be computationally demanding and may estimate a target that differs from prediction using a substantially smaller training sample. The split design and target remain more important than the number of folds.

The earlier material on artificial-masking calibration addresses a useful prediction check for outcomes whose values are known before masking. Cross-validation generalizes held-out evaluation to observed records, including their observation patterns when that is the target. Information criteria and Bayesian model evidence are other model-comparison approaches, but they answer different questions and do not turn observed data into evidence about unidentified missing-outcome distributions.

The central limitation is identification. If alternative MNAR specifications fit the same observable distribution but imply different distributions for unobserved outcomes, a score based only on observable records cannot determine which unobserved distribution is correct. Cross-validation can still reveal poor predictive performance on observable data and can help compare candidates for that task. It is not a validation certificate for the extrapolation from observed to missing outcomes, and it does not establish that the highest-scoring candidate is the true mechanism.