What Good Fit Can—and Cannot—Show
A model for incomplete data can reproduce the observed outcome distribution and the dropout rate extremely well, yet still make a poor or unsupported prediction about outcomes that were not observed. This is especially important under missing not at random (MNAR): after conditioning on recorded covariates, the chance an outcome is observed may depend on the outcome’s unobserved value. The missing outcomes cannot then be checked directly against the model’s predictions using the observed data alone.
The key distinction is between a model’s observable implications and the assumptions it uses to extend those implications to missing outcomes. Fit diagnostics can reveal some observable implications that do not match the data. But a good fit does not prove that an extrapolation is correct when different extrapolations are observationally indistinguishable.
The earlier material on observed-data distributions distinguishes information in the data from restrictions imposed on the complete-data model. Here, the practical consequence is central: observed outcomes reveal the outcome distribution among respondents, and response indicators reveal who responded. They do not, by themselves, reveal the distribution of \(Y\) among nonrespondents.
The Observed Likelihood Leaves an Extrapolation Unchecked
For clarity, write \(\rho(x)=P(R=1\mid X=x)\), let \(f_1(y\mid x)\) denote the outcome distribution among respondents, and let \(g(y\mid x)\) denote the outcome distribution among nonrespondents. The observed-data distribution has the following contributions:
The second contribution contains no \(Y\) for a nonrespondent. Consequently, \(g(y\mid x)\) does not appear in these observed-data probabilities. If two full-data models use the same distribution of \(X\), the same response probabilities, and the same respondent outcome distributions, but different \(g\)'s, they have the same observed-data distribution. Any likelihood or fit statistic based only on the observed data is therefore identical for those models.
The distinction matters for a target such as the population mean. Its value under a specified \(g\) is
The first conditional mean is informed by observed outcomes. The second depends on the nonrespondent distribution chosen by the model. Changing that distribution can change the population mean without changing the observed likelihood. This is not a computational defect: it is a limit on what the observed data can establish.
Worked Example: Two Binary MNAR Models With the Same Fit
Worked Example: Two Binary MNAR Models With the Same Fit
Hypothetical data: In a sample of 100 people, \(Y\) is a binary outcome. It is observed for 60 people; among them, 30 have \(Y=1\) and 30 have \(Y=0\). The other 40 outcomes are missing. Consider two candidate full-data models that preserve these observed counts.
In Model A, suppose 8 of the 40 missing outcomes equal 1, so \(P(Y=1\mid R=0)=8/40=0.20\). In Model B, suppose 32 of the 40 missing outcomes equal 1, so \(P(Y=1\mid R=0)=32/40=0.80\). Both models have the same response rate, \(60/100=0.60\), and the same respondent outcome proportion, \(30/60=0.50\).
The implied full-sample proportions are different. Under Model A,
Under Model B,
Thus, the models imply population proportions of 0.38 and 0.62, respectively—a difference of 0.24, or 24 percentage points. Both reproduce exactly the same observed response rate and the same observed outcome counts.
These are genuinely MNAR extensions, not merely different labels for a common response process. For example, under Model A the full-sample proportion with \(Y=1\) is \(38/100\), and the response probability among those with \(Y=1\) is \(30/38\approx0.7895\). Among those with \(Y=0\), it is \(30/62\approx0.4839\). Those response probabilities differ by outcome value. Under Model B they are \(30/62\approx0.4839\) and \(30/38\approx0.7895\), respectively, so dependence on \(Y\) remains but goes in the opposite direction.
The assumptions being compared are that the recorded counts are the same and that each proposed allocation of missing outcomes is a candidate full-data distribution. The missing outcomes are not observed, so these data cannot choose between the allocations. Any observed-data fit measure based on these counts is the same for both models; neither fit validates its own missing-outcome allocation.
Why the Nonidentification Persists
The general argument can be stated directly. Hold fixed \(P(X)\), \(\rho(x)\), and \(f_1(y\mid x)\), and choose any two candidate nonrespondent distributions \(g_A(y\mid x)\) and \(g_B(y\mid x)\). For a respondent, the observable probability contribution uses \(f_1\); for a nonrespondent, it uses only the probability of \(R=0\). Neither contribution uses \(g\). Therefore the two choices produce the same observed-data distribution. If the target depends on \(E_g(Y\mid X=x)\), however, it can differ between the choices.
This argument also explains why a larger sample does not, by itself, solve the problem. With more data, the response rate and respondent outcome distribution can be estimated more precisely. That can sharpen knowledge of the observable parts of the model, while leaving the nonrespondent outcome distribution unobserved. Precision about the observed-data distribution is not the same as information about an unrestricted MNAR extrapolation.
A parametric model may link the missing-outcome distribution to parameters estimated from observed data. That link can produce a definite prediction, but the prediction then relies on the assumed relationship. The model’s ability to return a unique estimate is not proof that the relationship is correct. If the link imposes restrictions that affect observable probabilities, those implications can be examined; restrictions that leave the observed distribution unchanged cannot be tested with that distribution alone.
Worked Example: Normal Outcomes and Different Extrapolations
Worked Example: Normal Outcomes and Different Extrapolations
Hypothetical setting: A continuous biomarker, measured in units, is observed for 75% of a population. Suppose the observed outcomes follow \(N(10,4)\), so their mean is 10 units and their variance is 4 square units. Compare two candidate extrapolations for the missing outcomes: Model A uses \(N(8,4)\), while Model B uses \(N(12,4)\).
Both candidates retain the same response rate and the same distribution for observed outcomes. They therefore give the same observed-data likelihood if their response models also agree. Yet the implied population means differ. Under Model A,
Under Model B,
The one-unit difference is due entirely to the assumed mean among missing outcomes. Each candidate is MNAR in this setting: its conditional outcome distribution differs between respondents and nonrespondents, so the response probability varies with the outcome under the implied joint distribution. The normal distributions and their variances are modeling assumptions for this illustration; the missing outcomes provide no direct data with which to verify either missing-group mean. A close fit to the observed normal distribution cannot settle the choice.
Worked Example: Good Fit Within Covariate Groups
Worked Example: Good Fit Within Covariate Groups
Hypothetical data: Consider 200 people divided equally between covariate groups A and B. The outcome is measured in score points. In group A, 80 of 100 outcomes are observed and their mean is 50 points. In group B, 40 of 100 are observed and their mean is 60 points. Suppose a model reproduces both response rates and both observed outcome distributions well.
Now compare two assignments for the missing-group means. Model A assigns means of 45 points in group A and 65 points in group B. The implied overall mean is
Model B assigns means of 65 points in group A and 45 points in group B. Its implied overall mean is
Both models preserve the same response counts and observed outcomes in each group, yet their overall means differ by 4.0 points. In both groups, each proposed missing-group mean differs from the observed-group mean, which is consistent with an MNAR extension conditional on group. A good fit to the observed data within each group is valuable: it checks that the model is not visibly failing on those data. But it does not tell us whether the missing mean in group A is 45 or 65, or whether the missing mean in group B is 65 or 45.
Diagnostics, Alternatives, and Limits
Diagnostics remain important because poor observable fit is a warning. A model that misses response rates across covariate groups, fails to describe the outcomes that are observed, or leaves systematic residual patterns should not be trusted merely because it offers an MNAR extrapolation. Useful checks include comparisons of observed and predicted response frequencies, residual checks among observed outcomes, and assessment of fit across relevant covariate strata. These checks can detect some model failures; they cannot reveal the outcomes of people whose outcomes are missing.
The same limitation applies to validation procedures that use only observed outcomes. For example, cross-validation can assess predictions when some observed outcomes are temporarily hidden. It tests prediction for cases drawn from the observed-outcome group under that procedure. It does not establish how well the model predicts outcomes among people whose outcomes were missing because their response process may differ.
It is useful to separate three situations:
- Observable misfit: the model fails to reproduce a feature of the observed data. This is evidence against the model’s observable implications.
- Observable compatibility: the model reproduces the observed distribution adequately. This supports the claim that it is compatible with the observed data, not that its missing-outcome predictions are correct.
- Untestable extrapolation: two or more assumptions about missing outcomes imply the same observed distribution but different targets. Fit to that distribution cannot distinguish them.
A MAR model, an MNAR model, and alternative MNAR specifications can be compared as modeling choices, but a better observed-data fit is not automatic evidence for the MNAR extrapolation. The comparison is informative when models make different observable predictions. If they fit the same observed distribution while differing only in unobserved outcomes, the observed-data fit cannot rank those extensions on that difference.
The limitation is not that all assumptions are forever unknowable. Additional information or design features may provide evidence about missing outcomes, and substantive restrictions may narrow the set of plausible extensions. But such information or restrictions must be stated and justified; they do not follow merely from a high likelihood, a small residual, or a good fit statistic. The appropriate conclusion from good fit is measured: the model is not contradicted by the observable features checked, while the MNAR extrapolation remains an assumption unless separately supported.
Common Pitfall
Correction: Matching observed outcomes and dropout rates checks observable implications. If different missing-outcome distributions produce the same observed-data distribution, those checks cannot distinguish their predictions. Report the fit evidence and the extrapolation assumption separately.
Assess whether the model describes response patterns and outcomes that were actually recorded, including across relevant covariates.
State which part of the prediction concerns outcomes with \(R=0\) and what assumption supplies that part.
Do not treat compatibility with observed data as validation of a restriction that leaves those data unchanged.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- If two models have identical response probabilities and identical outcome distributions among respondents, can their observed-data likelihoods distinguish different outcome distributions among nonrespondents? Why?
- A hypothetical binary-outcome sample has 50 observed cases, 20 with \(Y=1\), and 50 missing cases. One model assigns 10 missing outcomes to \(Y=1\); another assigns 40. Calculate the implied full-sample proportions under both models.
- Why does cross-validation that hides some observed outcomes fail to validate predictions for outcomes missing under an MNAR process?
- Suppose a model has a poor fit to observed response rates in one covariate group. What does that tell you, and what does it not tell you?
Solutions. (1) No. The observed likelihood uses the response probabilities and observed-outcome distributions, but not the distribution of \(Y\) among cases with \(R=0\); that distribution is absent from the observed data. (2) The total sample size is 100. The first proportion is \((20+10)/100=0.30\); the second is \((20+40)/100=0.60\). They differ by 0.30 despite sharing the same observed counts. (3) Temporarily hidden observed outcomes come from the group whose outcomes were observed. Under MNAR, their outcome distribution need not represent the distribution among nonrespondents, so successful prediction in that exercise does not establish prediction for the missing group. (4) The discrepancy is evidence that the model fails to describe an observable feature and should be investigated. It does not reveal the unobserved outcomes or identify which missing-outcome extrapolation is correct.