Model Fit Does Not Validate Missingness Assumptions

Good Fit Is Not Evidence for an Unseen Outcome Distribution

A model can reproduce every recorded outcome and every observed response rate, yet still rely on a missingness assumption that is wrong. The difficulty is not necessarily poor estimation or an inadequate fit statistic. Outcomes that were never recorded cannot directly show us how they would have been distributed. Different assumptions about those outcomes may therefore be compatible with exactly the same observed data.

This distinction matters for treatment effects. A sensitivity analysis can show that plausible assumptions lead to materially different estimates, even when the assumptions describe the observed data equally well. A good fit to observed data is useful evidence about the parts of a model that those data reveal. It is not, by itself, a test of an assumption about outcomes that are missing.

Definition: Let \(A\) denote treatment arm, \(Y\) the outcome of interest, and \(R\) an indicator equal to 1 when \(Y\) is observed and 0 when it is missing. An assumption about missingness is not identified by the observed data when distinct choices for the unobserved outcome distribution imply the same distribution for the data that are observed. A fit measure based only on observed data cannot distinguish those choices.

We will use a simple setting with two treatment arms and a continuous outcome. The treatment-effect estimand is the difference in population mean outcomes, treatment minus control. Assume treatment is randomized, the outcome is defined for every participant, and there is no interference between participants. Randomization supports a causal interpretation of the full-data arm contrast, subject to the usual consistency and study-design requirements. It does not, on its own, reveal outcomes missing after randomization.

For the calculations below, the reference assumption is that, within each arm, the mean outcome among participants with missing outcomes equals the mean among those with observed outcomes. In this no-covariate illustration, that is a mean-level version of the missing-at-random reference. It should not be confused with a universal statement that missingness is harmless. Conditional MAR analyses can use measured covariates, but they also need assumptions about how outcomes relate to those covariates and about the missing outcomes given them.

Where the Observed-Data Information Stops

A useful way to see the problem is to factor a complete-data distribution by the response pattern. Within arm \(a\), the factorization is

P(R=r\mid A=a)\, f(Y\mid A=a,R=r).

For participants with \(R=1\), the data reveal both their response pattern and their outcome; they inform \(P(R=1\mid A=a)\) and \(f(Y\mid A=a,R=1)\). For participants with \(R=0\), the data reveal the response pattern but not the outcome. They inform \(P(R=0\mid A=a)\), but not \(f(Y\mid A=a,R=0)\). The distribution of the missing outcomes must be supplied or restricted by an assumption.

For example, setting the missing-outcome distribution equal to the observed-outcome distribution is one restriction. Allowing missing outcomes to have a different mean is another. If both restrictions leave the observed response rate and observed-outcome distribution unchanged, the observed data cannot choose between them. A model-selection criterion calculated from the observed-data likelihood may distinguish models that imply different observed distributions; it cannot distinguish assumptions that imply the same observed distribution.

Key distinction: A model may fit the observed outcomes and response indicators well while its missingness assumption remains untestable from those data. Fit checks observable implications; sensitivity assumptions specify what the data do not identify.

Departures from a Reference Assumption

For a concise calculation, let \(r_a=P(R=1\mid A=a)\), and let \(m_{a1}=E(Y\mid A=a,R=1)\) be the observed mean in arm \(a\). Define the arm-specific mean departure

\delta_a=E(Y\mid A=a,R=0)-E(Y\mid A=a,R=1).

Thus \(\delta_a=0\) is the mean-equality reference assumption. A negative \(\delta_a\) means that the mean outcome among participants with missing outcomes is lower than the observed mean in that arm; a positive value means it is higher. Because \(r_a\) and \(m_{a1}\) are observed, but \(\delta_a\) is not generally identified, changing \(\delta_a\) can change the estimated arm mean without changing the observed-data fit.

The full arm mean is the weighted average of the means in the two response patterns. Substituting the definition of \(\delta_a\) gives

\begin{aligned} \mu_a &=r_a E(Y\mid A=a,R=1)+(1-r_a)E(Y\mid A=a,R=0)\\ &=r_a m_{a1}+(1-r_a)(m_{a1}+\delta_a)\\ &=m_{a1}+(1-r_a)\delta_a. \end{aligned}

Consequently, the treatment-minus-control contrast under a specified pair of departures is

\Delta(\delta_T,\delta_C) =(m_{T1}-m_{C1})+(1-r_T)\delta_T-(1-r_C)\delta_C.

This is an assumption-indexed calculation, not a way to estimate the departures from the observed outcomes. Its value is that it makes the effect of each assumption explicit: a departure matters in proportion to the fraction of its arm with missing outcomes.

Worked Example: Identical Observed Summaries, Different Treatment Effects

Hypothetical data. In a randomized trial, 100 participants are assigned to each arm. Outcomes are observed for 80 of 100 treatment participants and 70 of 100 control participants. The observed mean outcome is 12 units in the treatment arm and 10 units in the control arm. Assume these summaries are the same under each missingness assumption considered below.

Reference calculation. The response rates are \(r_T=0.80\) and \(r_C=0.70\). Under \(\delta_T=\delta_C=0\), the full-data arm means equal the observed means, so

\Delta(0,0)=(12-10)+(1-0.80)(0)-(1-0.70)(0)=2.00\text{ units}.

Alternative assumption. Suppose instead that treatment-arm missing outcomes average 6 units below the treatment observed mean, while control-arm missing outcomes average 4 units above the control observed mean. Then \(\delta_T=-6\) units and \(\delta_C=4\) units. The implied arm means are

\begin{aligned} \mu_T&=12+(1-0.80)(-6)=12-1.20=10.8\text{ units},\\ \mu_C&=10+(1-0.70)(4)=10+1.20=11.2\text{ units}. \end{aligned}

The implied treatment contrast is \(10.8-11.2=-0.400\) units. Thus the reference assumption gives a contrast of \(2.00\) units, while this alternative gives a contrast of \(-0.400\) units. The observed response rates and observed means are unchanged. The different conclusions arise entirely from assumptions about outcomes that were not recorded. These calculations establish what each assumption implies; they do not establish which assumption is true.

Why Fit Cannot Choose Between the Assumptions

The distinction can be expressed directly through the observed-data likelihood. For a participant in arm \(a\), an observed outcome contributes information about the outcome distribution in the \(R=1\) pattern. A missing outcome contributes no observed \(Y\) value: it contributes information that \(R=0\). In a pattern-mixture representation, the unobserved distribution \(f(Y\mid A=a,R=0)\) does not appear as an outcome-density contribution for a participant whose \(Y\) was not recorded. Its mean can change while the observed response rate and observed-outcome distribution stay fixed.

This is a form of observational equivalence: multiple full-data descriptions correspond to one observed-data description. A model with strong fit to the observed records can therefore support a useful description of those records without validating a particular extrapolation from responders to nonresponders. Greater sample size can estimate the observed response rates and observed outcome distributions more precisely. It does not, by itself, supply the missing outcomes or resolve this ambiguity.

Observed-data diagnostics remain important. They can reveal, for example, that a chosen model fails to represent observed outcome distributions, response rates, or relationships between observed outcomes and measured covariates. If observed outcomes vary over time, diagnostics can also reveal a poor fit to their recorded trajectory. Such failures matter: a model that cannot describe what was observed is not rescued by a plausible missingness assumption. But passing these diagnostics does not test whether the missing participants would have had the same outcomes as observed participants.

Worked Example: A Binary Outcome and Unseen Event Rates

Hypothetical data. Consider 100 participants per arm in a randomized trial. In the treatment arm, 80 outcomes are observed and 16 of those participants experience the event. In the control arm, 80 outcomes are observed and 24 experience the event. The observed event rates are \(16/80=0.200\) and \(24/80=0.300\), respectively.

Under a mean-equality reference for this binary outcome, use the observed event rate as the assumed event rate among participants with missing outcomes. Each arm then has an assumed full-data event rate equal to its observed rate: 0.200 in treatment and 0.300 in control. The treatment-minus-control risk difference is \(0.200-0.300=-0.100\), or a 10.0-percentage-point lower event rate in treatment.

Now consider an alternative in which the missing treatment participants have an event rate of 0.500 and the missing control participants have an event rate of 0.100. Each arm has 20 missing outcomes. The implied full-data event rates are

\begin{aligned} \hat p_T&=\frac{16+20(0.500)}{100}=\frac{26}{100}=0.260,\\ \hat p_C&=\frac{24+20(0.100)}{100}=\frac{26}{100}=0.260. \end{aligned}

The implied risk difference is \(0.260-0.260=0.000\). Both assumptions reproduce the same observed counts: 16 events among 80 observed treatment outcomes and 24 among 80 observed control outcomes. The unobserved event rates, which drive the contrast change, cannot be checked against those observed counts.

What Comparisons and Diagnostics Can Establish

Comparing assumptions is valuable precisely because fit alone cannot settle the missingness question. The comparison should keep the observed-data model and the target estimand clear, then vary the assumptions about the unobserved outcomes. The resulting estimates describe what follows under each assumption. They do not turn the best-fitting observed-data model into evidence that its missingness assumption is correct.

Observed-data fit checks assess whether the model represents recorded outcomes, response patterns, and measured covariate relationships adequately.
Sensitivity analyses assess how conclusions change across specified, substantively plausible assumptions about missing outcomes.
External information, when available and relevant, may help constrain the missing-outcome distribution. Its use requires its own assumptions about comparability and measurement.
Assumption-specific uncertainty intervals reflect sampling uncertainty conditional on the assumptions used. A narrow interval under one assumption does not show that the assumption is valid.

These approaches answer different questions. Diagnostics ask whether a model is in tension with observed data. Sensitivity analysis asks how results depend on assumptions that those data cannot determine. External information may add evidence about the unseen outcomes, but only to the extent that it is informative and applicable. No single fit statistic substitutes for this separation.

Worked Example: A Well-Fitting Covariate Model Still Needs an Assumption

Hypothetical data. Suppose a trial records a baseline risk category for every participant. Within each treatment arm and risk category, observed outcomes and response rates are adequately represented by a fitted model. However, outcomes are missing for some participants in both categories. The model is extended by assuming that, conditional on arm and risk category, the missing participants have the same mean outcome as participants whose outcomes are observed.

That conditional equality is an assumption about \(Y\) for \(R=0\), not a consequence of the model’s fit among participants with \(R=1\). A second model could retain exactly the same fitted observed response rates, observed outcome distributions, and covariate relationships, while assigning missing participants means 3 units lower than observed participants in the high-risk category and 1 unit higher in the low-risk category. The two models would agree on every recorded outcome and response indicator, yet they could imply different arm means if the categories have different missing fractions or treatment-arm composition.

The appropriate conclusion is not that one model fits better unless the models actually imply different observable distributions and that difference is assessed. Rather, compare the treatment-effect estimates under the two specified assumptions, explain why the departures are considered relevant, and state that observed-data fit does not adjudicate between them.

Common Pitfall and Practical Limits

Wrong reasoning: “The model has excellent fit to the observed outcomes and response rates, so the missing-at-random assumption is supported.” This confuses a model’s ability to reproduce observed information with evidence about the distribution of outcomes that were never observed.

Correction: State exactly which parts of the model are checked against observed data and which parts are assumptions about missing outcomes. Use sensitivity analysis to show the consequences of alternative assumptions. If all scenarios give similar conclusions, the result is less sensitive over the scenarios examined; it still does not prove that any one missingness assumption is correct.

There are limits to this approach. Sensitivity scenarios do not reveal the true missing outcomes merely by being numerous, and a wide or narrow range of results depends on which assumptions were included. Conversely, an implausible alternative is not informative just because it changes the estimate. Scenario choices need substantive justification, and uncertainty intervals conditional on scenarios should not be described as accounting for uncertainty about which missingness assumption is true unless that uncertainty has been explicitly modelled.

Key takeaway: Observed-data fit can challenge a model’s description of recorded data. It cannot, on its own, validate an assumption about the outcomes that were not recorded. When different plausible assumptions fit the observed data equally well but imply different treatment effects, report the estimates as assumption-dependent.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the continuous-outcome example, what treatment contrast follows from \(\delta_T=0\) and \(\delta_C=0\)?
With \(r_T=0.80\), \(r_C=0.70\), observed means 12 and 10, and departures \(\delta_T=-2\), \(\delta_C=1\), calculate the implied treatment contrast.
Why can a larger sample fail to identify the mean outcome among participants whose outcomes are missing?
Give one conclusion an observed-data diagnostic can support and one it cannot support.

Solutions. (1) The contrast is \(12-10=2.00\) units. (2) The implied means are \(\mu_T=12+0.20(-2)=11.6\) and \(\mu_C=10+0.30(1)=10.3\), so the contrast is \(11.6-10.3=1.30\) units. (3) A larger sample can estimate observed-data quantities more precisely, but it still does not reveal the unrecorded outcomes; distinct missing-outcome distributions can remain compatible with the observed data. (4) A diagnostic can indicate that a model poorly represents observed outcome distributions or response rates. It cannot establish that missing participants would have the same outcome distribution as observed participants.