Tracing Model Disagreement to Extrapolation Assumptions

Why Models Disagree

Two MNAR models can fit the observed data similarly and still produce different treatment-effect estimates. The reason is that they can make different predictions for outcomes that were not observed. To understand a disagreement, it is not enough to report that one model estimates a larger effect: we need to identify which unobserved outcomes differ, for whom, and under which assumptions.

This tutorial develops a way to trace that disagreement. The central comparison is not usually between raw model coefficients. It is between model-implied predictions for the same missing outcomes, conditional on the same observed information, and the contribution those prediction differences make to the treatment-effect contrast.

The intended target must be stated before comparing models. For example, in a randomized trial, \(\Delta_m\) may be the model-implied difference in mean outcomes under the two randomized assignments for the trial population. A causal interpretation additionally requires the usual design and consistency conditions for that estimand; the missing-data model alone does not establish causality.

The main attribution below assumes candidate models share the same target population, treatment-group pattern proportions, and observed-outcome component. Their difference is then attributable to their extrapolations for missing outcomes. If those shared components are not actually shared, the comparison needs to separate those differences as well.

A Weighted Decomposition of Disagreement

For clarity, first consider one target outcome and suppose \(H\) takes a finite set of values \(h\). Define \(w_{a,h}=P(H=h,R=0\mid A=a)\), the fraction of treatment group \(a\) whose outcome is missing in history stratum \(h\). Also define \(c_a=E\{Y I(R=1)\mid A=a\}\), the observed-outcome contribution to the group mean, with missing outcomes contributing zero to this expression. Since the observed and missing cases partition the group,

This expression is a standardization identity: observed outcomes contribute their actual weighted values, while the model supplies the conditional mean for each missing stratum. If models \(m\) and \(n\) have the same \(c_a\) and \(w_{a,h}\), subtract their treatment-effect contrasts. The shared observed terms cancel, leaving

Thus a prediction difference matters in proportion to the share of the treatment group to which it applies. A large difference in a rare missing pattern may contribute less to the overall contrast than a modest difference in a common pattern. The subtraction between arms also matters: if both models shift missing outcomes similarly in both groups, their treatment-effect estimates may remain close.

This identity gives a useful diagnostic sequence. First check whether observed-data contributions and pattern proportions are aligned. Then calculate model prediction differences within comparable strata. Finally weight those differences by the relevant missing fractions and subtract the control-arm contribution from the treatment-arm contribution.

Trace Disagreement Across Missing Patterns

A single missing fraction can conceal important differences in which outcomes are missing. In longitudinal settings, for example, participants may stop contributing outcomes at different visits. One model may imply a distinctive extrapolation for participants who leave early, while another may differ mainly for those observed until a later visit. The weighted decomposition can be applied pattern by pattern, using the observed history that defines each prediction stratum. This tutorial uses those pattern-specific contributions for attribution; it does not require a graph of post-dropout trajectories.

The comparison requires that each prediction refers to the same target outcome and a comparable history stratum. If one model conditions on current outcome and another on lagged outcome, compare their implied predictions for common histories rather than labeling their coefficients as if they had identical meanings. If one model lacks support for a history represented by the other, the comparison relies even more heavily on extrapolation and should be described accordingly.

Connect Prediction Differences to Model Parameters

Parameters can help explain why predictions differ, but parameter contrasts are meaningful only when the parameters have comparable definitions. A selection-model coefficient describing the association between an outcome and the probability of dropout is not automatically comparable to a pattern-mixture parameter describing a missing-versus-observed mean shift. Even coefficients with the same name can refer to different conditional distributions or scales.

A safer procedure is to state each parameter’s role, calculate the predictions it implies for a common set of histories, and then apply the weighted decomposition. In a simple pattern-mixture sensitivity model, define \(\delta_a\) as the difference between the mean outcome among missing and observed participants in arm \(a\), after conditioning on any specified common history. Then the missing mean is the corresponding observed mean plus \(\delta_a\). The \(\delta_a\) values are assumptions about unobserved outcomes, not quantities identified merely by fitting the observed data.

When Attribution Is and Is Not Clean

The decomposition is most informative when candidate models have been aligned on the observed-data components. If they use different covariate sets, outcome distributions, or fitted observed-data parameters, then the observed predictions or pattern proportions may also differ. In that case, a change in \(\Delta\) cannot be attributed entirely to extrapolation. Report the components that changed, or construct a controlled comparison that holds selected components fixed while varying the extrapolation assumption.

A useful extension is to add and subtract a common reference prediction. For any reference values \(c_a^*\) and \(w_{a,h}^*\), a difference between two fitted models can be organized into observed-component differences and prediction differences. The exact allocation can depend on which model supplies the reference weights, so state the chosen standardization. For the clean identity used above, the weights and observed components are identical across models; do not silently apply it when they are not.

The assumptions needed for a useful attribution are modest but important:

• The target outcome, treatment contrast, population, and history strata are defined consistently across candidates.
• The proportions \(w_{a,h}\) are estimated from observed data and have adequate support for the strata being compared.
• Each model’s conditional predictions \(q_m(a,h)\) are well-defined on those same strata.
• Any interpretation as a causal treatment effect is justified by the design or additional causal assumptions, rather than by the MNAR specification alone.

Failure modes include comparing coefficients on different scales, ignoring the missing fraction that weights a prediction difference, pooling dropout patterns that have different predictions, and treating a large difference in model estimates as proof that one model is wrong. Another warning is practical: if a stratum contains very few participants, its weight may be estimated imprecisely, and its conditional missing-outcome prediction may be weakly supported. The arithmetic decomposition can still be calculated, but its components should not be presented with more certainty than the data and assumptions justify.

This diagnostic has a different purpose from observed-data fit comparisons. Fit agreement asks how similarly models describe what was observed; prediction contrasts ask what they imply about outcomes not observed. Neither comparison alone validates an MNAR extrapolation, because the missing outcomes needed to check it are unavailable. A transparent analysis therefore reports the assumptions, the implied predictions, and their weighted contribution to the target contrast, without ranking a candidate as correct solely because of fit or because its treatment effect is preferred.