Quantifying Treatment-Effect Robustness Across MNAR Models

What Does Treatment-Effect Robustness Mean?

A treatment-effect estimate can change when the assumed MNAR mechanism changes, even if the same participants and observed data are used. Robustness across models asks how much the estimated effect varies over a prespecified set of plausible candidate models. It does not ask how much an estimate would vary across repeated samples under one model. Those are different sources of variation and require different summaries.

The comparison is meaningful only if every model targets the same treatment effect: the same outcome, population, time point, treatment contrast, and measurement scale. For example, estimates of a treatment-control difference in mean symptom score at week 12 can be compared directly if each model defines that same contrast. An odds ratio from one model and a mean difference from another cannot be summarized by their numerical range as though they were the same quantity.

This comparison presumes that the candidate models are fitted to the same analysis data and that their estimates refer to the same estimand. It also presumes that the models have been fitted successfully: estimates should be finite, and numerical convergence and standard-error calculations should be checked. The MNAR assumptions that distinguish candidates must be stated and scientifically interpretable. Observed-data fit alone cannot establish that an unobserved-outcome assumption is correct.

Range and Spread Across Models

Two simple summaries show different features of the candidate estimates. The range gives the lowest and highest estimate, and its width gives the full span. A root-mean-square spread gives a summary of typical distance from the candidate-set mean. Both are in the units of the treatment effect.

Here \(\bar{\theta}_{\mathrm{cand}}\) is the equally weighted arithmetic mean of the candidate estimates, and \(D_M\) is their root-mean-square spread about that mean. Equal weighting is simply a transparent descriptive convention; it does not imply that the candidate models are equally probable. The divisor is \(M\), because these summaries describe the specified finite set rather than estimate a variance in a larger population of models.

The spread has a useful algebraic form. Expanding each squared deviation gives \(\sum_m(\hat{\theta}_m-\bar{\theta}_{\mathrm{cand}})^2 =\sum_m\hat{\theta}_m^2-2\bar{\theta}_{\mathrm{cand}}\sum_m\hat{\theta}_m+M\bar{\theta}_{\mathrm{cand}}^2\). Since \(\sum_m\hat{\theta}_m=M\bar{\theta}_{\mathrm{cand}}\), this becomes \(\sum_m\hat{\theta}_m^2-M\bar{\theta}_{\mathrm{cand}}^2\). Thus,

This identity can simplify calculations, but the original definition makes the interpretation clearer. Both \(R\) and \(D_M\) are unchanged if the same constant is added to every candidate estimate, and both are multiplied by the absolute value of a common unit conversion factor. Neither summary accounts for sampling variability within a model.

Keep Within-Model Uncertainty Separate

Each model may also produce a standard error and an interval for its own estimate. For example, a large-sample Wald interval is \(\hat{\theta}_m\pm z_{0.975}SE_m\), where \(z_{0.975}\) is approximately 1.96. This interval has its usual repeated-sampling interpretation only when the model-based estimator is approximately normally distributed and \(SE_m\) is an appropriate estimate of its sampling standard deviation under model \(m\). It describes uncertainty conditional on that model’s assumptions; it does not measure how results change across candidate MNAR assumptions.

Because the candidate models use the same observed data, their estimates are generally dependent. It would therefore be incorrect to treat \(D_M\) as a standard error, calculate a conventional test from it, or interpret it as the uncertainty of an average over models. These summaries have no sampling probability attached to them. A model-specific confidence interval also does not account for uncertainty about which MNAR assumptions are suitable.

Interpret the Pattern, Not Just One Number

A range or spread is most useful alongside the individual estimates and a contextual description of their implications. If positive values favor treatment and a clinically important effect is defined by a prespecified threshold \(\Delta\), report whether estimates remain on the same side of zero and whether they remain on the same side of \(\Delta\). A count or fraction of candidate estimates meeting a criterion is descriptive of this candidate set; it is not the probability that the criterion is true.

The sign and threshold checks answer different questions. Estimates may all be positive but vary from negligible to clinically important. Or estimates may be tightly clustered but all lie close to a decision threshold. State the effect units and the direction that favors treatment so the range is interpretable in context.

Diagnostics, Alternatives, and Limitations

Before interpreting the summaries, inspect the candidate estimates and their fitting diagnostics. A numerical extreme may reflect a scientifically important extrapolation assumption, but it may also signal failed convergence, a boundary estimate, sparse information, or an unstable standard error. Investigate the fit rather than automatically removing the result. If a candidate is excluded, explain the prespecified or diagnostic reason and show how the comparison depends on that decision.

• Check comparability. Confirm the same treatment contrast, population, outcome time, and units for every estimate.
• Check fitting quality. Review convergence, parameter boundaries, warnings, and whether standard errors are available and plausible.
• Inspect the full list. A range is driven by its two extremes; report the estimates as well as a range or spread.
• Check candidate-set dependence. State how candidates were chosen and whether plausible alternatives change the direction or practical interpretation.
• Separate questions. Report model-specific standard errors or intervals for sampling uncertainty, and descriptive across-model summaries for assumption sensitivity.

These summaries complement, rather than replace, approaches that serve different purposes. Information criteria, cross-validation, and Bayesian model evidence address model comparison or predictive support under their respective assumptions; they do not make a candidate-set range a probability distribution. A single-model analysis provides inference conditional on one selected model, while a robustness summary displays how estimates differ across the specified alternatives. Model averaging, when justified, requires explicit weights and a defined inferential target; the unweighted candidate mean here is not model averaging.

A narrow range does not show that all plausible MNAR assumptions have been considered, and a wide range does not identify which assumption is best. The result depends on the candidates included, the common estimand, and the observed data. Robustness is therefore conditional: a statement such as “the estimated effect remains positive across these four prespecified models” is more precise than an unqualified claim that the effect is robust.