Why Summarize Several MNAR Models With an Envelope?
A sensitivity analysis may produce several estimates of the same outcome prediction or treatment effect because each candidate model makes a different assumption about the relationship between unobserved outcomes and dropout. A model envelope gives a compact summary of the spread of those estimates: its lower and upper bounds are the smallest and largest estimates among the selected models. It answers a focused question: how far apart do these candidate assumptions place the target quantity?
An envelope is useful for communicating sensitivity, but it does not establish which model is correct. In particular, the bounds are not automatically a confidence interval, a probability range for the true effect, or a guarantee that all plausible values are covered. They summarize a specified set of fitted models. Their interpretation therefore depends on defining the target consistently and choosing candidate MNAR assumptions with care.
Definition and Conditions for a Meaningful Envelope
The estimates must refer to the same scientific target. For example, comparing treatment effects at different follow-up times, in different target populations, or on different scales does not produce an envelope for one common effect. Nor is it meaningful to compare a model's treatment contrast with another model's treated-group mean and call the range an effect envelope.
Before constructing the envelope, state the target, the candidate-model set, and the basis for including those models. The candidates should address scientifically defensible alternatives about missing outcomes while using a common analysis population, outcome definition, and estimand. Each model must also be fit and summarized consistently. Observed-data checks can expose poor fit, but they cannot determine the unobserved outcomes or verify an MNAR assumption from observed data alone.
A model set should not be chosen merely because its extremes produce a desired conclusion. Sensitivity assumptions may be informed by subject-matter knowledge, external information, or a prespecified range of plausible departures. The envelope is conditional on that choice. Adding a more extreme candidate can widen the envelope; removing one can narrow it without changing any of the remaining estimates.
Constructing and Interpreting the Bounds
For a finite set of models, construction is direct: calculate the target estimate under each candidate, then identify the minimum and maximum. The same procedure applies to a predicted mean at a specified time, a population outcome prediction, or a treatment-effect contrast, provided that all candidate estimates target the same quantity. Always report the candidate estimates as well as the bounds; the endpoints alone hide whether the models cluster near one another or are spread throughout the range.
Name the outcome or contrast, timepoint, population, and scale. For a treatment effect, state the treatment-minus-control direction.
Describe which MNAR assumptions differ and why each candidate belongs in the sensitivity analysis.
Fit each candidate using the same observed data and calculate the same target quantity from each fitted model.
Take the minimum and maximum, report the width, and state explicitly that the bounds summarize this model set rather than a confidence interval.
The justification for the bounds is elementary but important. For every candidate \(m\in\mathcal M\), the definition of minimum and maximum gives $$ \hat\theta_{\min}\leq \hat\theta_m\leq \hat\theta_{\max}. $$ Thus every candidate point estimate lies inside the envelope. This fact makes no claim about an estimate from a model outside \(\mathcal M\), nor about a sampling distribution or the unknown true value.
For a continuously varying sensitivity parameter, the same idea can be expressed as the minimum and maximum of the estimate over a specified parameter range. In that setting, evaluating only a few grid points is not enough to guarantee the true extrema unless the function's behavior is known. If the estimate is not known to vary monotonically, inspect the interior of the range or use an appropriate numerical optimization; otherwise the reported bounds may miss an extremum.
Worked Example: Enveloping Week-12 Outcome Predictions
Hypothetical data. Suppose three defensible MNAR candidate models are fit to a trial and each predicts the mean week-12 symptom score, in points, for the same target population. The predictions are 6.2 points under model A, 5.7 under model B, and 4.9 under model C. Lower scores indicate fewer symptoms. Assume the models use the same score definition, target population, and follow-up time; their differences concern assumptions about unobserved outcomes after dropout.
The smallest prediction is 4.9 and the largest is 6.2, so $$ \hat\mu_{\min}=4.9,\qquad \hat\mu_{\max}=6.2,\qquad \text{width}=6.2-4.9=1.3\text{ points}. $$
The model envelope is \([4.9,6.2]\) points. Across these three specified assumptions, the predicted mean ranges over 1.3 points. This does not mean that the true mean has a probability of being in that range, and it does not include sampling uncertainty unless that uncertainty is separately incorporated. The direction of the symptom scale matters too: the lower prediction represents fewer symptoms, not necessarily a worse outcome.
The common target and scale make the comparison interpretable. If one candidate instead predicted the mean among completers while the others predicted the full target population, the three numbers would not estimate the same quantity and should not be combined into this envelope.
Worked Example: An Envelope for a Treatment Effect
Hypothetical data. Consider a trial in which higher outcome values are better. Three candidate MNAR models yield estimated treatment-minus-control mean differences at the prespecified endpoint of 2.4, 1.1, and \(-0.3\) outcome units. All three estimates use the same endpoint, scale, and target population, and differ in their assumptions about outcomes after dropout.
The bounds and width are $$ \hat\theta_{\min}=-0.3,\qquad \hat\theta_{\max}=2.4,\qquad \text{width}=2.4-(-0.3)=2.7\text{ outcome units}. $$
The envelope is \([-0.3,2.4]\) outcome units. Two candidate estimates are positive and one is slightly negative, so the direction of the estimated treatment effect is not robust across this model set. The width of 2.7 units measures the spread of the candidate point estimates; it is not a standard error or a confidence-interval width. It would be inaccurate to say that the treatment effect is known to lie between \(-0.3\) and 2.4 units.
The treatment contrast is consistently defined as treatment minus control, so a positive value favors treatment on this higher-is-better scale. That sign interpretation would reverse for a lower-is-better outcome or if the contrast direction were reversed.
Envelope Bounds Are Not Confidence Limits
A confidence interval addresses sampling uncertainty under a specified inferential procedure and model. A model envelope, as defined here, addresses variation among point estimates from a selected set of MNAR assumptions. These are different sources of uncertainty. A narrow envelope may accompany imprecise estimates, and a wide envelope may accompany estimates that are individually precise.
For example, one could calculate a confidence interval separately under each candidate model. The union of those intervals may convey both within-model sampling uncertainty and between-model variation, but its interpretation and coverage depend on the construction and assumptions. It is not automatically a standard confidence interval with a guaranteed coverage level. Do not label a point-estimate envelope a “95% interval” unless an explicit procedure has been defined that provides that inferential interpretation.
Worked Example: Robustness to a Clinical Threshold
Hypothetical data. Suppose a positive treatment effect is beneficial, and a clinical team has specified 1.0 outcome unit as a threshold for a clinically important benefit. Three plausible MNAR models estimate the same treatment contrast as 1.6, 1.2, and 0.7 units.
The envelope has bounds \(0.7\) and \(1.6\) units, with width $$ 1.6-0.7=0.9\text{ units}. $$ All three point estimates are positive, so the direction of the estimated effect is stable across these candidates. But the lower bound, 0.7, is below the 1.0-unit threshold, while the upper bound, 1.6, is above it. Therefore, the conclusion that the effect reaches the specified threshold is not stable across this model set.
This comparison is meaningful because the threshold is on the same scale as the effect and the candidate estimates share an estimand. It does not establish whether the true effect is above the threshold; neither a model envelope nor a comparison of point estimates alone accounts for sampling uncertainty.
Diagnostics, Alternatives, and Limitations
An envelope is only as informative as its inputs. A useful diagnostic is to inspect the full candidate-estimate table alongside the bounds. Check that extreme estimates correspond to the intended sensitivity assumptions, rather than a coding error, a changed target population, a different estimand, or an unstable fit. Compare candidate models with relevant observed-data diagnostics and review convergence and plausibility. Good observed-data fit is useful, but it does not validate a model's assumptions about missing outcomes.
Also check the model set itself. If the range is based on a coarse grid for a continuous sensitivity parameter, verify that the apparent extrema are not artifacts of sparse evaluation. If the envelope is unexpectedly narrow, ask whether the chosen assumptions genuinely span the intended uncertainty. If it is very wide, identify which assumptions drive the endpoints rather than presenting the width as self-explanatory.
| Summary | What it communicates | What it does not establish |
|---|---|---|
| One candidate estimate | The target estimate under one specified model | Robustness to alternative MNAR assumptions |
| Model envelope of point estimates | The range across the selected candidate models | Sampling uncertainty or coverage of the true value |
| Model averaging | A combined estimate using specified model weights | The same quantity as the minimum-to-maximum range |
| Confidence interval under one model | Sampling uncertainty under that model and procedure | Robustness to alternative MNAR assumptions |
A single-model analysis is simpler but hides sensitivity to alternative assumptions. Model averaging combines estimates according to specified weights and answers a different question; an envelope does not assign probabilities or relative credibility to its candidates. A confidence interval under one model can be valuable for sampling uncertainty, but it does not replace an assessment of model sensitivity. These summaries can complement one another if each is labeled and interpreted accurately.
The main limitation is dependence on the candidate set. The envelope includes every selected point estimate by construction, but may omit values from plausible models that were not considered. It is not a worst-case bound over all possible MNAR mechanisms. Nor does the envelope rank models: the endpoint estimates are not necessarily the least or most credible candidates. Report the assumptions that define the set, the estimates for each candidate, the target and scale, and the bounds. Treat the result as a sensitivity summary, not as a substitute for scientific judgment or inferential uncertainty.
Common Mistake
Check Your Understanding
Answer each question using the definitions and distinctions in this tutorial. Then compare your reasoning with the solutions.
- Three models estimate a common outcome mean as 8.1, 7.4, and 9.0 units. Give the envelope and its width.
- Why is an envelope not meaningful if one model estimates the full-population treatment effect and another estimates the treatment effect among completers?
- Candidate treatment-effect estimates are \(-0.2\), 0.6, and 1.3 units. What can be said about the direction of the estimates across this set?
- A researcher calls the range of three candidate point estimates a 95% confidence interval because all three estimates fall inside it. Identify the error.
- A sensitivity estimate is evaluated at five points in a continuous parameter range, and the smallest grid value is reported as the lower envelope bound. What should be checked before treating it as the true minimum over that range?
Solutions. (1) The envelope is \([7.4,9.0]\) units, and its width is \(9.0-7.4=1.6\) units. (2) The estimates target different quantities, so their range does not summarize model variation for one common estimand. (3) The estimated direction is not stable: two estimates are positive and one is negative. This says nothing by itself about sampling uncertainty or the true effect. (4) Containment of the calculated point estimates does not give the range a confidence level; a confidence procedure with justified coverage is needed for that interpretation. (5) Check whether the estimate varies monotonically across the range. If not, examine interior values or optimize over the range, since a coarse grid may miss a smaller value.