Comparing Treatment Effects Across MNAR Specifications

Why Compare Treatment Effects Across MNAR Specifications?

An MNAR model can fit the observed data while making assumptions about unobserved outcomes that the observed data cannot, by themselves, verify. Those assumptions matter because a treatment-effect estimate for the full analysis population includes outcomes for participants whose endpoint was not observed. Comparing treatment effects across candidate MNAR specifications makes the consequence of those assumptions visible: the question is not only whether models reproduce the observed data, but also whether their extrapolations lead to materially different conclusions about the treatment effect.

A fair comparison requires a common target. Hold fixed the outcome definition, the analysis population, the observed outcomes, and the treatment contrast. Then vary the MNAR specification and report how the resulting estimate changes. If those ingredients change at the same time, a difference between estimates cannot be attributed clearly to the missing-outcome assumptions.

Define the Estimand and the Calculation

Suppose a randomized study has two arms, treatment $a=1$ and control $a=0$, and a continuous endpoint measured at a prespecified follow-up. Assume this endpoint is defined for every participant in the analysis population. The target considered here is the difference in population mean endpoint values, treatment minus control. The example calculations use arm-specific means; they are not adjusted for covariates.

Definition: Under MNAR specification $g$, let $r_a$ be the fraction of participants in arm $a$ with an observed endpoint, $\bar{y}_{a,\mathrm{obs}}$ the mean among those with an observed endpoint, and $\bar{y}_{a,\mathrm{mis}}^{(g)}$ the model-implied mean among those with a missing endpoint. The estimated arm mean and treatment effect are $$ \hat{\mu}_a^{(g)} = r_a\bar{y}_{a,\mathrm{obs}} + (1-r_a)\bar{y}_{a,\mathrm{mis}}^{(g)}, \qquad \hat{\tau}^{(g)} = \hat{\mu}_1^{(g)}-\hat{\mu}_0^{(g)}. $$ The missing-outcome mean is an average over the missing participants in that arm under specification $g$; it need not be the same for every participant.

This calculation follows by splitting the arm’s full-population mean into the observed and missing portions. In each arm, the observed portion has population fraction $r_a$, and the missing portion has fraction $1-r_a$. Their weighted means therefore give the full-arm mean. Taking the treatment-minus-control difference gives the stated treatment-effect estimate. If a model predicts different missing outcomes for different participants, its predictions must first be averaged over the missing participants in that arm.

The calculation assumes the observed outcomes and missingness fractions are computed in the same defined analysis population for every candidate specification. The specifications must use the same endpoint and treatment contrast, and the model-implied missing-outcome means must be obtained under the candidate assumptions being compared. In a randomized study, randomization supports a treatment comparison, but it does not identify the unobserved outcomes. The treatment-effect interpretation also depends on the endpoint being meaningful and defined for the population; an outcome made undefined by an event such as death requires a separately justified estimand rather than routine missing-value imputation.

Worked Example: Calculating One Model-Specific Treatment Effect

Consider hypothetical data from a randomized study with 100 participants in each arm. The endpoint is a score in points, with higher scores representing better outcomes. In the control arm, 80 scores are observed and their mean is 50 points. In the treatment arm, 70 scores are observed and their mean is 58 points. Under candidate specification A, the model-implied means among participants with missing scores are 48 points for control and 54 points for treatment. Calculate the full-population treatment-effect estimate.

The observed fractions are $r_0=80/100=0.80$ and $r_1=70/100=0.70$. Thus the missing fractions are $0.20$ and $0.30$, respectively. The weighted arm means are

\hat{\mu}_0^{(A)} =0.80(50)+0.20(48) =40+9.6 =49.6\ \text{points},

\hat{\mu}_1^{(A)} =0.70(58)+0.30(54) =40.6+16.2 =56.8\ \text{points}.

Therefore,

\hat{\tau}^{(A)} =56.8-49.6 =7.2\ \text{points}.

Under specification A, the estimated mean endpoint score is 7.2 points higher in the treatment arm than in the control arm for the defined population. This is an estimate under the stated missing-outcome assumptions, not a claim that the same effect would follow under every MNAR model.

Compare Specifications Without Changing the Target

The observed fractions and observed means in this example are held fixed. Only the model-implied missing-outcome means change. This isolates the part of the calculation that depends on the MNAR specification.

Worked Example: Comparing Three Missing-Outcome Assumptions

Using the same hypothetical study data, compare three candidate specifications. A implies missing-outcome means of 48 points for control and 54 for treatment. B implies 45 for control and 50 for treatment. C implies 44 for control and 46 for treatment. Calculate each pair of arm means and each treatment effect.

For specification A, the calculation above gives control mean 49.6 points, treatment mean 56.8 points, and treatment effect 7.2 points.

For specification B, the arm means are

\hat{\mu}_0^{(B)} =0.80(50)+0.20(45) =40+9 =49.0\ \text{points},

\hat{\mu}_1^{(B)} =0.70(58)+0.30(50) =40.6+15 =55.6\ \text{points}.

Thus $\hat{\tau}^{(B)}=55.6-49.0=6.6$ points.

For specification C,

\hat{\mu}_0^{(C)} =0.80(50)+0.20(44) =40+8.8 =48.8\ \text{points},

\hat{\mu}_1^{(C)} =0.70(58)+0.30(46) =40.6+13.8 =54.4\ \text{points}.

Thus $\hat{\tau}^{(C)}=54.4-48.8=5.6$ points. The comparison is:

Specification	Control missing mean (points)	Treatment missing mean (points)	Estimated effect (points)
A	48	54	7.2
B	45	50	6.6
C	44	46	5.6

All three estimates use the same observed means, missingness fractions, endpoint, and analysis population. The range from 5.6 to 7.2 points is therefore a difference across these particular extrapolations. It is not a confidence interval, and it does not measure all uncertainty in the study.

What Drives the Difference?

The comparison can be made more transparent by expressing each missing mean as a departure from that arm’s observed mean. Let $\delta_0=\bar{y}_{0,\mathrm{mis}}-\bar{y}_{0,\mathrm{obs}}$ and $\delta_1=\bar{y}_{1,\mathrm{mis}}-\bar{y}_{1,\mathrm{obs}}$. In this example, the observed treatment-minus-control difference is $58-50=8$ points. Because the missing fractions are 0.20 for control and 0.30 for treatment, the model-specific estimate becomes

\hat{\tau} =8+0.30\delta_1-0.20\delta_0.

To see why, the treatment arm’s full mean is $58+0.30\delta_1$: its observed and missing portions together have baseline mean 58, with the departure applied only to the missing 30%. Similarly, the control arm’s full mean is $50+0.20\delta_0$. Subtracting the control mean from the treatment mean yields the expression above. This identity shows why equal-looking changes in the two arms need not cancel: the arms have different missing fractions.

Worked Example: Finding a Missing-Outcome Assumption at a Decision Boundary

Suppose, hypothetically, that a difference of 6 points has been specified in advance as a threshold for a particular interpretation. Keep the control missing mean at 44 points, so $\delta_0=44-50=-6$. What treatment missing mean would make the estimated effect exactly 6 points?

Set the sensitivity expression equal to 6:

6=8+0.30\delta_1-0.20(-6) =9.2+0.30\delta_1.

Solving gives

0.30\delta_1=-3.2,\qquad \delta_1=-10.6667\ \text{points}.

The treatment missing mean at this boundary is $58-10.6667=47.3333$ points, or about 47.3 points. With this control missing mean, a treatment missing mean above 47.3 points gives an estimated effect above 6 points; a lower treatment missing mean gives an estimated effect below 6 points. For example, specification C uses 46 points for the treatment missing mean and gives 5.6 points, below the threshold. This calculation locates a boundary in the assumed missing means; it does not establish which assumption is correct.

Interpretation, Diagnostics, and Alternatives

A useful report states the common estimand, the observed-data quantities held fixed, the candidate specifications, and the resulting effect estimates in endpoint units. If a conclusion is similar across plausible specifications, it is less sensitive to this particular choice of extrapolation. If it changes materially, the result should be described as dependent on the MNAR assumptions, and the assumptions driving that change should be made explicit.

Several checks help identify a misleading comparison:

Check the target is common. Confirm that each estimate concerns the same endpoint, time point, treatment contrast, and analysis population. Different targets cannot be interpreted as a sensitivity comparison.
Reproduce the observed-data inputs. Verify that each calculation uses the same observed arm means and missingness fractions. A discrepancy may indicate a changed sample, endpoint, or analysis rather than an MNAR effect.
Inspect the implied outcomes. Compare missing-outcome means with observed outcome ranges and with the scientific context. Implausible values can flag an unsuitable extrapolation, although plausible values do not prove a model is correct.
Check the arithmetic and arm weights. The observed and missing fractions must sum to one within each arm. Using one arm’s response fraction for both arms can distort the contrast.
Separate point-estimate sensitivity from uncertainty. Differences across model-specific estimates do not replace standard errors or uncertainty intervals calculated under each model. Nor does overlap or nonoverlap of such intervals alone quantify uncertainty about which MNAR specification is appropriate.

Selection models and pattern-mixture models can represent assumptions about missing outcomes in different ways. For this comparison, the relevant common output is the model-implied distribution, or at least mean, of the missing outcomes in each arm. A selection model may imply that distribution through its outcome and missingness components; a pattern-mixture model may specify it conditional on missingness pattern. The calculation of the marginal arm mean is the same once those implied means are available. This does not make the models equivalent: their assumptions and implications can differ even when they give similar treatment-effect estimates.

Other comparisons answer different questions. Comparing observed-data fit assesses how candidate models reproduce observed information; it cannot by itself establish that their unobserved-outcome extrapolations are correct. Comparing implied missing-outcome distributions can reveal where extrapolations differ, while the treatment-effect comparison shows whether those differences matter for the estimand. A single preferred model may be useful for a primary analysis, but it should not conceal sensitivity to plausible alternatives.

These calculations also have limits. The weighted-mean formula is an aggregation identity, not evidence that a candidate model’s missing-outcome predictions are valid. In an applied analysis, model fitting, covariate adjustment, repeated measurements, and uncertainty estimation may require more elaborate procedures. If covariates or participant histories are involved, predictions should be averaged over the same missing participants and target population for each candidate. Differences caused by changing those components should not be attributed solely to the MNAR specification.

Common pitfall: “The observed data are the same, so any treatment-effect estimate must be the same.” This overlooks that the full-population mean includes missing outcomes. With a missing fraction of 0.30 in the treatment arm, changing its assumed missing-outcome mean by 5 points changes that arm’s mean by $0.30(5)=1.5$ points. The correction is to hold the observed components fixed, calculate the missing contribution under each specification, and compare the resulting estimates without treating the observed data as if they determined the missing outcomes.

Key takeaway: A treatment-effect comparison across MNAR specifications is interpretable only when the estimand and observed-data components are held fixed. Report the model-specific estimates in context, identify the missing-outcome assumptions behind them, and distinguish their spread from within-model statistical uncertainty.

Check Your Understanding

Use the weighted-mean calculation and explain what each result does—and does not—show.

In a control arm, 90 of 100 participants have observed outcomes with mean 20. Under a candidate model, the missing-outcome mean is 15. Calculate the full-arm mean.
In a treatment arm, 80 of 100 participants have observed outcomes with mean 24. The model-implied mean among missing participants is 19. Using the control result in question 1, calculate the treatment effect, treatment minus control.
In the study from the main examples, suppose the treatment missing mean is 50 points and the control missing mean remains 44 points. Calculate the treatment effect and say whether it exceeds the hypothetical 6-point threshold.
Why is the range of estimates from specifications A, B, and C not a confidence interval?

Solutions. 1. The observed fraction is 0.90 and the missing fraction is 0.10, so the mean is $0.90(20)+0.10(15)=18+1.5=19.5$ points. 2. The treatment mean is $0.80(24)+0.20(19)=19.2+3.8=23.0$ points. Subtracting the control mean gives $23.0-19.5=3.5$ points. 3. Here $\delta_1=50-58=-8$ and $\delta_0=44-50=-6$. Thus $\hat{\tau}=8+0.30(-8)-0.20(-6)=8-2.4+1.2=6.8$ points, which exceeds 6 points. 4. The range reports differences among selected model-specific point estimates. It has no stated repeated-sampling coverage level and does not include all uncertainty in estimating model parameters or selecting assumptions, so it is not a confidence interval.