Sensitivity Analysis with Sparse Missing Outcomes

Why Sparse Missing Outcomes Need Careful Interpretation

When only a small proportion of outcomes are missing, a fixed delta adjustment may have a limited effect on an overall mean. That is a statement about how the estimate responds to the adjustment, not evidence that the chosen delta is correct. The observed outcomes do not, by themselves, reveal how the missing outcomes differ from their predictions under missing at random (MAR). A small missing fraction can therefore make a sensitivity scenario’s numerical effect modest while leaving the scenario’s key assumption weakly informed by the data.

This distinction matters when assessing whether a conclusion is robust. Robustness to a specified range of deltas describes the consequences of those assumptions. It does not show that the assumptions in that range are likely. The focus here is an additive delta adjustment for a continuous outcome; the same central distinction between estimate sensitivity and information about the sensitivity parameter applies more broadly.

Definition: Let $n$ be the number of participants, $m$ the number with a missing outcome, and $q=m/n$ the missing-outcome fraction. Under an additive delta scenario, each imputed missing outcome is shifted from its MAR imputation by a fixed amount $\delta$, in the outcome’s units: $$ Y_{i,\mathrm{mis}}^{(\delta)}=Y_{i,\mathrm{mis}}^{(\mathrm{MAR})}+\delta. $$ The observed outcomes are not shifted. A positive $\delta$ assumes that missing outcomes are, on average, higher than their MAR predictions by $\delta$; a negative $\delta$ assumes they are lower. The value of $\delta$ is a sensitivity assumption, not a parameter identified by the observed outcomes alone.

How the Missing Fraction Scales the Mean Shift

Suppose each of $M$ completed data sets contains the same $n$ participants and the same $m$ outcome-missing participants. In completed data set $l$, let $\bar{Y}^{(\mathrm{MAR})}_l$ be the arithmetic mean before the delta adjustment and $\bar{Y}^{(\delta)}_l$ the mean after it. Only $m$ values change, and each changes by $\delta$. Thus the total of the $n$ outcomes changes by $m\delta$, so the mean changes by $m\delta/n=q\delta$:

\bar{Y}^{(\delta)}_l = \bar{Y}^{(\mathrm{MAR})}_l+\frac{m\delta}{n} = \bar{Y}^{(\mathrm{MAR})}_l+q\delta.

Averaging these completed-data means over the $M$ imputations gives the same relationship for the pooled point estimate. For a fixed $\delta$, the size of the mean shift is the missing fraction multiplied by the shift assigned to each missing outcome. A small $q$ therefore attenuates the shift in the overall mean. It does not reduce the assumed difference $\delta$ among the missing outcomes themselves.

Worked Example: A Small Shift in the Overall Mean

Consider a hypothetical sample of $n=200$ participants, with $m=8$ outcomes missing. Suppose the pooled mean under MAR is 50 points. Examine an additive scenario with $\delta=-10$ points.

The missing fraction is $q=8/200=0.04$. The shift in the pooled mean is $$ q\delta=0.04(-10)=-0.40\text{ points}. $$ Therefore, the adjusted pooled mean is $50-0.40=49.60$ points. The assumed difference for each missing outcome is still $-10$ points relative to its MAR prediction; the overall mean moves by only $-0.40$ points because only 4% of outcomes are missing.

For comparison, if the same data were evaluated at $\delta=+10$ points, the pooled mean would be $50+0.40=50.40$ points. These calculations describe the estimate under specified assumptions. They do not establish that either delta is plausible or that the missing outcomes actually differ by either amount.

Treatment Contrasts Can Respond Differently

For a treatment contrast, the overall missing fraction alone is not enough to determine the effect of a delta adjustment. Let $q_T$ and $q_C$ be the missing-outcome fractions in the treatment and control groups, and let $\delta_T$ and $\delta_C$ be their respective additive shifts. Define the contrast as treatment minus control. The adjusted contrast changes by the treatment-group mean shift minus the control-group mean shift:

\widehat{\theta}^{(\delta)}-\widehat{\theta}^{(\mathrm{MAR})} = q_T\delta_T-q_C\delta_C.

If the two groups have the same missing fraction and the same delta, the changes cancel in the contrast, even though both group means move. If the fractions or deltas differ, cancellation need not occur. Consequently, a low overall missing percentage can conceal a higher missing fraction in one group, and even a small group-specific shift can matter to a contrast when the groups differ in their missingness or assumptions.

Worked Example: Different Missing Fractions by Group

In a hypothetical trial, the MAR treatment-minus-control contrast in mean outcome is $-3.0$ points. The treatment group has $q_T=0.02$ missing outcomes and the control group has $q_C=0.10$. Consider the scenario $\delta_T=-5$ points and $\delta_C=-8$ points.

The treatment mean changes by $0.02(-5)=-0.10$ points. The control mean changes by $0.10(-8)=-0.80$ points. Hence the contrast changes by $$ q_T\delta_T-q_C\delta_C = 0.02(-5)-0.10(-8) = -0.10-(-0.80) = 0.70\text{ points}. $$ The adjusted contrast is $-3.0+0.70=-2.30$ points. It is less negative than the MAR contrast because the control mean moved downward more than the treatment mean. The computation does not say whether these group-specific deltas are appropriate; it shows what follows if they are assumed.

If instead both groups had the same missing fraction and a common delta, their group means would each shift by the same amount and the treatment-minus-control contrast would not shift. That cancellation is an algebraic consequence of the setup, not a general property of sparse missingness.

What Changes When Uncertainty Is Pooled?

Sensitivity analysis concerns uncertainty as well as point estimates. Under Rubin’s rules, let $Q_l$ be the estimate from completed data set $l$, $U_l$ its estimated variance, $\bar{Q}$ the average of the $M$ estimates, $W$ the average of the $U_l$, and $B$ the sample variance of the $Q_l$. The pooled variance is

\bar{Q}=\frac{1}{M}\sum_{l=1}^{M}Q_l, \qquad W=\frac{1}{M}\sum_{l=1}^{M}U_l, \qquad B=\frac{1}{M-1}\sum_{l=1}^{M}(Q_l-\bar{Q})^2, \qquad T=W+\left(1+\frac{1}{M}\right)B.

For the arithmetic mean with a fixed delta and the same missingness pattern in every imputation, each $Q_l$ increases by exactly $q\delta$. Adding the same constant to every estimate leaves their sample variance unchanged, so $B$ is unchanged. The within-imputation variance $U_l$, however, can change because shifting the imputed values can change the variability among the completed outcomes. It must be recomputed from the delta-adjusted completed data. For other estimands, such as nonlinear summaries, the estimates may not all shift by the same constant; their between-imputation variation may change and must also be recomputed.

The effect on within-imputation variance can be seen directly. In one completed data set, let $S$ be the set of $m$ imputed outcomes, let $\bar{y}$ be the completed-data mean, and let $s^2$ be the sample variance. Shifting the $m$ values in $S$ by $\delta$ changes the centered sum of squares by

(n-1)\left(s_{\delta}^{2}-s^{2}\right) = 2\delta\sum_{i\in S}(y_i-\bar{y}) + \delta^2m\left(1-\frac{m}{n}\right).

To justify this expression, write the original centered sum of squares as $\sum_i(y_i-\bar{y})^2$. After shifting, the new mean is $\bar{y}+m\delta/n$. Expanding the new centered sum of squares and collecting terms gives the original sum plus $2\delta\sum_{i\in S}(y_i-\bar{y})+\delta^2(m-m^2/n)$. Dividing the change by $n-1$ yields the displayed variance change. This also shows why a fixed shift does not guarantee that within-imputation variability increases: the cross-product term can be positive or negative.

Worked Example: A Fixed Shift Changes Within-Data Variability

Take one hypothetical completed data set with four outcomes: two observed values, 10 and 12, and two imputed values, 11 and 13. The original mean is $11.5$. The original centered sum of squares is $$ (10-11.5)^2+(12-11.5)^2+(11-11.5)^2+(13-11.5)^2 = 2.25+0.25+0.25+2.25 = 5. $$ Thus the original sample variance is $s^2=5/(4-1)=1.667$ points squared.

Now use $\delta=2$ points. The adjusted completed data are 10, 12, 13, and 15, with mean 12.5. Their centered sum of squares is $$ (10-12.5)^2+(12-12.5)^2+(13-12.5)^2+(15-12.5)^2 = 6.25+0.25+0.25+6.25 = 13. $$ The adjusted sample variance is $13/3=4.333$ points squared. In this example, the shift increases within-data variability.

If this same fixed shift is applied to the same missing positions in every imputation, each imputed-data mean rises by $q\delta=(2/4)(2)=1$ point. The between-imputation variance of those means stays the same, because all means receive the same one-point addition. The within-imputation variance must still be recomputed; the example shows that it can change.

Assumptions, Diagnostics, and Alternatives

A delta-adjusted analysis needs a defensible MAR imputation model as its reference, including appropriate predictors and a suitable outcome model. The delta scenario then adds an explicit assumption about the distribution of missing outcomes relative to that reference. For a constant additive delta, it assumes the same shift for all missing outcomes to which it is applied. If deltas differ by group or subgroup, those distinctions must be defined before calculating a contrast. The imputation procedure must also generate enough imputations for Monte Carlo error to be acceptably small relative to the uncertainty being reported.

With sparse missing outcomes, report the missing count and fraction overall and by analysis group. Check whether a small total count is concentrated in a particular group or relevant pattern. Inspect the imputed outcomes and the adjusted values for inconsistencies with the outcome’s possible range; an additive shift can, for example, produce impossible values for a bounded measure. Compare the MAR results with the specified delta scenarios, recompute the completed-data estimate and its variance under each scenario, and check that simulation error is not mistaken for a meaningful change. These checks assess the calculation and its implications; they do not identify delta from the observed outcomes.

A complete-case analysis avoids imputing missing outcomes but generally changes the analyzed sample and relies on assumptions about who has complete outcomes; it is not automatically a neutral alternative. An MAR multiple-imputation analysis uses the specified observed-data model without a delta departure. Delta-adjusted imputation makes the departure explicit and examines its consequences. The methods answer different conditional questions, so agreement among them is informative about robustness to the choices examined, not proof that missingness is harmless.

Common pitfall: “Only 4% of outcomes are missing, so the data show that the missing outcomes cannot differ much from their MAR predictions.” This reasoning confuses a small effect on the overall mean, $q\delta$, with information about $\delta$. The observed data can constrain the overall estimate’s response to a chosen delta while providing little direct information about the missing outcomes’ departures from MAR. State the assumed delta and its units, and interpret the result as conditional on that assumption.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

A sample has $n=500$ outcomes, of which $m=15$ are missing. Under a scenario with $\delta=6$ units, by how much does the pooled arithmetic mean change?
A treatment group has a 5% missing fraction and a control group has a 5% missing fraction. Both use $\delta=-4$ units. What is the change in the treatment-minus-control contrast?
For a pooled arithmetic mean, why does a fixed delta applied to the same missingness pattern leave $B$, the between-imputation variance of the estimates, unchanged?
Does a small missing fraction show that delta is close to zero? Explain the distinction.
For a fixed-delta analysis, name one component of pooled uncertainty that must be recomputed even when the between-imputation variance of the mean is unchanged.

Solutions. (1) $q=15/500=0.03$, so the mean changes by $q\delta=0.03(6)=0.18$ units. (2) The contrast change is $0.05(-4)-0.05(-4)=0$ units; both group means shift equally. (3) Each completed-data mean receives the same constant $q\delta$, which does not change the sample variance across those means. (4) No. A small missing fraction can limit the overall mean’s response to a fixed delta, but the observed outcomes alone do not establish the value of delta for missing outcomes. (5) The within-imputation variance $W$ must be recomputed from the adjusted completed data because shifting imputed values can alter their variability.