Defining a Tipping Point for a Missing-Data Sensitivity Analysis

Why Define a Tipping Point?

A missing-data sensitivity analysis asks whether a treatment-effect conclusion would remain the same under different plausible assumptions about outcomes that were not observed. A tipping point marks the boundary at which that conclusion changes. The boundary is not a property of the data alone: it depends on the estimand, the sensitivity model, and the conclusion rule chosen for the analysis.

For example, a conclusion that the treatment effect is statistically distinguishable from zero can change at a different sensitivity-parameter value from a conclusion that the effect exceeds a clinically meaningful threshold. The word “tipping” therefore has a precise meaning only after the conclusion being monitored has been stated. The introductory material on delta adjustment provides the setting: a sensitivity parameter modifies predictions for missing outcomes while leaving observed outcomes unchanged.

Definition: A Boundary Between Conclusions

Definition: Let \(\delta\) denote a sensitivity parameter in a specified missing-data model, and let \(\Omega\) be the set of its values under consideration. Let \(C(\delta)\) be the prespecified conclusion from the analysis at \(\delta\), such as “the estimated effect exceeds a clinical threshold” or “the confidence interval excludes zero in the favorable direction.” A tipping point is a value at the boundary between regions of \(\Omega\) that yield different values of \(C(\delta)\). In one dimension, it is a boundary value where arbitrarily nearby parameter values, within the range under consideration, yield different conclusions.

This definition separates three ideas that are sometimes conflated. The sensitivity parameter describes an assumption about missing outcomes. The estimated treatment effect is an analysis result under that assumption. The conclusion is the output of a prespecified rule applied to the analysis result. A tipping point concerns a change in the third item as the first varies; it need not occur where the estimated effect is zero.

For a continuous parameter, the boundary may be an exact value or one of several values. If the analysis is evaluated only on a finite grid, the results can identify a change between two evaluated values, but do not by themselves establish the exact boundary between them. Equality conventions also matter: a rule using “greater than or equal to” can classify a boundary value differently from a rule using “greater than.” The rule must be fixed before interpreting the boundary.

A Longitudinal Delta-Adjusted Trial

Consider a hypothetical randomized trial comparing a treatment with a control at a final follow-up visit. The outcome is a symptom score in points, with larger scores indicating better health. The estimand is the difference in mean final-visit scores, treatment minus control, for the randomized groups. Suppose a reference imputation model predicts missing final-visit outcomes from observed histories. For this illustration, \(\delta\) is an additive adjustment, in score points, applied to the predicted final-visit outcomes of participants missing that visit in the treatment group. The control-group predictions receive no adjustment.

This specification is an assumption, not something identified by the observed data. It presumes that the reference predictions are suitable starting values and that a common additive shift is a useful way to represent departures for the treatment-group missing outcomes. It also fixes which arm and visit the shift applies to. Changing those choices defines a different sensitivity analysis and can produce a different tipping point.

Suppose 12 of 20 treatment participants have observed final-visit scores with mean 8.2, and the reference model predicts a mean of 7.8 for the 8 participants missing that visit. In the control group, 15 of 20 scores are observed with mean 7.4, and the 5 missing outcomes have reference predictions with mean 7.1. The delta-adjusted treatment mean is the weighted average of the observed outcomes and adjusted missing-outcome predictions:

\bar{Y}_{T}(\delta) = \frac{12(8.2)+8(7.8+\delta)}{20} = 8.04+0.400\delta.

The control mean does not depend on \(\delta\):

\bar{Y}_{C} = \frac{15(7.4)+5(7.1)}{20} = 7.325.

Thus the estimated treatment contrast, in score points, is

\hat{\theta}(\delta) = \bar{Y}_{T}(\delta)-\bar{Y}_{C} = 0.715+0.400\delta.

The coefficient 0.400 is the fraction of the treatment group whose final outcomes are missing. Increasing the assumed outcomes for those participants by one point increases the treatment-group mean by 0.400 points. It does not increase it by a full point because the other 12 participants’ observed outcomes are not shifted.

Worked Example: Tipping a Prespecified Clinical-Threshold Conclusion

Using the hypothetical trial above, suppose the prespecified conclusion is that the estimated treatment effect exceeds 0.500 score points. This rule concerns the point estimate, not a test or confidence interval. Determine the boundary value of \(\delta\) for this rule.

The rule is satisfied when \(\hat{\theta}(\delta)>0.500\). Substituting the delta-adjusted contrast gives

0.715+0.400\delta>0.500.

Solving for \(\delta\),

\delta>\frac{0.500-0.715}{0.400} =-0.5375\text{ points}.

At \(\delta=0\), the estimated contrast is 0.715 points, above the threshold. At \(\delta=-0.600\) points, it is \(0.715+0.400(-0.600)=0.475\) points, below the threshold. The boundary is \(-0.5375\) points: under this rule, values above it satisfy the threshold and values below it do not. Because the rule uses a strict inequality, equality at the boundary does not satisfy the conclusion.

This result is conditional on the stated adjustment model, the hypothetical predictions, and the point-estimate rule. It does not establish that a shift of this size is plausible, nor does it incorporate sampling uncertainty.

The Conclusion Rule Determines the Boundary

An analysis may define its conclusion using an estimate and its uncertainty rather than a clinical threshold for the point estimate alone. For instance, the conclusion might be that a two-sided 95% confidence interval lies entirely above zero. That rule can tip at a different value of \(\delta\), because the interval’s lower endpoint—not just the estimated contrast—determines the conclusion.

Worked Example: A Confidence-Interval Conclusion

Continue with the same hypothetical contrast, \(\hat{\theta}(\delta)=0.715+0.400\delta\). For this simplified illustration, suppose the analysis reports a standard error of 0.300 points throughout the sensitivity range being considered. Assume a normal-based 95% confidence interval and define the favorable conclusion as “the interval lies entirely above zero.” This constant-standard-error assumption is made only to keep the arithmetic transparent; an actual analysis must recompute its uncertainty at each value of \(\delta\).

The lower confidence limit is

L(\delta)=\hat{\theta}(\delta)-1.96(0.300) =0.715+0.400\delta-0.588 =0.127+0.400\delta.

The conclusion holds when \(L(\delta)>0\), so the boundary solves

0.127+0.400\delta=0 \quad\Longrightarrow\quad \delta=-0.3175\text{ points}.

At \(\delta=0\), the lower limit is 0.127 points, so the interval is above zero under this rule. At \(\delta=-0.500\), the lower limit is \(0.127+0.400(-0.500)=-0.0730\) points, so the interval includes zero and the rule is no longer satisfied. The tipping point is therefore \(-0.3175\) points for this confidence-interval conclusion, not the \(-0.5375\)-point boundary for the earlier point-estimate threshold.

The distinction is substantive: both boundaries use the same estimated contrast function, but they monitor different conclusions. In practice, standard errors and confidence limits may themselves vary with \(\delta\), especially when the analysis combines imputed datasets. In that case, the boundary must be based on the confidence interval actually produced at each \(\delta\), not on the constant-standard-error simplification used here.

A Longitudinal Estimand Can Change the Tipping Point

The choice of outcome summary also matters. A longitudinal estimand might average outcomes across visits rather than focus only on the final visit. This changes how a final-visit adjustment contributes to the treatment contrast. The following example keeps the same hypothetical trial but defines a different estimand and a different prespecified conclusion.

Worked Example: Averaging Two Follow-Up Visits

Suppose the treatment-minus-control contrast at an earlier follow-up is 0.400 points and is unaffected by \(\delta\), because all outcomes at that visit are observed. Define the longitudinal estimand as the equally weighted average of the earlier- and final-visit contrasts. The final-visit contrast is \(0.715+0.400\delta\). Suppose the prespecified conclusion is that the average contrast exceeds 0.300 points.

The average contrast is

\hat{\theta}_{\mathrm{avg}}(\delta) = \frac{0.400+(0.715+0.400\delta)}{2} = 0.5575+0.200\delta.

Set this equal to the 0.300-point threshold to find the boundary:

0.5575+0.200\delta=0.300 \quad\Longrightarrow\quad \delta=\frac{0.300-0.5575}{0.200} =-1.2875\text{ points}.

At \(\delta=0\), the estimated average contrast is 0.5575 points, above the threshold. At \(\delta=-1.500\), it is \(0.5575+0.200(-1.500)=0.2575\) points, below the threshold. The boundary is \(-1.2875\) points for this average-outcome estimand and rule. It differs from the final-visit boundary because the earlier visit contributes to the estimand but is unaffected by the final-visit adjustment.

Assumptions, Diagnostics, and Alternatives

A tipping point is interpretable only in relation to a defensible analysis at each sensitivity-parameter value. At minimum, specify the outcome scale and direction, target estimand, missing-data adjustment, parameter range, and conclusion rule. The reference model and imputation procedure must be appropriate for the observed data and outcome structure; the delta adjustment must be applied to the intended missing outcomes and not inadvertently to observed outcomes. For longitudinal data, the visit, treatment arm, and missingness pattern to which each adjustment applies must be unambiguous.

Useful checks include verifying that the analysis at \(\delta=0\) reproduces the designated reference analysis, confirming that the adjustment changes only the intended predicted outcomes, and recomputing the treatment effect and uncertainty over the sensitivity range. Review the adjusted outcome distributions as well as their means: a plausible mean can conceal implausible individual predictions or distributions. Also check the sign convention. Here, negative \(\delta\) lowers the treatment-group predictions, and a positive treatment-minus-control contrast favors treatment. Reversing either convention changes the verbal interpretation of the boundary.

A reported boundary should be accompanied by the conclusion rule and enough context to judge its relevance. Compare the tipping value with clinically plausible delta values and with the predictions implied by the adjustment. If the boundary lies far outside a defensible range, that may indicate robustness over the range considered; it does not prove the missing-data assumption or validate extrapolation beyond that range. If the analysis uses only a grid, report the adjacent evaluated values where the conclusion differs rather than claiming an exact boundary without further calculation.

Tipping-point analysis complements, rather than replaces, other approaches. A primary analysis under a reference missing-data assumption gives a result at one specified assumption; sensitivity analysis asks how that result changes under alternatives. Pattern-mixture delta adjustment makes the departure explicit through adjusted distributions for missing outcomes. Selection models represent missingness through a model for observation probabilities conditional on outcomes and covariates. These approaches encode assumptions differently, so their sensitivity parameters and boundaries are not directly interchangeable. No tipping point identifies which missing-data assumption is true.

A common mistake is to call the value where the estimated treatment effect reaches zero “the tipping point.” That is correct only if the prespecified conclusion is specifically about the sign of the point estimate. If the rule instead concerns a clinical threshold or whether a confidence interval excludes zero, the relevant boundary is where that rule changes. Another mistake is to report a boundary without stating which arm and missing outcomes the delta modifies; without that information, its direction and units cannot be interpreted.

Key takeaway: Define the conclusion first, then vary the sensitivity parameter under a clearly stated missing-data model. The tipping point is the boundary where that conclusion changes—not a universal feature of the treatment effect.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In the first example, what is the tipping point for the rule that the estimated final-visit contrast exceeds 0.500 points, and what does the boundary mean?
Why is the confidence-interval tipping point different from the point-estimate-threshold tipping point in the examples?
Suppose only 25% of a treatment group has missing final outcomes and the adjusted missing predictions increase by 2 points. By how much does the treatment-group mean increase, assuming all else is fixed?
Why is it insufficient to report a tipping point as “delta equals −0.5” without further information?

Solutions. (1) The boundary is \(-0.5375\) points, found by solving \(0.715+0.400\delta=0.500\). Under the strict “exceeds” rule, values above the boundary satisfy the rule and values at or below it do not. (2) The confidence-interval rule depends on the lower confidence limit, which includes the standard-error contribution; the point-estimate rule compares the estimate directly with 0.500 points. They monitor different conclusions. (3) The mean increases by \(0.25(2)=0.500\) points because only one quarter of the group is shifted. (4) The number has no clear interpretation without the outcome units, adjustment direction and target outcomes, estimand, parameter range, and conclusion rule. Those details determine what changes at the boundary.