Tipping Point Analysis for Sensitivity to Missing Data

Introduction

Missing outcome data are one of the most important threats to the interpretation of clinical trials.

Patients may discontinue treatment, withdraw consent, miss visits, become lost to follow-up, experience an intercurrent event, or otherwise fail to provide the planned outcome measurements.

The statistical problem is not simply that fewer observations are available. The key question is: What would the missing outcomes have looked like if they had been observed?

That question cannot generally be answered directly from the observed data.

Consequently, analyses of incomplete clinical trial data require assumptions about the unobserved outcomes.

What Is a Tipping Point Analysis?

A tipping point analysis is a sensitivity analysis in which missing outcomes are repeatedly imputed under increasingly unfavorable assumptions.

For each combination of assumptions, the primary analysis is repeated.

The resulting grid of analyses shows where the study conclusion changes from statistically significant to nonsignificant, or from clinically meaningful to not clinically meaningful.

The boundary between these regions is the tipping point.

For example, suppose a treatment demonstrates a favorable treatment difference under the primary analysis.

A tipping point analysis might show that the result remains statistically significant unless missing patients in the experimental arm have outcomes approximately 8 points worse than comparable observed patients.

That is much more informative than simply reporting one alternative imputation model.

Why Missing Data Matter

Consider a randomized clinical trial comparing an experimental treatment with control.

Suppose the outcome is measured using a continuous scale in which lower values are better.

Let:

• \(Y=1\) denote an observed outcome for a patient with complete data.
• \(Y=0\) denote an outcome that is missing.
• \(X\) denote baseline covariates and other observed information.

The observed-data likelihood contains information about patients with observed outcomes.

But the distribution of outcomes among patients whose outcomes are missing is not directly observed.

Consequently, different assumptions about those missing outcomes can produce different treatment-effect estimates.

Missingness Mechanisms

A useful starting point is to distinguish the major missingness mechanisms.

Missing Completely at Random

Under MCAR, missingness is unrelated to both observed and unobserved outcomes, conditional on the design.

MCAR is generally a strong assumption and is rarely credible as a literal description of clinical trial dropout.

Missing at Random

Under MAR, missingness may depend on observed information, but after conditioning on that information it does not depend on the unobserved outcome.

Many modern primary analyses of longitudinal clinical trial data are based on MAR assumptions.

Missing Not at Random

Under MNAR, missingness depends on the unobserved outcome even after accounting for observed information.

MNAR is particularly important for sensitivity analysis because the unobserved outcomes cannot be empirically verified.

The Role of the Primary Analysis

A tipping point analysis should normally be interpreted relative to a prespecified primary analysis.

For example, suppose the primary analysis is an MMRM under MAR.

The sensitivity analysis does not necessarily replace the MMRM.

Instead, it asks: Would the primary conclusion remain if the unobserved outcomes departed from the assumptions underlying the primary analysis?

This distinction is important because a sensitivity analysis is intended to evaluate robustness, not to retroactively select whichever missing-data model produces the preferred result.

A Simple Conceptual Example

Suppose a two-arm study has the following observed-data results at Week 24:

The observed treatment difference is:

The negative difference favors the experimental treatment because lower values represent greater improvement.

But 30 patients have missing Week 24 outcomes.

The tipping point question is: How unfavorable would the missing Week 24 outcomes need to be before the treatment advantage disappeared?

Why Simple Complete-Case Analysis Is Not Enough

One tempting approach is to analyze only patients with observed Week 24 outcomes.

That is a complete-case analysis.

However, if dropout is related to prognosis, treatment response, adverse events, or other observed characteristics, complete-case analysis may be biased.

More importantly, deleting patients does not solve the fundamental problem: the outcomes that were not observed remain unknown.

Delta Adjustment

One common framework for sensitivity analysis is the delta-adjusted pattern-mixture model.

Suppose the primary MAR-based imputed outcome has conditional mean:

A delta adjustment modifies this mean by a prespecified quantity \(\delta\).

For a continuous outcome:

The direction of \(\delta\) depends on the clinical interpretation of the outcome.

If lower values are better, a positive \(\delta\) makes the missing outcome worse.

If higher values are better, a negative \(\delta\) would make the missing outcome worse.

Delta Adjustments and the Tipping Point

Suppose:

represents the primary MAR assumption.

Increasing \(\delta\) might represent increasingly unfavorable outcomes among patients with missing data.

For example:

The tipping point is the region in which the primary conclusion changes.

One-Dimensional Tipping Point Analysis

The simplest tipping point analysis changes one sensitivity parameter.

For example:

For each value of \(\delta\):

1. Generate imputations under the specified assumption.
2. Complete the missing outcomes.
3. Run the prespecified analysis.
4. Record the treatment effect and inferential result.

The results can then be plotted against \(\delta\).

Two-Dimensional Tipping Point Analysis

A more informative analysis often varies assumptions separately by treatment group.

Let:

A grid of combinations can then be evaluated:

This creates a sensitivity surface rather than a single sensitivity analysis.

What Does "Tipping" Mean?

The term tipping point refers to the point at which the substantive conclusion changes.

The conclusion might be defined using:

• Statistical significance
• A confidence interval crossing zero
• A prespecified clinically important difference
• A responder-rate threshold
• A regulatory decision criterion

For a superiority analysis, one common criterion is:

when negative values favor treatment.

The precise definition should be established before the sensitivity analysis is interpreted.

Worked Example: Continuous Endpoint

Suppose a randomized trial compares an experimental treatment with control.

The primary endpoint is change from baseline at Week 24, where lower values indicate improvement.

The primary analysis estimates:

with a 95% confidence interval:

The confidence interval excludes zero, so the primary analysis favors the experimental treatment.

Now suppose missing outcomes are handled under MAR in the primary analysis.

A sensitivity analysis considers increasingly unfavorable assumptions for the experimental treatment.

In this hypothetical example, the tipping point lies between approximately \(\delta=4\) and \(\delta=5\).

The Tipping Point Is Not a Magic Number

The tipping point should not be interpreted as a scientifically established threshold.

It is a description of the sensitivity of the result to a particular family of assumptions.

For example, saying: "The tipping point is \(\delta=4.5\)" does not mean that 4.5 is the true amount of bias caused by missingness.

It means that within the chosen sensitivity model, assumptions more unfavorable than approximately 4.5 units would change the specified conclusion.

Reference-Based Imputation

Tipping point analyses can also be constructed using reference-based imputation.

Instead of directly specifying a numerical delta, the investigator specifies how the post-discontinuation trajectory of a patient should relate to another reference group.

Examples include:

• Jump to reference: after discontinuation, outcomes behave like the reference treatment group.
• Copy reference: post-discontinuation outcomes follow the reference group's distribution.
• Copy increments in reference: the patient's future changes follow changes observed in the reference group.
• Copy increments in control: post-withdrawal trajectory follows the control trajectory.

Reference-based approaches can provide clinically interpretable sensitivity scenarios.

Tipping Point Analysis With Pattern-Mixture Models

A pattern-mixture model groups patients according to their missingness pattern.

Conceptually:

The distribution of outcomes can then be specified separately for different missingness patterns.

For sensitivity analysis, the unobserved component can be varied systematically.

This is useful because MNAR assumptions are otherwise difficult to express directly in a conventional outcome-regression framework.

Delta-Based Pattern-Mixture Model

Suppose the MAR-implied conditional distribution for a missing outcome is:

A delta-adjusted model might instead assume:

For a normally distributed outcome this corresponds to shifting the conditional mean by \(\delta\).

The exact implementation depends on the imputation model and endpoint.

Multiple Imputation Is Usually Required

A tipping point analysis should generally account for uncertainty introduced by the imputation process.

Simply replacing each missing value with a single deterministic number can understate uncertainty.

Instead, multiple imputation generates multiple completed datasets.

For each sensitivity scenario:

1. Generate \(M\) imputed datasets.
2. Analyze each dataset.
3. Combine the estimates using Rubin's rules.
4. Record the pooled estimate, standard error, confidence interval, and p-value.

The within-imputation variance is:

and the between-imputation variance is:

The total variance is:

The resulting standard error is:

Choosing the Sensitivity Grid

The sensitivity grid should cover assumptions that are scientifically plausible and sufficiently broad to determine whether a tipping point exists.

For example:

and:

This produces:

sensitivity scenarios.

Each cell can contain the estimated treatment effect, confidence interval, p-value, or a binary indicator such as significant/not significant.

Example Tipping Point Matrix

Suppose negative values favor treatment and each cell reports whether the primary superiority conclusion remains statistically significant.

The boundary between "Yes" and "No" represents the tipping region.

Why the Direction of the Grid Matters

Sensitivity analysis should move in clinically meaningful directions.

For an endpoint where larger values are better:

might represent increasingly unfavorable missing outcomes.

For an endpoint where smaller values are better:

might represent increasingly unfavorable missing outcomes.

Therefore, the sign of a delta should never be interpreted without knowing the endpoint direction.

Tipping Point Analysis for Binary Endpoints

The same conceptual framework can be used for binary endpoints.

Suppose the outcome is response:

A sensitivity analysis can vary the assumed response probability among patients with missing outcomes.

For example:

The analysis can then determine the point at which the treatment conclusion changes.

Binary Outcomes Require Special Care

For binary outcomes, simply adding a continuous delta to the response probability can produce impossible values.

A valid probability must satisfy:

Therefore, binary tipping point analyses may instead use:

• Directly specified response probabilities
• Odds-ratio adjustments
• Logit-scale shifts
• Reference-based response probabilities

For a logit-scale sensitivity parameter:

where:

This guarantees that the resulting probability remains between zero and one.

Tipping Point Analysis for Time-to-Event Endpoints

Tipping point concepts can also be applied to time-to-event analyses, but the implementation becomes more complex.

Patients may be censored before experiencing the event.

The missing-data problem may therefore involve assumptions about the unobserved event time rather than simply an unobserved continuous measurement.

Sensitivity analyses may consider:

• Alternative censoring assumptions
• Worst-case event-time assumptions
• Pattern-mixture models
• Inverse-probability weighting sensitivity parameters
• Reference-based assumptions

The exact method should be selected according to the estimand and the time-to-event analysis strategy.

Connection With the Estimand Framework

Missing-data sensitivity analysis should be aligned with the trial's estimand.

An estimand specifies the treatment effect of interest, including how intercurrent events are handled.

For example, consider a patient who permanently discontinues study treatment because of an adverse event.

The analysis must distinguish between:

• What treatment effect is being estimated?
• What does the post-discontinuation outcome represent?
• How should the outcome after the intercurrent event be interpreted?
• What missing-data assumption is appropriate for that estimand?

Tipping Point Analysis and MNAR

A tipping point analysis is often described as an MNAR sensitivity analysis.

That description is reasonable when the sensitivity model explicitly departs from MAR assumptions.

However, it is important to be precise.

The analysis does not demonstrate that the data are MNAR.

Instead, it evaluates whether conclusions remain robust under a set of assumptions that are incompatible with the primary MAR model.

How to Choose Plausible MNAR Assumptions

The sensitivity grid should not be arbitrary.

Potential sources of justification include:

• Observed outcomes before discontinuation
• Observed outcomes among patients with similar dropout patterns
• Outcomes in the control group
• Historical clinical trial data
• External registries
• Clinical expert judgment
• Observed treatment discontinuation mechanisms
• Previous studies of the same disease and endpoint

The goal is to construct a range of assumptions that is clinically interpretable, not merely mathematically convenient.

A Useful Interpretation Scale

One practical approach is to describe the sensitivity assumptions qualitatively.

This can make the sensitivity analysis much easier for clinical and regulatory stakeholders to interpret.

R Implementation: Basic Delta Grid

A simple tipping point analysis can begin by defining a grid of sensitivity parameters.

delta_grid <- seq(
  from = 0,
  to = 10,
  by = 1
)

delta_grid

Here positive values represent increasingly unfavorable assumptions for an endpoint where larger values are worse.

Constructing Sensitivity Scenarios

Suppose the MAR-imputed values are stored in a vector called y_mar.

impute_delta <- function(y_mar, delta) {

  y_mar + delta

}

The function can then be applied across the sensitivity grid.

imputed_sets <- lapply(
  delta_grid,
  function(delta) {
    impute_delta(
      y_mar = y_mar,
      delta = delta
    )
  }
)

This illustrates the basic idea, but a real clinical trial analysis should perform multiple imputation and preserve the appropriate covariance and analysis structure.

A Simple Treatment-Effect Calculation

For illustration, suppose each completed dataset contains an outcome y and treatment indicator trt.

estimate_effect <- function(dat) {

  mean(dat$y[dat$trt == 1]) -
    mean(dat$y[dat$trt == 0])

}

The sensitivity estimate can then be calculated for each delta.

results <- data.frame(
  delta = delta_grid,
  estimate = sapply(
    imputed_sets,
    estimate_effect
  )
)

results

This simple example demonstrates the mechanics but is not a substitute for the prespecified clinical-trial analysis model.

Multiple Imputation With a Delta Adjustment

In a real analysis, one common workflow is:

1. Fit the imputation model under MAR.
2. Generate multiple imputed datasets.
3. Apply the sensitivity adjustment to the imputed missing outcomes.
4. Analyze every completed dataset using the planned analysis model.
5. Pool the estimates.
6. Repeat for every sensitivity parameter.

Finding the Tipping Point Programmatically

Suppose the primary conclusion is considered successful when the upper confidence limit is below zero.

results$tipping_indicator <-
  results$upper_ci >= 0

The first sensitivity scenario satisfying this condition can be identified with:

first_tipping <-
  results[
    which(results$upper_ci >= 0)[1],
  ]

first_tipping

In practice, it is preferable to report the interval between the last scenario that preserves the conclusion and the first scenario that overturns it rather than implying an exact tipping point that was never evaluated.

Interpolation Should Be Used Carefully

Suppose:

The tipping point lies somewhere between 4 and 5 under the chosen model.

One could interpolate the boundary, but the inferential result itself is based on the actual statistical model, not on a linear interpolation assumption.

Creating a Tipping Point Heatmap

For two-dimensional sensitivity analyses, a heatmap can display the result of each combination of assumptions.

library(ggplot2)

ggplot(
  results,
  aes(
    x = delta_control,
    y = delta_treatment,
    fill = significant
  )
) +
  geom_tile() +
  labs(
    x = "Control delta",
    y = "Treatment delta",
    fill = "Conclusion"
  ) +
  theme_minimal()

A publication-quality implementation can replace the binary indicator with the estimated treatment effect, p-value, or confidence-limit value.

Displaying the Treatment Effect Across Delta

A one-dimensional analysis can also be displayed as a sensitivity curve.

ggplot(
  results,
  aes(
    x = delta,
    y = estimate
  )
) +
  geom_line() +
  geom_point() +
  geom_hline(
    yintercept = 0,
    linetype = "dashed"
  ) +
  labs(
    x = "Sensitivity parameter",
    y = "Estimated treatment effect"
  ) +
  theme_minimal()

The point where the confidence interval crosses the decision threshold is often more informative than the point estimate alone.

What Should the Tipping Point Table Contain?

A useful table can include:

• Sensitivity parameter(s)
• Imputation assumption
• Number of imputations
• Estimated treatment effect
• Standard error
• 95% confidence interval
• p-value
• Decision indicator

For example:

How to Interpret a Robust Result

Suppose the treatment conclusion remains favorable across a broad range of clinically plausible missing-data assumptions.

That provides evidence that the conclusion is robust to the evaluated departures from the primary missing-data assumption.

For example:

How to Interpret a Fragile Result

Suppose a small departure from MAR causes the confidence interval to cross zero.

That indicates that the conclusion is sensitive to assumptions about missing outcomes.

This does not automatically invalidate the primary analysis.

Instead, it means the evidence should be interpreted with greater caution.

What a Tipping Point Does Not Tell You

A tipping point analysis does not tell you:

• Which missing-data assumption is true.
• How many missing outcomes actually had the assumed values.
• Whether MAR or MNAR is correct.
• That the tipping point itself is clinically likely.
• That all missing-data mechanisms have been evaluated.

It tells you how the conclusion behaves within the specified sensitivity model.

Common Mistake: Calling the Worst Case "the" Sensitivity Analysis

A single worst-case analysis may be useful as a stress test.

But it does not show how the conclusion changes across assumptions.

A tipping point analysis is more informative because it identifies the transition from robust to non-robust conclusions.

Common Mistake: Using Unrealistic Sensitivity Values

A tipping point at an extreme value may be mathematically interesting but clinically irrelevant.

For example, if the missing outcome would need to be 40 points worse than the MAR prediction to overturn the conclusion, the result may be highly robust.

Conversely, if a difference of only 1 point changes the conclusion, the result may deserve substantial scrutiny.

The clinical plausibility of the sensitivity assumptions is therefore critical.

Common Mistake: Changing the Analysis Model

The purpose of a tipping point analysis is generally to alter assumptions about the missing outcomes while keeping the substantive analysis framework aligned with the primary analysis.

For example, if the primary analysis is a prespecified MMRM, changing the sensitivity analysis to an unrelated complete-case t-test creates multiple differences between the analyses.

Then it becomes difficult to determine whether the result changed because of the missing-data assumption or because the analysis model changed.

Common Mistake: Treating Missingness as a Treatment Failure Automatically

Forcing every missing outcome to the worst possible value can be overly conservative and may not correspond to the estimand.

The correct sensitivity analysis depends on:

• The endpoint
• The estimand
• The intercurrent-event strategy
• The missingness pattern
• The clinical context

Common Mistake: Ignoring Differential Missingness by Treatment Group

Missingness rates may differ between treatment groups.

Suppose:

A sensitivity analysis that assumes identical behavior among missing patients in both groups may not adequately reflect the actual missingness structure.

This is one reason treatment-specific sensitivity parameters can be useful.

Reference-Based Tipping Point Example

Suppose patients discontinuing experimental treatment are assumed, after discontinuation, to behave somewhere between the experimental trajectory and the control trajectory.

A sensitivity parameter can describe the degree of movement toward the reference trajectory.

Conceptually:

where:

• \(\lambda=0\) corresponds to the MAR-based prediction.
• \(\lambda=1\) corresponds to the reference trajectory.
• Intermediate values represent partial movement toward the reference.

The precise statistical implementation depends on the reference-based imputation method.

Tipping Point Analysis and Regulatory Interpretation

Sensitivity analyses are particularly important when missing data are substantial enough that the primary conclusion could reasonably depend on unobserved outcomes.

A useful sensitivity analysis should therefore make clear:

• Which missing-data assumptions were used.
• Why the assumptions were considered plausible.
• How the assumptions depart from the primary analysis.
• Whether the conclusion changes.
• Where the tipping region occurs.
• Whether the tipping assumptions are clinically plausible.

The goal is transparency rather than producing a particular favorable result.

How Many Imputations Should Be Used?

The number of imputations should be sufficiently large to make Monte Carlo error small relative to the precision required for the analysis.

Older analyses often used relatively small numbers such as:

Modern analyses frequently use substantially more imputations, particularly when the fraction of missing information is large.

Values such as:

may be considered depending on the complexity of the analysis and the required Monte Carlo precision.

Assessing Monte Carlo Stability

A tipping point analysis may contain dozens or hundreds of sensitivity scenarios.

Each scenario can involve multiple imputed datasets.

It is therefore important to verify that results are stable to the number of imputations.

For example, one might compare results using:

The exact choices depend on the analysis and computational resources.

A Complete Tipping Point Workflow

Recommended Sensitivity Analysis Table

How to Report the Tipping Point in a CSR

A clear CSR presentation should avoid simply saying: "The tipping point was 5 units."

Instead, describe the sensitivity model and the transition.

How to Report a Two-Dimensional Tipping Point

When separate treatment and control assumptions are varied, the report should describe the region rather than selecting one arbitrary cell.

For example:

Tipping Point vs. Worst-Case Analysis

Tipping Point vs. LOCF

Last Observation Carried Forward is sometimes presented as though it were a sensitivity analysis.

However, LOCF makes a specific deterministic assumption about the unobserved outcome trajectory.

A tipping point analysis instead evaluates a range of assumptions.

Tipping Point vs. Complete-Case Analysis

Complete-case analysis discards observations with missing outcomes.

Tipping point analysis instead attempts to evaluate what the missing outcomes could plausibly have been under different assumptions.

The latter is therefore much more directly connected to the missing-data sensitivity question.

What Makes a Good Tipping Point Analysis?

A strong analysis has five characteristics.

1. Clinically interpretable assumptions.
2. A sufficiently broad sensitivity grid.
3. Proper propagation of imputation uncertainty.
4. Consistency with the estimand and primary analysis.
5. Transparent reporting of the tipping region.

What Makes a Weak Tipping Point Analysis?

Common problems include:

• Choosing the sensitivity grid after seeing the results.
• Using arbitrary delta values without clinical justification.
• Using only one deterministic imputation.
• Changing the analysis model between primary and sensitivity analyses.
• Reporting only the most favorable sensitivity result.
• Failing to show the assumptions at which the conclusion changes.
• Calling an extreme scenario "realistic" without justification.

Protocol and SAP Considerations

The sensitivity analysis should ideally be prespecified.

The protocol or SAP should describe:

• The primary estimand
• The primary missing-data assumption
• The primary analysis method
• The sensitivity analysis framework
• The sensitivity parameters
• The range of sensitivity parameters
• The number of imputations
• The analysis model applied after imputation
• The pooling method
• The definition of a tipping point
• The criteria for clinical interpretation

Data Quality Before Sensitivity Analysis

A tipping point analysis should not substitute for understanding why data are missing.

Before running the sensitivity analysis, examine:

• Rates of missingness by treatment arm
• Missingness by visit
• Reasons for discontinuation
• Timing of discontinuation
• Baseline characteristics
• Observed outcomes before dropout
• Treatment exposure
• Adverse events
• Rescue medication or subsequent therapy

These patterns can inform whether particular sensitivity assumptions are clinically plausible.

Example Missingness Summary

If treatment discontinuation is strongly related to worsening outcomes, a sensitivity analysis that assumes missing outcomes are somewhat worse than MAR may be particularly relevant.

Interpreting a Tipping Point Clinically

The most useful question is often not: "Where is the mathematical tipping point?"

It is: "Are the assumptions required to overturn the conclusion clinically plausible?"

Consider two studies.

Even though both analyses technically have tipping points, Study B may provide substantially stronger robustness evidence if a 12-unit departure is clinically implausible.

The Most Important Concept

The central purpose of tipping point analysis is not to manufacture a statistically significant result.

It is to determine how much the missing outcomes would have to differ from the primary missing-data assumption before the study's conclusion changes.

The analysis therefore converts an abstract concern about missing data into a more concrete question:

That question can be answered across a clinically meaningful range of assumptions.

Practical Checklist

Summary of the Worked Example

Suppose the primary analysis produces:

with:

The treatment is favorable under the primary MAR analysis.

A delta-adjusted sensitivity analysis then considers increasingly unfavorable missing outcomes.

The conclusion therefore appears robust to the smaller departures but not to the more extreme assumptions.

Bottom Line

References

Little, R.J.A. (1993). Pattern-mixture models for multivariate incomplete data. Journal of the American Statistical Association, 88, 125–134.

Little, R.J.A. & Rubin, D.B. (2019). Statistical Analysis with Missing Data, 3rd ed. Wiley.

Carpenter, J.R., Roger, J.H. & Kenward, M.G. (2013). Analysis of longitudinal trials with protocol deviation: a framework for relevant, accessible sensitivity analyses. Statistical Methods in Medical Research, 22, 3–18.

Kenward, M.G., Carpenter, J.R. & White, I.R. (2017). Reference-based sensitivity analysis via multiple imputation using chained equations. Biometrics.

National Research Council (2010). The Prevention and Treatment of Missing Data in Clinical Trials. National Academies Press.

ICH E9(R1). Addendum on estimands and sensitivity analysis in clinical trials.

Rubin, D.B. (1987). Multiple Imputation for Nonresponse in Surveys. Wiley.