MMRM Part 2: Sensitivity Analyses and Robustness

Introduction

A well-specified MMRM can provide an efficient analysis of longitudinal clinical trial data, particularly when repeated continuous outcomes are collected at multiple post-baseline visits and some observations are missing.

But a statistically significant treatment effect from one MMRM specification does not automatically establish that the conclusion is robust.

MMRM inference can depend on assumptions about:

• The covariance structure of repeated observations
• The missing-data mechanism
• The variables included in the model
• The treatment-by-visit interaction
• The definition of the analysis population
• The handling of intercurrent events
• The estimation and degrees-of-freedom method

For this reason, a clinical trial analysis should generally distinguish between the primary MMRM and a set of prespecified sensitivity analyses designed to investigate whether the primary conclusion depends materially on particular assumptions.

Quick Review: What Is an MMRM?

Suppose subject \(i\) has repeated continuous outcomes at visits \(j=1,\ldots,J\). Let:

• \(Y_{ij}\) = outcome for subject \(i\) at visit \(j\)
• \(T_i\) = treatment indicator
• \(B_i\) = baseline covariate
• \(V_j\) = visit indicator

A common MMRM can be written conceptually as:

The treatment effect is generally represented by the treatment-by-visit interaction terms rather than by one universal treatment coefficient.

For a prespecified primary visit \(J\), the treatment contrast might be:

with inference based on the estimated least-squares means or estimated marginal means at that visit.

Why the Primary MMRM Is Not Enough

Consider a trial in which the primary analysis estimates a treatment difference of:

with a 95% confidence interval of:

The result appears statistically significant.

Now suppose an alternative reasonable covariance structure produces:

with a similar confidence interval.

That supports the robustness of the conclusion.

But suppose a prespecified MNAR sensitivity analysis produces:

with a confidence interval crossing zero.

That tells a very different story.

Three Broad Classes of MMRM Sensitivity Analysis

MMRM sensitivity analyses can be organized into three broad categories.

These categories should not be treated as interchangeable.

Covariance Structure Sensitivity

One of the most common MMRM sensitivity analyses changes the covariance structure used to describe within-subject repeated measurements.

The covariance matrix for subject \(i\) can be represented as:

For example, with four post-baseline visits:

An unstructured covariance matrix estimates each variance and covariance separately.

Alternative structures impose restrictions on this matrix.

Why Covariance Structure Matters

The covariance structure does not usually change the fixed-effect point estimate dramatically when the mean model is correctly specified and the data are well behaved.

However, it can affect:

• Standard errors
• Degrees of freedom
• Confidence intervals
• Test statistics
• Least-squares mean contrasts
• Convergence

Therefore, covariance sensitivity is primarily an assessment of whether the inference is dependent on a particular representation of within-subject correlation.

Unstructured Covariance

The unstructured covariance matrix is attractive because it imposes relatively few assumptions.

For \(J\) visits, however, the number of covariance parameters grows quickly:

For example:

With many visits and limited sample size, the unstructured covariance can become unstable.

Covariance Structure Selection

A common workflow is to fit several plausible structures and compare their fit.

Useful criteria include:

• AIC
• AICC
• BIC
• Convergence behavior
• Positive-definiteness of the estimated covariance matrix
• Clinical and temporal plausibility

A lower information criterion indicates better relative fit among the models being compared, but it should not be treated as an automatic reason to change the prespecified primary analysis.

Example: Covariance Sensitivity Table

The treatment effect is very similar across all four structures.

That would generally be reassuring evidence that the conclusion is not being driven by one particular covariance specification.

Missing Data: The Central Sensitivity Issue

The most important robustness question for many longitudinal clinical trials is not the covariance structure.

It is the question:

What would the missing outcomes have been?

An MMRM under a missing-at-random assumption does not literally recover the unobserved values.

Instead, it uses the observed data and model structure to estimate the distribution of the outcomes that are relevant for inference.

Missingness Mechanisms

Three commonly discussed missingness concepts are:

The distinction between MAR and MNAR is especially important for sensitivity analysis.

MAR in an MMRM

A standard likelihood-based MMRM is commonly interpreted under an MAR assumption.

Conceptually:

where \(M_i\) denotes the missingness pattern.

The missingness probability may depend on information that has already been observed, but after conditioning on that observed information it does not depend on the unobserved outcome itself.

Why MAR Cannot Be Tested Directly

The defining feature of MNAR concerns outcomes that were not observed.

Because the missing outcomes are unavailable, the assumption that missingness does not additionally depend on those unobserved outcomes cannot generally be verified from the observed data alone.

This is why MNAR sensitivity analyses are important.

Pattern-Mixture Models

A pattern-mixture approach conditions the outcome distribution on the observed missingness pattern.

Let \(R\) represent a missingness pattern.

A pattern-mixture model considers:

Different assumptions are then made about the distribution of outcomes for subjects with different missingness patterns.

For example, subjects who discontinue treatment because of lack of efficacy may be assigned a different post-discontinuation outcome distribution than subjects who remain in the study.

Delta Adjustment

One of the most intuitive MNAR sensitivity approaches modifies imputed outcomes by a prespecified amount.

Suppose the MAR-based imputed mean for a missing outcome is:

A delta-adjusted sensitivity analysis might define:

For an endpoint where lower values represent improvement, a positive \(\delta\) would make the imputed outcome worse.

For example:

could be used to investigate progressively less favorable assumptions.

Delta Adjustment in a Treatment-Control Trial

Suppose lower scores are better and the MAR analysis estimates:

A sensitivity analysis might worsen missing treatment outcomes by a specified delta while leaving the control group unchanged.

The resulting treatment difference could then be evaluated across a grid of values.

This suggests that the conclusion is sensitive to sufficiently large departures from MAR.

The Tipping Point

A tipping-point analysis identifies how severe a departure from the primary missing-data assumption must be before the study conclusion changes.

Suppose \(\delta_T\) is the smallest unfavorable adjustment that causes the confidence interval to include zero.

Then:

The result is not simply "robust" or "not robust."

Instead, it answers a more useful question:

How unfavorable would the missing outcomes need to be before the treatment conclusion changes?

Reference-Based Imputation

Reference-based imputation provides another framework for sensitivity analysis under departures from MAR.

The key idea is that post-discontinuation outcomes for one treatment group may be modeled using information from another group.

Common reference-based strategies include:

• Jump to Reference
• Copy Reference
• Copy Increment from Reference

These approaches can encode clinically meaningful assumptions about what would happen after treatment discontinuation.

Jump to Reference

Under a jump-to-reference assumption, once a subject in the active treatment group becomes unobserved, their future outcome trajectory is assumed to follow the reference treatment trajectory.

Conceptually:

where \(T\) represents treatment and \(C\) represents control.

This can be conservative when the active treatment effect is expected to disappear after discontinuation.

Copy Reference

Copy-reference approaches make a related assumption about the trajectory after missingness.

For example, the future increments observed in the reference group may be used to project the missing active-treatment outcomes.

The conceptual distinction is:

Why Reference-Based Sensitivity Analyses Are Useful

These methods are useful when treatment discontinuation has a clear clinical interpretation.

For example, suppose subjects discontinue an investigational treatment because of lack of efficacy.

It may be unreasonable to assume that the treatment effect remains unchanged forever after discontinuation.

A reference-based analysis can provide a structured sensitivity analysis for that possibility.

Multiple Imputation as a Sensitivity Framework

Multiple imputation can be used both for primary analyses and for sensitivity analyses.

For \(M\) imputed datasets, the treatment effect estimate from imputation \(m\) is:

The pooled estimate is:

Rubin's rules combine the within-imputation and between-imputation variability.

Let:

and:

The total variance is:

The square root of \(T\) provides the pooled standard error.

MMRM vs. Multiple Imputation

An important conceptual distinction is that a standard likelihood-based MMRM does not require explicit imputation of each missing value.

Under its modeling assumptions, the likelihood uses all available observed outcomes.

Multiple imputation instead creates completed datasets and combines the resulting estimates.

Worst-Case Analyses

A simple worst-case analysis might assign highly unfavorable values to missing observations.

Such analyses can be useful as stress tests, but they should be interpreted carefully.

If the assumed worst-case outcome is biologically or clinically impossible, the analysis may provide little meaningful information.

LOCF Is Not Automatically a Robustness Analysis

Last observation carried forward (LOCF) is sometimes presented as a simple sensitivity analysis.

However, carrying the last observed value forward imposes a strong and often implausible assumption about the patient's future trajectory.

It effectively assumes:

That is rarely justified merely because the endpoint is longitudinal.

LOCF may be included as a historical or supplementary analysis when required by a particular analysis plan, but it should not automatically be described as a preferred modern missing-data sensitivity analysis.

Alternative Baseline Specifications

The primary MMRM may include baseline as a continuous covariate.

A sensitivity analysis might examine reasonable alternatives, such as:

• Baseline as a continuous covariate
• Baseline category
• Stratification factor adjustment
• Additional prespecified prognostic covariates

However, arbitrary post hoc covariate selection should be avoided.

Alternative Treatment-by-Visit Specifications

A standard MMRM often includes a treatment-by-visit interaction:

This allows the treatment difference to vary over time.

A sensitivity analysis might examine whether conclusions are materially affected by alternative prespecified representations of time.

Examples include:

• Categorical visit
• Continuous time
• Polynomial time terms
• Restricted time windows

For most clinical-trial MMRMs, categorical visit is often preferred because it does not impose a particular functional form on the trajectory.

Why Continuous Time Can Be Risky

Suppose a model assumes:

This implies a particular linear relationship between time and outcome.

If the actual clinical trajectory is nonlinear, the model can be misspecified.

Therefore, continuous-time sensitivity analyses should be justified by the clinical trajectory rather than introduced merely because they use fewer parameters.

Analysis Population Sensitivity

A robustness assessment can also compare analysis populations.

Examples include:

• Full analysis set
• Intent-to-treat population
• Per-protocol population
• Modified intent-to-treat population

The exact terminology and population definitions depend on the trial.

Sensitivity Analysis vs. Estimand Analysis

This distinction is increasingly important in modern clinical-trial analysis.

A sensitivity analysis asks whether the inference is robust to assumptions within the same general target of inference.

An estimand analysis may instead change the scientific question.

Intercurrent Events

Longitudinal trials often contain events that occur after treatment initiation and affect the interpretation of subsequent outcomes.

Examples include:

• Treatment discontinuation
• Rescue medication
• Switching treatment
• Death
• Permanent withdrawal

The analysis should distinguish the missing-data problem from the estimand question created by the intercurrent event.

Kenward-Roger vs. Satterthwaite

MMRM analyses often use a small-sample degrees-of-freedom approximation.

Two commonly encountered approaches are:

• Kenward-Roger
• Satterthwaite

These methods can produce slightly different standard errors and degrees of freedom, particularly in smaller samples or complex covariance structures.

A sensitivity analysis may compare these methods when the study is sufficiently small or the inference is close to the decision threshold.

Why Degrees of Freedom Matter

The test statistic for a treatment contrast can be expressed as:

The p-value also depends on the reference distribution and its degrees of freedom.

Therefore, two analyses with almost identical treatment estimates can sometimes produce slightly different p-values if the estimated standard error or degrees of freedom differs.

Robust Standard Errors Are Not a Universal MMRM Solution

It is tempting to assume that a sandwich or empirical covariance estimator automatically makes an MMRM robust.

That conclusion is too strong.

Robust variance methods address certain forms of variance misspecification, but they do not automatically solve:

• Missing-data assumptions
• Mean-model misspecification
• Nonrandom dropout
• Incorrect estimands
• Inadequate sample size
• Incorrect handling of intercurrent events

Example Clinical Trial

Consider a randomized Phase III trial comparing an active treatment with placebo.

The primary endpoint is change from baseline in a continuous disease-severity score measured at Weeks 4, 8, 12, and 16.

Suppose:

Primary MMRM Result

Suppose the primary analysis gives:

with:

and:

The result supports a treatment benefit at Week 16 under the primary analysis assumptions.

The next question is:

Would reasonable alternative assumptions change that conclusion?

Sensitivity Analysis Plan

A reasonable sensitivity program might include:

Example Robustness Results

This example illustrates an important pattern.

The treatment estimate is stable across covariance structures and under MAR multiple imputation.

However, increasingly unfavorable MNAR assumptions eventually eliminate statistical significance.

That does not mean the primary analysis was invalid.

It means that the strength of the conclusion depends to some extent on the assumption that the missing outcomes are not substantially worse than predicted under the MAR-based analysis.

How to Interpret a Sensitivity Analysis

There are at least four useful questions to ask.

1. Does the point estimate change materially?
2. Does the confidence interval change materially?
3. Does the direction of treatment effect change?
4. Does the clinical conclusion change?

The fourth question is often the most important.

Statistical vs. Clinical Robustness

Statistical robustness and clinical robustness are not identical.

Suppose the minimally clinically important difference is 3 points.

If sensitivity analyses produce:

the clinical conclusion is quite stable.

But suppose they produce:

The statistical significance might remain in some analyses, but the clinical interpretation is becoming less certain.

Direction-of-Effect Robustness

A particularly simple robustness criterion is whether the estimated treatment effect remains in the same direction.

For example:

Even if the p-value crosses 0.05 at one sensitivity parameter, consistent direction and clinically meaningful effect estimates may still provide important evidence.

Confidence-Interval Robustness

The confidence interval is often more informative than the p-value.

Suppose the primary analysis gives:

and the sensitivity analysis gives:

These are highly similar.

But if the sensitivity analysis gives:

the uncertainty has changed substantially.

Missingness by Treatment Group

Sensitivity analyses should pay particular attention to differential dropout.

Suppose:

The reason for the imbalance matters.

For example, active-treatment discontinuations might be caused by:

• Lack of efficacy
• Adverse events
• Patient preference
• Administrative reasons

Each mechanism can suggest different plausible assumptions about subsequent unobserved outcomes.

Reason for Missingness Should Be Used

A useful sensitivity framework can stratify assumptions by reason for discontinuation.

The exact assumptions should be justified clinically rather than selected arbitrarily.

Sensitivity to Visit Windows

Clinical trials often allow assessment windows around scheduled visits.

For example, Week 16 might permit assessments from Day 106 through Day 126.

Sensitivity analyses can examine whether restricting or redefining the visit window materially changes the result.

This is especially relevant when:

• Visits are highly asynchronous
• Treatment discontinuation is common
• Outcome trajectories change rapidly
• Protocol deviations affect assessment timing

Sensitivity to Baseline Adjustment

Baseline adjustment can improve precision when baseline outcome strongly predicts the post-baseline outcome.

A common MMRM includes baseline as a covariate:

A sensitivity analysis might assess whether the treatment conclusion remains stable under a reasonable alternative baseline specification.

Sensitivity to Stratification Factors

If randomization was stratified by important baseline factors, the primary model may adjust for those factors.

A sensitivity analysis can examine whether omitting or alternatively coding those factors changes the treatment contrast.

However, this should be interpreted in the context of the randomization scheme and prespecified analysis plan.

Convergence Is Part of Robustness

A covariance structure that fails to converge reliably is not a useful sensitivity model merely because it is theoretically attractive.

Potential warning signs include:

• Nonconvergence
• Boundary estimates
• Near-singular covariance matrices
• Extremely large standard errors
• Optimization warnings
• Strong dependence on starting values

SAS: Primary MMRM

A typical SAS implementation uses PROC MIXED.

proc mixed data=analysis method=reml;
    class subject treatment visit;

    model change =
        baseline
        treatment
        visit
        treatment*visit
        / ddfm=kr solution;

    repeated visit /
        subject=subject
        type=un;

    lsmeans treatment*visit
        / diff cl;

run;

The exact coding depends on the study design, but the key components are:

• Subject identifier
• Treatment
• Visit
• Baseline covariate
• Treatment-by-visit interaction
• Repeated covariance structure
• Small-sample degrees-of-freedom method

SAS: Alternative Covariance Structures

The covariance structure can be changed using the TYPE= option.

/* Unstructured */
repeated visit /
    subject=subject
    type=un;

/* Compound symmetry */
repeated visit /
    subject=subject
    type=cs;

/* Heterogeneous AR(1) */
repeated visit /
    subject=subject
    type=arh(1);

/* Toeplitz */
repeated visit /
    subject=subject
    type=toep;

The resulting treatment estimates and confidence intervals can then be compared across models.

SAS: Satterthwaite Degrees of Freedom

An alternative to Kenward-Roger is Satterthwaite-type degrees of freedom.

proc mixed data=analysis method=reml;
    class subject treatment visit;

    model change =
        baseline
        treatment
        visit
        treatment*visit
        / ddfm=satterthwaite solution;

    repeated visit /
        subject=subject
        type=un;

    lsmeans treatment*visit
        / diff cl;

run;

This can be useful as a technical sensitivity analysis when small-sample inference is important.

SAS: Multiple Imputation

A common multiple-imputation workflow uses PROC MI followed by analysis of each imputed dataset and pooling with PROC MIANALYZE.

proc mi data=analysis
        out=mi_data
        nimpute=100
        seed=20260914
        method=mcmc;

    var baseline
        week4
        week8
        week12
        week16;

run;

The precise imputation model should reflect the longitudinal structure, treatment groups, auxiliary variables, and missingness assumptions of the trial.

SAS: Analyze the Imputed Datasets

proc mixed data=mi_data;
    by _imputation_;

    class subject treatment visit;

    model change =
        baseline
        treatment
        visit
        treatment*visit
        / ddfm=kr solution;

    repeated visit /
        subject=subject
        type=un;

    lsmeans treatment*visit
        / diff cl;
    
    ods output
        Estimates=estimates;

run;

The resulting estimates can then be combined using Rubin's rules through PROC MIANALYZE.

SAS: Delta-Adjusted Multiple Imputation

For an MNAR sensitivity analysis, the imputed values can be shifted according to a prespecified delta.

data mi_delta;
    set mi_data;

    if treatment = "Active" and missing(week16) then
        week16 = week16 + 2;

run;

The actual implementation should be based on the imputation method and data structure used in the study.

R: MMRM with nlme

An MMRM-style model can be fitted in R using nlme::lme.

library(nlme)

fit_un <- lme(
  fixed =
    change ~ baseline +
      treatment * visit,

  random = ~ 1 | subject,

  correlation =
    corSymm(form = ~ visit | subject),

  weights =
    varIdent(form = ~ 1 | visit),

  data = analysis,
  method = "REML"
)

summary(fit_un)

The exact specification depends on how the repeated covariance is represented and on the coding of the visit variable.

R: Alternative Covariance Structures

For example, a compound-symmetry structure can be represented using:

fit_cs <- lme(
  fixed =
    change ~ baseline +
      treatment * visit,

  random = ~ 1 | subject,

  correlation =
    corCompSymm(form = ~ visit | subject),

  data = analysis,
  method = "REML"
)

An AR(1)-type structure can be represented using:

fit_ar1 <- lme(
  fixed =
    change ~ baseline +
      treatment * visit,

  random = ~ 1 | subject,

  correlation =
    corAR1(form = ~ visit | subject),

  data = analysis,
  method = "REML"
)

The primary analysis should use the prespecified model, while these alternatives can be used to assess robustness.

R: Modern MMRM Packages

In modern clinical-trial workflows, packages designed specifically for MMRM can also be used.

For example, the mmrm package provides functionality for MMRM models with several covariance structures and small-sample inference options.

library(mmrm)

fit <- mmrm(
  change ~
    treatment * visit +
    baseline +
    us(visit | subject),
  data = analysis
)

summary(fit)

The precise formula syntax should be matched to the installed package version and the study's data structure.

R: Sensitivity Analysis Loop

A useful programming strategy is to fit several prespecified covariance structures and collect the treatment contrast from each model.

structures <- c(
  "unstructured",
  "compound_symmetry",
  "ar1",
  "toeplitz"
)

results <- data.frame(
  structure = structures,
  estimate = NA_real_,
  lower = NA_real_,
  upper = NA_real_,
  p_value = NA_real_
)

# Fit each prespecified model,
# extract the Week 16 treatment contrast,
# and populate results.

For production analysis, the extraction code should be validated carefully against the exact model output.

Build a Sensitivity Matrix

A useful final programming output is a sensitivity matrix.

This makes it much easier to distinguish which assumption is responsible for any change in the result.

One-Factor-at-a-Time Sensitivity

A particularly interpretable strategy is to change one major assumption at a time.

For example:

This isolates the source of sensitivity.

Do Not Create Hundreds of Analyses

Sensitivity analysis can become counterproductive if every conceivable statistical variation is explored.

A better approach is to identify the assumptions that are:

• Most consequential
• Scientifically plausible
• Potentially controversial
• Relevant to regulatory interpretation
• Relevant to the clinical estimand

Primary, Supportive, and Sensitivity Analyses

A statistical analysis plan should distinguish among different roles.

How to Report Sensitivity Analyses

A strong clinical-trial report should not simply state:

"Sensitivity analyses were consistent with the primary analysis."

Instead, report enough information to allow the reader to understand what was tested.

For example:

• Alternative covariance structure
• Missing-data assumption
• Imputation method
• Analysis population
• Treatment contrast
• Estimate
• Confidence interval
• P-value where appropriate

Example Reporting Table

How to Describe a Robust Result

A useful interpretation might be:

This is more informative than simply reporting a collection of p-values.

What a Non-Robust Result Looks Like

Suppose the results were:

Here the covariance structure is not the concern.

The major issue is sensitivity to assumptions concerning post-discontinuation outcomes.

That distinction is important when communicating the evidence.

Common Mistakes

1. Changing the covariance structure solely because it produces a smaller p-value. The primary covariance structure should be prespecified.
2. Treating MAR as proven. MAR is an assumption, not a directly testable fact about unobserved outcomes.
3. Calling every alternative model a sensitivity analysis. Some analyses address a different estimand or scientific question.
4. Using LOCF automatically. LOCF imposes a strong assumption about future outcomes and is not inherently more conservative or more robust.
5. Reporting only whether p < 0.05. Effect estimates and confidence intervals are essential.
6. Ignoring reasons for discontinuation. Different reasons can imply different plausible missing-data assumptions.
7. Using arbitrary MNAR deltas. Sensitivity parameters should have clinical or statistical justification.
8. Ignoring convergence problems. An unstable covariance model is not a useful robustness analysis.
9. Changing multiple assumptions simultaneously without explanation. This makes it difficult to determine why the result changed.
10. Creating an enormous sensitivity-analysis matrix. Sensitivity analysis should focus on the assumptions most relevant to the scientific conclusion.
11. Confusing per-protocol analysis with missing-data sensitivity. A per-protocol analysis can represent a different estimand.
12. Assuming a robust standard error solves all MMRM problems. Variance robustness does not solve mean-model, missing-data, or estimand problems.

A Practical MMRM Robustness Workflow

What Should Be Prespecified in the SAP?

The Statistical Analysis Plan should describe the sensitivity strategy before the results are known.

At minimum, consider specifying:

• Primary MMRM model
• Primary covariance structure
• Degrees-of-freedom method
• Primary estimand
• Primary visit
• Missing-data assumptions
• Alternative covariance structures
• Multiple-imputation methodology if used
• MNAR sensitivity framework
• Delta values or delta ranges
• Reference-based assumptions where applicable
• Tipping-point methodology
• Analysis populations
• Rules for intercurrent events
• Criteria for interpreting robustness

A Useful Sensitivity Hierarchy

Not all sensitivity analyses carry equal importance.

The exact hierarchy depends on the trial.

The Most Important Distinction: Covariance vs. Missing Data

It is useful to separate two questions that are often confused.

Question 1: Covariance

How are repeated observations from the same subject correlated?

Question 2: Missing Data

What can reasonably be assumed about outcomes that were not observed?

Changing the covariance structure addresses the first question.

MNAR sensitivity analyses address the second.

Interpreting a Tipping-Point Grid

Suppose the treatment-group delta is varied across:

and the conclusion changes from significant to nonsignificant between \(\delta=1\) and \(\delta=2\).

A useful presentation is:

The tipping point is approximately between \(\delta=1\) and \(\delta=2\).

The scientific interpretation depends on whether such a departure is clinically plausible.

Clinical Interpretation of the Delta

Suppose the endpoint is a symptom score where one point corresponds to a meaningful change.

Then a delta of 2 has an intuitive interpretation.

If the sensitivity analysis requires missing treatment outcomes to be approximately two points worse than predicted under MAR before the conclusion changes, that may be reasonably robust.

If only a 0.2-point departure causes the conclusion to disappear, the result is much more sensitive.

Robustness Across the Entire Response Trajectory

MMRM sensitivity analysis should not focus exclusively on the primary visit.

The estimated treatment difference may be examined across all visits:

Consistent trajectories provide additional evidence that the observed result is not an artifact of one particular covariance structure.

What Does "Robust" Actually Mean?

There is no universal numerical definition of robustness.

In practice, a result is more convincing when:

• The direction of effect is stable.
• The magnitude of effect remains clinically meaningful.
• Confidence intervals remain reasonably similar.
• Alternative covariance structures produce similar conclusions.
• Reasonable missing-data departures do not reverse the conclusion.
• Results are consistent across relevant analysis populations.
• The conclusion is not driven by one unusual modeling decision.

Conversely, a result deserves additional scrutiny when a small plausible change in assumptions causes a major change in the estimated treatment effect.

Regulatory Perspective

For confirmatory clinical trials, sensitivity analyses are particularly important because missing data and intercurrent events can materially affect the interpretation of treatment effects.

A strong analysis plan should therefore make clear:

• What the primary estimand represents
• How intercurrent events are handled
• What assumptions underlie the primary analysis
• Which sensitivity analyses challenge those assumptions
• How the results will be interpreted

The purpose is not to manufacture a favorable result under every imaginable scenario.

The purpose is to determine whether the primary conclusion remains credible under reasonable alternative assumptions.

Validation and Reproducibility

Sensitivity analyses should be programmed with the same rigor as the primary analysis.

Important validation steps include:

• Independent programming or review where appropriate
• Verification of treatment contrasts
• Verification of covariance specifications
• Verification of imputation seeds and datasets
• Verification of delta adjustments
• Verification of pooling calculations
• Verification of confidence intervals
• Verification of tipping-point thresholds
• Traceability from SAP specification to output

Recommended Sensitivity Analysis Set

For a typical longitudinal confirmatory trial, a reasonable sensitivity package might contain:

The Complete Decision Framework

Final Worked Example Summary

The Most Important Concepts

There are several concepts worth remembering.

1. Covariance robustness and missing-data robustness are different. Changing UN to ARH(1) does not test an MNAR assumption.
2. MMRM under MAR is not automatically robust to MNAR. The treatment conclusion should be challenged using appropriate missing-data sensitivity analyses.
3. Delta adjustments make MNAR assumptions explicit. The size of the delta should have a meaningful interpretation.
4. Tipping-point analyses quantify fragility. They show how large a departure from the primary assumption is required to change the conclusion.
5. Reference-based methods provide structured alternatives. They are particularly useful when treatment discontinuation has clinical meaning.
6. Effect estimates matter more than p-values alone. A stable treatment estimate across analyses is strong evidence of robustness.
7. Different estimands are not merely sensitivity analyses. Changing the scientific question should be identified explicitly.
8. Prespecification matters. Sensitivity analyses should be defined before the results are known whenever possible.

References

Mallinckrodt, C.H., Clark, W.S., and David, S.R. (2001). Accounting for dropout bias using mixed-effects models. Journal of Biopharmaceutical Statistics.
Mallinckrodt, C.H., Lin, Q., and Molenberghs, G. (2014). Handling missing data in longitudinal clinical trials. Journal of Biopharmaceutical Statistics.
Kenward, M.G. & Roger, J.H. (1997). Small sample inference for fixed effects from restricted maximum likelihood. Biometrics, 53, 983–997.
Little, R.J.A. (1995). Modeling the drop-out mechanism in repeated-measures studies. Journal of the American Statistical Association, 90, 1112–1121.
Carpenter, J.R., Roger, J.H. & Kenward, M.G. (2013). Analysis of longitudinal trials with protocol deviation: a framework for relevant, accessible, and informative sensitivity analyses. Statistical Medicine, 32, 168–183.
Cro, S., Morris, T.P., Kenward, M.G., Carpenter, J.R. (2020). Sensitivity analysis for clinical trials with missing continuous outcome data using controlled multiple imputation. Statistical Methods in Medical Research.
National Research Council (2010). The Prevention and Treatment of Missing Data in Clinical Trials. National Academies Press.
ICH E9(R1) (2019). Addendum on Estimands and Sensitivity Analysis in Clinical Trials. International Council for Harmonisation.
Little, R.J.A., D'Agostino, R., Cohen, M.L., et al. (2012). The prevention and treatment of missing data in clinical trials. New England Journal of Medicine, 367, 1355–1360.