Why Reporting Is Part of Sensitivity Analysis
A sensitivity analysis can produce several plausible treatment-effect estimates without establishing which missing-data assumption is correct. Reporting should therefore let readers see, in one place, what each assumption says about unobserved outcomes, why it may be clinically plausible, how the estimate changes, and how uncertain it is. A list of model names or a statement that results were “robust” does not provide that information.
This tutorial uses a hypothetical stroke trial with a 90-day functional outcome. The examples are for reporting practice, not real trial evidence. They focus on how to present sensitivity results; they do not select a preferred MNAR model. The primary analysis and sensitivity analyses should target the same estimand and use the same outcome definition. Otherwise, a difference between estimates could reflect a changed scientific question rather than a changed missing-data assumption.
Define the Outcome and Contrast Before Comparing Models
Suppose the trial randomizes 100 patients to an investigational treatment and 100 to control. The endpoint is a hypothetical 90-day functional utility score from 0 to 100, with higher scores indicating better function. For this example, death by day 90 is assigned a score of 0; a missing score is an outcome not observed at day 90. This explicit endpoint definition distinguishes an observed death score from a missing functional report. A trial using another death or functional-outcome definition must report that definition instead.
Let \(\mu_T\) and \(\mu_C\) denote the mean 90-day scores in the treatment and control groups under the stated analysis. The effect is the mean difference \(\Delta=\mu_T-\mu_C\), in score points. Thus, a positive value favors treatment on this scale, and zero represents no mean difference. A confidence interval that includes zero indicates that the data and model do not rule out zero at the corresponding two-sided confidence level; it does not establish that the treatments are equivalent.
For a concise comparison, the hypothetical primary analysis uses a missing-at-random (MAR) assumption conditional on the analysis information. In plain language, once the observed information in the model is taken into account, outcomes that are missing do not have a further systematic difference from outcomes that are observed. An MNAR sensitivity analysis allows missing outcomes to differ further. In a delta analysis, a specified shift in score points represents that difference. The shift is applied to the model-predicted outcomes for participants with missing scores; it is not a shift to the scores of participants whose outcomes were observed.
The primary analysis and each sensitivity analysis should use the same randomized groups, endpoint, estimand, and effect scale. In an actual analysis, the model and uncertainty calculation must also be appropriate to the design and imputation procedure. The summaries below assume those analyses have been completed correctly. The simple mean calculations illustrate how the assumptions affect the estimate, not a substitute for a trial’s full analysis model.
Worked Example: Show How Assumptions Change the Estimate
Worked Example: Show How Assumptions Change the Estimate
In the hypothetical trial, 80 of 100 treatment-group participants and 85 of 100 control-group participants have observed 90-day scores. Their respective observed means are 62 and 58 points. In this simplified illustration, the primary MAR imputation predicts a mean of 62 for missing treatment-group scores and 58 for missing control-group scores. The primary analysis therefore gives an estimated difference of 4 points.
For an arm-level illustration, the completed mean is the weighted mean of the observed and imputed scores. The treatment-group calculation under the primary assumption is
The control-group calculation is
Now consider a delta scenario in which missing treatment-group outcomes average 8 points below their MAR predictions, while missing control-group outcomes average 3 points below their MAR predictions. The imputed means are \(62-8=54\) and \(58-3=55\). The resulting treatment mean is
The control mean and effect are
This assumption reduces the estimated advantage from 4 to 2.85 points, a change of \(-1.15\) points relative to the primary analysis. The arithmetic also shows why the shifts alone do not equal the change in treatment effect: the shifts apply only to missing outcomes, and the groups have different proportions missing.
Worked Example: Attach Uncertainty to the Comparison
Worked Example: Attach Uncertainty to the Comparison
Suppose the completed hypothetical analyses report a standard error of 2.00 points for the primary estimate and 2.05 points for the delta estimate. For illustration, use a normal-based 95% confidence interval, calculated as estimate \(\pm 1.96\times SE\). In a real report, the interval must use the variance procedure appropriate to the analysis; the standard errors here are stipulated results, not values recoverable from the group means alone.
For the primary estimate, the interval is
For the delta estimate, it is
The primary interval is just above zero, whereas the delta interval includes zero. This comparison describes how the estimates and their uncertainty differ under the stated assumptions. It does not mean there is a 95% probability that the fixed treatment effect lies inside either calculated interval, and it does not give the probability that a null hypothesis is true.
Worked Example: Assemble a Clinically Interpretable Table
Worked Example: Assemble a Clinically Interpretable Table
The following additional scenarios are hypothetical completed analyses. A conservative scenario imputes a mean of 50 for missing treatment-group scores and 58 for missing control-group scores. A reference-based scenario uses a control-like mean of 58 for missing treatment-group scores and 58 for missing control-group scores. The latter is a mean-level illustration of a control-reference assumption; an actual reference-based analysis would specify and implement its distributional assumptions, not merely replace means.
For the conservative scenario, the treatment mean is \([80(62)+20(50)]/100=59.60\), while the control mean is \([85(58)+15(58)]/100=58.00\), giving a difference of 1.60 points. For the reference-based illustration, the treatment mean is \([80(62)+20(58)]/100=61.20\), and the control mean remains 58.00, giving a difference of 3.20 points.
Assume the hypothetical analyses report standard errors of 2.10 points for the conservative scenario and 2.02 points for the reference-based scenario. Their 95% intervals are \(1.60\pm1.96(2.10)=(-2.52,\ 5.72)\) points and \(3.20\pm1.96(2.02)=(-0.76,\ 7.16)\) points. The resulting compact report is:
| Analysis | Assumption about missing scores | Clinical rationale | Estimated mean difference (95% CI), points | Conclusion relative to primary |
|---|---|---|---|---|
| Primary MAR | Missing outcomes follow the model predictions conditional on observed information | Starting analysis; no additional unobserved shift after conditioning | 4.00 (0.08, 7.92) | Reference result |
| Delta: −8 treatment, −3 control | Missing scores are below MAR predictions by 8 and 3 points, respectively | Allow poorer unobserved function in both groups, more so in treatment | 2.85 (−1.17, 6.87) | 1.15 points smaller; interval includes zero |
| Conservative delta | Missing treatment mean 50; missing control mean 58 | Stress a relatively unfavorable outcome pattern for missing treatment participants | 1.60 (−2.52, 5.72) | 2.40 points smaller; interval includes zero |
| Control-reference illustration | Missing treatment scores follow a control-like mean; control missing scores retain the MAR mean | Ask what follows if missing treatment outcomes resemble control outcomes | 3.20 (−0.76, 7.16) | 0.80 points smaller; interval includes zero |
A reader can see that all three sensitivity point estimates are smaller than the primary estimate, but each sensitivity interval includes zero. The table does not show that any scenario is true, nor does it turn “includes zero” into proof of no treatment effect. The rationale column is part of the analysis report: it helps readers judge whether a scenario addresses a credible clinical concern, rather than treating a model label as self-explanatory.
Justification for the Weighted-Mean Calculation
In the simple illustration, the estimated arm mean combines the observed and imputed participants because the target is the mean for the full randomized group, not only for participants with recorded outcomes. If \(n_o\) outcomes are observed with mean \(\bar{x}_o\), \(n_m\) are missing and assigned an imputed mean \(\bar{x}_m\), and \(n=n_o+n_m\), the completed arm mean is
The formula follows by adding the total score represented by each group, \(n_o\bar{x}_o\) and \(n_m\bar{x}_m\), then dividing by the total number represented. Replacing \(\bar{x}_m\) with a sensitivity-specific imputed mean changes the completed arm mean by
Thus, for a two-group difference, unequal missing fractions or unequal shifts can change the contrast in ways that are not obvious from the delta values alone. Real analyses may include baseline covariates, repeated measures, multiple imputation, or other outcome models. Their estimates and standard errors need not equal these elementary weighted-mean calculations, but the reporting principle is unchanged: describe the assumption and report the estimate and uncertainty generated by the specified analysis.
Diagnostics, Alternatives, and Reporting Limits
Before finalizing a table, check that every row targets the same contrast and that the direction of the effect is unmistakable. Verify the imputation shifts’ signs and units, the number and proportion of missing outcomes by group, and whether assumed scores are possible on the endpoint’s scale. If the outcome is bounded, an assumed mean outside its possible range signals a problem; for non-normal or ordinal outcomes, a mean shift may also fail to describe how the outcome distribution changes. Report the model’s actual assumption rather than implying that a shift in a mean specifies every feature of a distribution.
A useful report can include a concise table and a forest plot of the estimates with confidence intervals. The table carries the clinical rationale and exact assumption; the plot makes the direction and spread of estimates easy to scan. A tipping-point summary answers a different question: how far must an assumption move before a chosen conclusion changes? It can complement the table, but should not replace scenario descriptions, uncertainty intervals, or a clear primary result.
Sensitivity results have important limits. They assess consequences of specified assumptions; they do not identify the unobserved outcomes from the observed data alone. A clinically plausible scenario is not proven by being prespecified, and a wide interval may reflect imprecision even when point estimates are similar. Conversely, a narrow interval does not remove uncertainty about whether the missing-data assumption is credible. Do not select a single scenario merely because it gives the most favorable or least favorable result.
- Wrong reasoning: “The point estimate stays positive in every scenario, so the result is robust.” Correction: State how much the estimates change, show their intervals, and explain that conclusions depend on the estimand and the specified assumptions. In the example, all point estimates are positive, but all three sensitivity intervals include zero.
- Wrong reasoning: “A delta of −8 means the treatment effect falls by 8 points.” Correction: The delta shifts the imputed mean only for the missing treatment outcomes. Its contribution to the arm mean is weighted by the missing fraction; changes in both arms also affect the contrast.
- Wrong reasoning: “The reference-based row is self-explanatory.” Correction: State whose outcome pattern is used as the reference, which participants are affected, and the endpoint scale. Name the distributional implementation when relevant.
For a polished report, include the analysis population, outcome definition and time point, effect scale and favorable direction, primary assumption, missingness counts, scenario rationale, estimate, and uncertainty. These details let a reader interpret the table without guessing what a label means.
State the endpoint, time point, analysis population, estimand, and effect direction before presenting scenario results.
Describe the assumed missing-outcome pattern in the outcome’s units and give its clinical rationale.
Use the same contrast and show each estimate with its standard error or confidence interval, alongside the primary analysis.
Describe changes in estimates and uncertainty without claiming that any sensitivity assumption is verified by the data.
Check Your Understanding
Answer each question using the definitions and hypothetical results above. Brief solutions follow the questions.
- If the missing treatment-group mean shifts down by 5 points and 20% of that group is missing, what is the resulting change in that group’s overall mean in the simple weighted-mean calculation?
- Why is the delta-scenario estimate 2.85 points rather than the primary 4 points minus 8 points?
- What can be concluded from the delta-scenario 95% interval of \((-1.17,\ 6.87)\) points?
- Name two details that should accompany a “control-reference” label in a sensitivity-results table.
Solutions. (1) The change is \(0.20(-5)=-1.00\) point in the overall treatment-group mean; only the missing fifth of the group receives the shift. (2) The treatment missing outcomes shift by −8, but comprise 20% of that group; control missing outcomes also shift by −3 and comprise 15% of that group. The weighted arm means are 60.40 and 57.55, so the difference is \(60.40-57.55=2.85\) points. (3) Under the stated analysis and interval procedure, the interval includes zero, so the analysis does not rule out a zero mean difference at the two-sided 5% level. It does not prove no effect or give the probability that the null is true. (4) State which group’s outcomes serve as the reference and which participants receive that assumption; also specify the outcome scale and, for an implemented reference-based model, its distributional rule.