Why Conclusions Need an Explicit Structure
A treatment-effect estimate from a sensitivity analysis is not a result detached from assumptions. It is an estimate under a particular account of the unobserved outcomes, together with uncertainty conditional on that account. A useful report therefore connects four things: the assumption, the estimate and its interval, the clinical interpretation, and the limitation. Leaving any one of these implicit makes it difficult to tell what the result supports.
This tutorial develops a reusable reporting template rather than a new estimation method. Consider a continuous clinical outcome for which higher scores are better, measured in points at a prespecified follow-up. Define the treatment effect as the mean outcome in the treatment group minus the mean outcome in the control group. Positive effects favor treatment; zero is the null value of no mean difference. Suppose, for illustration, that a gain of 3 points is the prespecified minimum clinically important benefit. The units and direction must be stated because the same numerical estimate can mean different things on a different scale or for an outcome where lower is better.
A report should also identify the target population, outcome, follow-up time, treatment contrast, and analysis method. These define what the estimate refers to. For a sensitivity analysis, the assumption should be specific enough that a reader can distinguish scenarios. For example, a report might describe a reference analysis using a missing-at-random assumption and a sensitivity scenario in which imputed outcomes among missing treatment participants are shifted downward by a stated number of points relative to the reference model. If the control arm is treated differently, say so. A label such as “conservative scenario” is not a substitute for the actual assumption.
A Reusable Reporting Template
For each primary or prespecified sensitivity scenario, give the information in the same order. Keeping the structure fixed makes comparisons easier and reduces the risk that a favorable scenario receives fuller interpretation than an unfavorable one.
Name the population, outcome, follow-up time, treatment contrast, direction, and units. State any clinical threshold used to interpret the effect.
Describe the missing-outcome assumption and its arm-specific settings or other sensitivity values. Do not rely on an unexplained scenario label.
Give the estimate, its standard error if useful, and a confidence interval, stating the method used to construct the interval.
Compare the estimate and interval with both the null value and the clinical threshold. Distinguish the point estimate from the range of effects compatible with the interval.
Explain what remains assumption-dependent, what the interval does not capture, and whether the conclusion changes under other prespecified scenarios.
A confidence interval describes sampling uncertainty under the stated analysis and scenario. It does not, by itself, measure uncertainty about which missing-data assumption is correct. If different plausible assumptions produce different estimates, that variation should be reported as sensitivity of the conclusion, not hidden inside a confidence interval from one scenario.
Worked Examples
Worked Example: A Reference-Scenario Conclusion
Suppose a hypothetical randomized study evaluates a treatment effect on a symptom score at 12 weeks, in points, with higher scores better. The target is the mean treatment-minus-control difference in the study population. A gain of 3 points is considered clinically important. Under a reference analysis that assumes missing outcomes are missing at random conditional on the variables in the analysis model, the estimated effect is 5.00 points with \(SE=1.50\) points. Assume the estimator is approximately normally distributed conditional on this analysis and the standard error is valid. Use a two-sided 95% Wald interval.
The interval is estimate plus or minus 1.96 standard errors:
Assumption. The result is conditional on the stated missing-at-random assumption and the specified outcome model; it is not a result under every possible account of the missing outcomes.
Interpretation. The point estimate is a 5.00-point mean benefit, exceeding the 3-point clinical threshold. The 95% interval excludes zero, so it is incompatible with no mean difference at the corresponding two-sided 5% level under the model. However, it includes effects below 3 points. Thus, the estimate is above the clinical threshold, but the interval does not establish that the true mean benefit exceeds that threshold.
Limitation. The interval quantifies sampling uncertainty conditional on the reference assumption and model. It does not establish that missingness is missing at random or quantify uncertainty over competing missingness assumptions.
Worked Example: Comparing Two Sensitivity Scenarios
Use the same hypothetical outcome, direction, 12-week follow-up, 3-point clinical threshold, and treatment-minus-control estimand. Suppose the analysis plan prespecifies a reference scenario and a less favorable scenario in which missing treatment outcomes are shifted downward relative to the reference predictions, while the control-arm assumption is unchanged. The shift is specified in the analysis plan; the estimates below are outputs from the corresponding fitted analyses, not calculations from raw participant data. Assume for each scenario that the estimator is approximately normal and that its reported standard error is valid conditional on that scenario.
In the reference scenario, the estimate is 5.00 points with \(SE=1.50\). As shown above, the 95% interval is \([2.06,7.94]\) points. In the less favorable scenario, the estimate is 1.20 points with \(SE=1.60\). Its interval is:
Under the less favorable assumption, the point estimate is below the 3-point threshold, and its interval includes both zero and effects exceeding 3 points. The data therefore leave substantial uncertainty about the treatment effect under this scenario. In contrast, the reference interval excludes zero but includes effects below the clinical threshold. The appropriate comparison is not “effective” versus “ineffective” based only on whether an interval crosses zero. Rather, the results show that the estimated magnitude and its interpretation depend on the missing-outcome assumption.
Limitation. These intervals are conditional on their respective scenarios. The difference between the scenario-specific estimates is not itself a standard error, and the two intervals do not combine to give a probability that either scenario is correct. Report the scenario definitions and their prespecified status so that readers can judge the range of assumptions examined.
Worked Example: Separating Statistical and Clinical Uncertainty
In a second hypothetical analysis of the same kind of outcome, suppose a prespecified sensitivity scenario gives an estimate of 3.40 points and \(SE=1.10\) points. Use the same two-sided 95% Wald method and the 3-point clinical threshold. Assume the outcome model and standard error are adequate under this stated scenario.
The margin of error is \(1.96(1.10)=2.156\) points, so:
The point estimate exceeds the clinical threshold, and the interval excludes zero. Nevertheless, the interval includes values below 3 points. A complete conclusion is therefore: “Under the specified sensitivity assumption, the estimated mean benefit is 3.40 points (95% CI 1.24 to 5.56); the interval excludes no mean difference but includes effects below the 3-point clinical threshold.” This wording preserves both the statistical and clinical interpretation.
It would be inaccurate to say that the interval demonstrates a benefit of at least 3 points. The lower endpoint is 1.244 points, which is positive but below 3. The interval is also not a 95% probability statement about the fixed treatment effect. Under repeated sampling and the stated model assumptions, the interval procedure has 95% coverage; in this particular analysis, the interval is a summary of uncertainty.
Why the Interval Justification Matters
The Wald interval used in these examples follows directly from an approximate normal sampling distribution. If \(\hat{\Delta}\) estimates the treatment effect under a fixed sensitivity scenario and has standard error \(SE\), the assumed approximation is \((\hat{\Delta}-\Delta)/SE \sim N(0,1)\). The central 95% of a standard normal distribution lies approximately between \(-1.96\) and \(1.96\). Rearranging the corresponding inequality gives the interval:
This justification depends on the approximate normality and calibration of the standard error under the fitted model and the chosen assumption. For small samples, highly skewed estimates, boundary problems, or a different inferential procedure, this Wald interval may be inadequate; the report should use and name an appropriate interval method instead. Whichever method is used, the interval must be calculated under the same scenario as the estimate it accompanies.
Diagnostics, Comparisons, and Common Failure Modes
Before writing a conclusion, check that the scenario table and the prose agree. Verify the treatment-effect direction, units, clinical threshold, estimate, interval endpoints, and whether the interval includes zero or the clinical threshold. Confirm that each estimate is paired with the correct scenario and standard error. A swapped arm, reversed contrast, or threshold stated in different units can reverse the conclusion while leaving the arithmetic apparently plausible.
The structured template is useful alongside other summaries, but it does not replace them. A table can efficiently display estimates and intervals across many scenarios; a plot can show how effects change across a range of sensitivity settings. The narrative template explains what a selected estimate means and makes its assumptions visible. A single primary analysis is more concise, but alone it can conceal how conclusions respond to alternative assumptions. Conversely, a long list of scenarios without a defined target, clinical threshold, or explanation can overwhelm rather than clarify. Use the same reporting fields for the primary scenario and the prespecified sensitivity scenarios, and distinguish exploratory analyses where relevant.
Several warning signs suggest that reporting is going wrong:
- Vague assumptions. “A pessimistic analysis” does not tell the reader what was assumed about unobserved outcomes. State the actual scenario settings.
- Overstating the interval. An interval that excludes zero but crosses the clinical threshold does not establish a clinically important effect.
- Confusing uncertainty sources. A narrow interval within one scenario does not show that the result is robust to other scenarios.
- Selective emphasis. Highlighting only the most favorable scenario, particularly when alternatives were prespecified, can give a distorted account. Report the planned comparisons consistently.
- Calling a scenario “validated” by model fit. Good fit to observed data does not determine whether the unobserved outcomes follow that scenario. Fit diagnostics and sensitivity assumptions address different questions.
The main limitation of any assumption-conditional template is that clear reporting cannot make an unverifiable assumption true. Sensitivity analyses show how conclusions respond to specified alternatives; they do not generally identify which alternative is correct from observed data alone. The chosen scenarios should therefore have a clinical or scientific rationale, and the report should acknowledge when relevant assumptions or ranges were not examined.
Check Your Understanding
Answer each question using the stated treatment-effect direction and clinical threshold. Brief solutions follow the questions.
- An estimate is 4.2 points with a 95% interval of 0.5 to 7.9 points. Zero is the null and 3 points is the clinical threshold. What can be concluded?
- Why should a report not describe a 95% interval from one sensitivity scenario as covering uncertainty about whether that scenario is correct?
- A report gives an estimate and interval but labels the assumption only “conservative.” What essential information is missing?
- Under an assumption, an estimate is 2.0 points with \(SE=1.0\). Calculate a two-sided 95% Wald interval and interpret it relative to zero and a 3-point threshold.
Solutions. (1) The interval excludes zero, so it is compatible with a positive mean effect under the stated model, but it includes effects below 3 points. It does not establish a benefit of at least 3 points. (2) The interval reflects sampling uncertainty conditional on that scenario and its model; it does not assign probabilities to competing missingness assumptions. (3) The report must specify what the scenario assumes about missing outcomes, including relevant arm-specific settings or departures from the reference analysis. The label alone is not reproducible or interpretable. (4) The margin is \(1.96(1.0)=1.96\), giving \(2.0\pm1.96=[0.04,3.96]\) points. This interval narrowly excludes zero, but it includes effects below 3 points; under the stated scenario, evidence for a positive mean effect does not establish a clinically important benefit.