Why the Results Narrative Must Name Its Assumptions
A sensitivity analysis can produce several treatment-effect estimates, each conditional on a different assumption about unobserved outcomes. A useful report does more than list those numbers: it tells readers which conclusions hold across the scenarios considered, which change, and what assumptions distinguish the scenarios. Without that connection, a reader cannot tell whether the result is robust to the specified assumptions or depends on a particular one.
The goal is not to announce that one scenario is “correct” because it gives a preferred answer. Sensitivity scenarios represent alternative assumptions, not observed versions of the missing data. A model’s fit to observed data does not, by itself, establish which missingness assumption is right. Instead, report the conclusions conditional on the assumptions and describe the scope of the scenarios examined.
The estimand and its scale must remain clear. In the examples below, \(\theta\) is the treatment-group mean minus the control-group mean for an outcome where lower scores are better. Thus, a negative effect favours treatment. A different outcome or sign convention could reverse that interpretation, so a report should state the scale and direction rather than relying on “positive” or “negative” alone.
A conclusion also needs a criterion. For instance, one may describe statistical evidence of benefit when a conditional 95% confidence interval lies entirely below zero. A separate clinical criterion might be whether the effect is at least 5 points better, represented by \(\theta\leq -5\). Whether an interval supports that clinical threshold is a distinct question from whether it excludes zero. The earlier material on confidence intervals conditional on each assumption, confidence envelopes over sensitivity sets, and clinical benefit thresholds under sensitivity analysis supplies those underlying concepts; here the emphasis is how to report their implications.
A Precise Rule for Stable and Changing Conclusions
Before writing, identify the scenarios being summarized and the conclusion rule applied to each. For a benefit rule based on a 95% conditional interval, define \(C_s=1\) if \(U_s<0\), and \(C_s=0\) otherwise. The benefit conclusion is stable over a set \(S\) exactly when all \(C_s\) values agree. If at least one differs, the conclusion changes somewhere in that set. This classification is conditional on both the intervals and the chosen rule; it is not a probability statement about which assumption is true.
The rule follows directly from the stated criterion: under scenario \(s\), the interval excludes zero in the beneficial direction precisely when its upper endpoint is below zero. If every scenario satisfies that inequality, benefit is supported in every scenario; if none does, it is supported in none. If some do and some do not, the classification changes. Equivalent rules can be written for a clinical threshold, but the threshold and direction must be specified. For example, an interval supports an effect of at least 5 points in the beneficial direction when \(U_s<-5\).
This binary summary should not replace the estimates and intervals. Two scenarios may receive the same classification while their effect estimates differ substantially. Conversely, a small shift can change the classification if an interval endpoint is near the decision boundary. Report enough numerical detail for readers to see both the magnitude and the source of the classification.
Worked Example: Statistical Benefit Holds Only Under Some Assumptions
Consider a hypothetical trial with a 0-to-100 symptom score, where lower scores are better. The estimand is the treatment-minus-control mean difference. Three prespecified scenarios use progressively less optimistic assumptions about unobserved treatment-group outcomes; the control-group assumption is held fixed. The labels describe assumptions, not observed data. Suppose the scenario-specific estimates and standard errors are:
- Scenario A, the reference assumptions: \(\hat{\theta}_A=-9.0\) points and \(SE_A=1.53\).
- Scenario B, a moderate departure from the reference assumptions: \(\hat{\theta}_B=-6.0\) points and \(SE_B=1.53\).
- Scenario C, a larger departure: \(\hat{\theta}_C=-3.0\) points and \(SE_C=2.04\).
For illustration, calculate 95% normal-approximation intervals as \(\hat{\theta}_s\pm1.96SE_s\). Under A:
Under B the same calculation gives \(-6.0\pm1.96(1.53)=[-9.00,-3.00]\) points. Under C it gives \(-3.0\pm1.96(2.04)=-3.0\pm4.00=[-7.00,1.00]\) points. The intervals are conditional on their respective assumptions and the stated approximation.
Using the rule \(U_s<0\), A and B support statistical evidence of benefit, whereas C does not: its interval includes zero. A careful report would say that evidence of benefit is maintained under the reference and moderate-departure assumptions, but not under the larger departure considered. It would not say that treatment is beneficial “in two-thirds of the scenarios”; the scenarios are not a random sample from a probability distribution over assumptions.
Suppose the prespecified clinical threshold is a reduction of at least 5 points. Only A has its entire interval below \(-5\). B’s point estimate is \(-6\), but its interval extends to \(-3\); therefore, the point estimate meets the threshold while the interval does not establish an effect at least that large. This distinction should be explicit in the narrative.
Writing the Change Without Overstating It
When a conclusion changes, identify the boundary between the scenarios in ordinary scientific language. State which assumption differs, how the estimate or interval changes, and which conclusion follows under each. Avoid attributing a change to “missingness” in general when the actual contrast is between specific assumptions, such as different departures in one arm while the other arm is held fixed. Also avoid implying that a scenario is more plausible unless that judgment is justified independently and clearly described.
A concise narrative can follow a simple sequence: define the estimand and direction; summarize conclusions that remain stable; identify the scenarios where the classification changes; state the assumption distinguishing those scenarios; and give the relevant estimates and intervals. This is a writing discipline, not a substitute for presenting the numerical results. A table may carry detailed scenario values, while the prose makes the cross-scenario interpretation clear.
Worked Example: Separating a Stable Direction From a Changing Threshold Conclusion
In a second hypothetical analysis, treatment minus control is measured in points on a functioning scale where higher is better. Here a positive effect favours treatment. A clinical benefit is defined in advance as at least 4 points. Two plausible sensitivity scenarios give:
- Scenario D, equal departures in both arms: \(\hat{\theta}_D=5.2\), with 95% conditional interval \([1.1,9.3]\) points.
- Scenario E, a larger adverse departure in the treatment arm: \(\hat{\theta}_E=4.4\), with 95% conditional interval \([0.3,8.5]\) points.
Both intervals lie above zero, so both support a positive treatment effect under the stated statistical criterion. But neither interval lies entirely above the clinical threshold of 4 points: \(1.1<4\) in D and \(0.3<4\) in E. Thus, the direction-of-effect conclusion is stable, while the stronger claim that the effect is at least 4 points is not established in either scenario. The point estimate in D exceeds 4, but that alone is not enough to say the interval supports the threshold.
A suitable report is: “The estimated effect favoured treatment under both examined assumptions (5.2 points under equal arm departures and 4.4 points under the larger adverse treatment-arm departure); both conditional 95% intervals excluded zero. Neither interval established an effect of at least 4 points.” This wording keeps the stable statistical conclusion separate from the clinical-threshold conclusion. It also avoids calling the result clinically robust merely because both point estimates exceed zero.
A change in classification often occurs because an estimate or interval endpoint crosses a decision boundary. Reporting the relevant boundary helps readers understand what changed without suggesting that the data identify which assumption is true. If the interval remains on the same side of zero but the point estimate crosses a clinical threshold, describe that as a change in threshold-level point-estimate classification—not as a change in statistical evidence of any benefit. Keep each claim tied to its own criterion.
Worked Example: Reporting a Conclusion That Reverses
A third hypothetical analysis considers a harmful-event risk difference, treatment minus control, in percentage points. Negative values favour treatment by indicating fewer events. The prespecified criterion for evidence of reduced risk is that the conditional 95% interval lies entirely below zero. Under a reference assumption, suppose \(\hat{\theta}_F=-2.4\) percentage points and \(SE_F=0.90\). Under a substantially different, prespecified assumption about unobserved outcomes, suppose \(\hat{\theta}_G=0.8\) percentage points and \(SE_G=0.85\).
Using the normal approximation, the first interval is:
It lies below zero, so it supports reduced risk under F. For G:
This interval includes zero and extends in both directions; it does not support a reduced-risk conclusion under G. The change is therefore not simply “the effect got smaller”: the estimate changes sign, and the interval no longer excludes zero. A balanced report would state that reduced risk was supported under the reference assumption but not under the substantially different assumption, and would name that assumption precisely in the final report. It would not conclude that treatment increases risk under G, because that interval also includes zero.
Diagnostics, Alternatives, and Common Pitfalls
Before finalizing the prose, check that each sentence can be traced to a scenario, assumption, estimate, interval, and decision rule. If the narrative calls a conclusion stable, verify that the criterion gives the same classification in every scenario included in the stated set. If it calls a conclusion changed, identify at least one scenario on each side of the classification boundary. Check units, the direction of benefit, the confidence level, and whether the clinical threshold is being treated as a point-estimate rule or an interval-based rule.
Several warning signs suggest the narrative is going wrong:
- Assumption labels are vague. “Under worse missingness” is not sufficiently informative if readers cannot tell which arm or departure changed.
- Only the preferred scenario is described. Omitting other prespecified scenarios can make a conditional result look assumption-free.
- Point estimates are treated as certainty. A point estimate beyond a threshold does not mean its confidence interval supports that threshold.
- Nonsignificance is called robustness. Failure to exclude zero in every scenario is not evidence that the effect is absent or unchanged; it may reflect uncertainty.
- Scenario counts are treated as probabilities. “Most scenarios show benefit” does not mean benefit is probable unless a defensible probability model over assumptions has been specified.
- Observed-data fit is used to select a missingness assumption. Good fit to observed outcomes cannot by itself validate assumptions about outcomes that were not observed.
The main alternative to prose alone is to present a scenario table or a sensitivity summary alongside the narrative. A table makes exact estimates and assumptions easier to inspect; prose explains the conclusion that follows from them. A single headline estimate is shorter but can conceal assumption dependence. A broad summary over a sensitivity set can describe the range of effects, but it should not erase which assumptions produce the endpoints or imply that every value is equally plausible. The earlier material on robust conclusions across assumption sets provides the relevant distinction between a conclusion holding over a specified set and a conclusion holding under one chosen scenario.
The limitation is fundamental: the narrative summarizes conclusions under assumptions; it cannot determine the truth of those assumptions from the sensitivity results alone. Stability applies only to the scenarios examined, the estimand and scale used, and the decision criteria stated. It does not guarantee stability under unexamined assumptions, another outcome scale, or a different clinical threshold. When uncertainty about the assumptions is itself central, report that limitation directly rather than presenting a scenario-conditional finding as unconditional.
Check Your Understanding
Answer each question by applying the stated rule and distinguishing the assumptions from the conclusions.
- A treatment-minus-control estimate is beneficial when negative. In scenario J, the 95% interval is \([-5,-1]\); in scenario K, it is \([-4,2]\). Using an interval entirely below zero as the rule for evidence of benefit, is that conclusion stable?
- A clinical threshold is an effect of at least 3 points in the beneficial direction, so the threshold is \(\theta\leq-3\). An interval is \([-6,-2]\). Does it establish an effect at least that large?
- Why is “benefit occurred in 4 of 5 scenarios, so the probability of benefit is 80%” not justified by a scenario count alone?
- Write one sentence for hypothetical scenarios with intervals \([-8,-2]\) and \([-7,1]\), where the second scenario assumes a larger adverse departure in the treatment arm.
Solutions. (1) No. J supports benefit because its upper endpoint is \(-1<0\); K does not because its interval includes zero. State that the conclusion changes between the two assumptions. (2) No. The upper endpoint is \(-2\), and \(-2\) is greater than \(-3\), so the interval is not entirely at or below the clinical threshold. (3) The scenarios are specified assumptions, not necessarily random draws from a probability distribution over assumptions. Their count does not give a probability. (4) For example: “Benefit was supported under the reference assumption, but not under the larger adverse treatment-arm departure, whose conditional interval included zero.”