Why Compare Scenarios Side by Side?
A sensitivity analysis often produces several plausible descriptions of the unobserved outcomes. Each description corresponds to a delta scenario: a specified departure from the reference imputation assumptions for missing values. The practical question is not only what an estimate is under one scenario, but whether the conclusion changes across the prespecified alternatives.
A useful comparison places each scenario’s estimate, uncertainty interval, and decision classification together. This makes it possible to distinguish a modest movement in the estimate from a change in the conclusion, such as moving from evidence of benefit to an inconclusive result. The comparison is meaningful only when every scenario addresses the same estimand and uses a coherent set of assumptions.
The delta parameters may be defined on an outcome scale or a model scale, depending on the imputation model. Their values and direction must be interpreted using the convention chosen for the analysis. The comparison does not determine which scenario is true. It shows what the analysis implies under each stated assumption.
Conditions for a Fair Comparison
Before comparing numerical results, check that the scenarios are comparable. The outcome, target population, contrast, and estimand must be the same in every row. For example, a treatment-minus-control difference should not be compared with a control-minus-treatment difference without first putting them on the same scale and orientation. Estimates should also use the same units, interval level, and decision rules.
Within each scenario, the imputation and analysis procedures must be appropriate for the stated assumptions. The reference imputation model and delta adjustment must be implemented as intended; the analysis model must target the same estimand in every scenario; and the reported uncertainty interval must be valid for that scenario’s analysis. A comparison cannot repair a misspecified imputation model or an incorrectly applied delta.
The scenario set should be chosen before inspecting which scenarios preserve a desired conclusion. Its values should have a defensible interpretation, informed where appropriate by calibration to external information and by checks of the implied distributions. This helps prevent a comparison from becoming a search for assumptions that produce a preferred result. Each scenario’s interval describes uncertainty conditional on that scenario. It does not, by itself, account for uncertainty about which delta scenario is appropriate.
State the estimand, contrast direction, population, outcome, and units to be held constant across scenarios.
Record each scenario’s full delta specification and its interpretation, not just a short label such as “high.”
For each scenario, present the estimate and uncertainty interval on the same scale and at the same confidence level.
Use a stated statistical or practical threshold consistently; distinguish a point-estimate classification from an interval-based conclusion.
Report what remains stable, what changes, and that conclusions are conditional on the scenarios examined.
Comparing Estimates and Intervals
Let \(s\) index the prespecified scenarios. For scenario \(s\), suppose the analysis supplies a pooled estimate \(\hat{\theta}_s\), its standard error \(SE_s\), and degrees of freedom \(df_s\). A two-sided interval at confidence level \(1-\alpha\), when the stated analysis uses a \(t\)-based interval, is
This expression is applied separately within each scenario. Comparing intervals means comparing their locations and widths; it is not a formal test that two scenario-specific estimates differ. The scenarios generally reuse the same observed data, so their estimates are not independent. Overlap or non-overlap of separate intervals is not a substitute for a procedure designed to estimate uncertainty in a difference between scenarios.
For a finite set of scenarios, the point-estimate range can be summarized as
This range is a descriptive measure of sensitivity to the selected assumptions. It is not a standard error, confidence interval, or probability distribution for the unknown delta. Its value depends on which scenarios were included, so it must be reported together with their definitions.
Worked Example: Comparing Continuous-Outcome Estimates
Suppose a hypothetical trial estimates the treatment-minus-control difference in a continuous symptom score, measured in points, with higher values indicating greater improvement. Three prespecified delta scenarios represent progressively less favorable assumptions for missing treatment-arm outcomes. The scenario-specific analyses give the following estimates, standard errors, and degrees of freedom. Assume the same 95% \(t\)-interval level is used throughout and that \(t_{0.975,40}=2.021\).
| Scenario | Estimate (points) | SE (points) | df |
|---|---|---|---|
| A | 5.20 | 2.00 | 40 |
| B | 4.40 | 2.10 | 40 |
| C | 3.60 | 2.20 | 40 |
For scenario A, the margin of error is \(2.021(2.00)=4.042\) points, so the interval is \(5.20\pm4.042\), or \([1.158,9.242]\) points. For scenario B, it is \(4.40\pm2.021(2.10)=4.40\pm4.244\), giving \([0.156,8.644]\) points. For scenario C, it is \(3.60\pm2.021(2.20)=3.60\pm4.446\), giving \([-0.846,8.046]\) points.
The point estimates range from 3.60 to 5.20 points, a sensitivity range of \(5.20-3.60=1.60\) points. The estimates are all positive, but the 95% interval in scenario C includes zero. Under a rule that calls the result statistically significant when its two-sided 95% interval excludes zero, scenarios A and B meet that rule and scenario C does not. This difference in classification does not establish that the scenarios’ estimates differ statistically from one another.
Decision Classifications Need Explicit Rules
A decision classification is not a property of an estimate alone. It depends on the estimand’s orientation, the threshold, and the rule being applied. For a two-sided test of \(H_0:\theta=0\), a 95% interval entirely above zero indicates evidence of a positive effect at the corresponding 5% level under the interval procedure. An interval that includes zero does not establish that the effect is zero; it indicates that the interval does not exclude zero.
Practical decisions may use a different threshold. A clinical threshold can be applied to a point estimate, to an interval, or to a decision rule involving consequences. These choices answer different questions. For example, a point estimate beyond a clinically important threshold does not mean the interval supports effects beyond that threshold. State which classification is being reported rather than using “clinically meaningful” as an undefined label.
Worked Example: Statistical Evidence Versus a Clinical Threshold
In a hypothetical study of an adverse binary outcome, define the treatment-minus-control risk difference in percentage points, so negative values favor treatment. The prespecified practical threshold is a reduction of at least 5 percentage points. Use the same 95% interval rule as above, with \(t_{0.975,80}=1.990\).
| Scenario | Estimate (percentage points) | SE (percentage points) |
|---|---|---|
| A | -7.0 | 2.5 |
| B | -5.2 | 2.6 |
| C | -3.4 | 2.7 |
For A, the margin is \(1.990(2.5)=4.975\), giving \(-7.0\pm4.975=[-11.975,-2.025]\) percentage points. For B, the margin is \(1.990(2.6)=5.174\), giving \(-5.2\pm5.174=[-10.374,-0.026]\). For C, the margin is \(1.990(2.7)=5.373\), giving \(-3.4\pm5.373=[-8.773,1.973]\).
Under the stated statistical rule, A and B have intervals entirely below zero, while C does not. If the practical classification is based on whether the point estimate reaches a reduction of at least 5 percentage points, A and B reach that threshold and C does not. But none of the three intervals lies entirely at or below \(-5\) percentage points. Thus the estimates reach the practical threshold in two scenarios, while the intervals do not establish that the reduction is at least 5 points in any scenario. These are different statements, not conflicting calculations.
Coherence, Diagnostics, and Failure Modes
A scenario table should make the assumptions traceable. For a treatment comparison, for example, an alternative may specify linked shifts for missing outcomes in both arms. A scenario that makes missing treatment outcomes less favorable and missing control outcomes more favorable can be coherent if that joint pattern was prespecified and justified. Mixing the treatment-arm adjustment from one scenario with the control-arm adjustment from another after seeing their results may instead create an unplanned assumption set.
Check the following before interpreting a comparison:
- Same target: Confirm that every row estimates the same contrast, in the same direction, population, and units.
- Complete scenario definitions: Include all relevant delta values or rules, including those for different groups or patterns, rather than relying on vague labels.
- Comparable uncertainty summaries: Use the same interval level and ensure each interval corresponds to its own scenario-specific estimate and uncertainty calculation.
- Reasonable implied values: Look for impossible values, excessive concentration at a boundary, or implausible distributions under any scenario. Such patterns can signal that a large shift is interacting poorly with the outcome scale or model.
- Stable implementation: Check that unexpected irregularities are not due to coding errors, changed analysis specifications, or insufficient imputation and computation.
A smoothly changing estimate does not guarantee that every scenario is plausible, and an abrupt change should prompt investigation rather than automatic dismissal. Conversely, a conclusion that remains unchanged over a limited set of scenarios is robust only to that set, not to every possible departure from the reference assumptions.
How This Comparison Differs from Other Summaries
| Approach | What it contributes | What it does not establish |
|---|---|---|
| One reference scenario | A focused estimate and interval under one stated assumption set | Whether conclusions would change under alternatives |
| Side-by-side scenario comparison | How estimates, intervals, and classifications vary over a prespecified set | A probability that any scenario is correct or a formal test between scenarios |
| Sensitivity curve or grid | How an estimate or decision changes across a range of delta values | Whether the examined range exhausts plausible assumptions |
| Tipping-point analysis | Where a prespecified conclusion first changes as assumptions vary | A summary of every feature of the scenario-specific uncertainty |
A scenario comparison is often clearer than reporting only a reference result, particularly when the decision changes across plausible assumptions. A sensitivity curve or tipping-point analysis can add information about behavior between selected scenarios. Neither approach removes the need to justify the assumptions or specify the decision rule. Choosing one scenario as “best” or averaging results across scenarios requires additional justification; the side-by-side table alone supplies neither a probability weight for a scenario nor a model for uncertainty about delta.
Common Mistakes and Interpretation Limits
- Calling the scenario range a confidence interval. The range summarizes the estimates under the scenarios selected. It has no automatic confidence level.
- Comparing confidence intervals as if they were a test of the difference. Separate intervals describe uncertainty within scenarios. Their overlap does not provide a formal test between scenario estimates.
- Changing the decision rule between rows. A classification is comparable only when the same threshold and rule are applied throughout.
- Conflating statistical and practical conclusions. Excluding zero is not the same as exceeding a clinically important threshold.
- Describing conditional intervals as covering uncertainty in delta. A scenario-specific interval conditions on its delta specification. It does not include uncertainty about that specification.
The central limitation is that a finite scenario set cannot show what happens under assumptions it does not contain. Conclusions should therefore name the scenarios examined and describe stability precisely: for example, “the interval excluded zero in two of the three prespecified scenarios,” rather than “the result is robust” without qualification.
Check Your Understanding
Answer the questions, then check the solutions below.
- Two scenarios give estimates of 2.0 and 3.5 units. What is their point-estimate sensitivity range, and what does it not represent?
- A scenario estimate is \(-4\) percentage points with \(SE=2\) percentage points and a 95% critical value of 2. State the interval and whether it excludes zero.
- Why is it not valid to call a scenario with an interval that includes zero “no effect”?
- What must be held constant for a side-by-side scenario comparison to be interpretable?
Solutions. (1) The range is \(3.5-2.0=1.5\) units. It is a descriptive range over the scenarios examined, not a confidence interval or standard error. (2) The interval is \(-4\pm2(2)=[-8,0]\) percentage points; it includes zero, so it does not exclude zero. (3) Inclusion of zero means the interval procedure does not rule out zero at the stated level; it does not establish that the true effect is exactly zero. (4) At minimum, hold the estimand, population, contrast direction, outcome scale and units, interval level, and decision rule constant; vary only the prespecified delta assumptions being compared.