Why Plot Effects Across Assumptions?
A set of sensitivity results can be numerically correct yet difficult to compare in a table. A forest plot or interval plot makes the movement of estimates and their uncertainty visible across specified assumptions. In this setting, however, each row represents an assumption set, not an independently sampled study or subgroup. The display must therefore make the assumptions legible and avoid suggesting that every row is equally plausible or that the rows can be averaged without justification.
The earlier material on confidence intervals conditional on each assumption establishes the interpretation of an interval within a fixed assumption set. Here the focus is how to display those estimates and intervals faithfully, and how to prevent visual choices from changing their meaning.
A forest plot or interval plot places each \(\hat{\theta}(a)\) on a shared horizontal effect scale and draws a horizontal interval from \(L(a)\) to \(U(a)\). A point, square, or circle marks the estimate; the line and its end caps mark the confidence limits. These plots are often called forest plots when multiple estimates are displayed in rows, but the familiar forest-plot layout does not make assumption rows comparable to separate randomized trials. In particular, there is no default pooled diamond or across-row average.
Designing the Display
Before plotting, decide what the horizontal scale represents and state its units and direction. For example, if \(\theta(a)\) is a risk difference defined as treatment risk minus control risk, negative values favor treatment. A vertical reference at zero marks no difference. If a clinically important benefit is defined by a risk difference of \(-5\) percentage points or lower, a second reference at \(-5\) percentage points can show that threshold. The direction and threshold must be stated rather than left for readers to infer.
Use one common, linear horizontal scale for all rows that share an effect measure. Show enough of the scale to locate the confidence intervals and relevant reference lines. Label the rows with the actual assumptions, not shorthand that requires readers to consult an unlinked document. If some scenarios are primary or judged more plausible on substantive grounds, identify those categories with text, spacing, or a separate column. Explain the basis for the categories. Do not let row order, marker size, or color imply a probability or ranking that the analysis did not establish.
An interval plot is especially useful when the intended comparison is a small set of conditional estimates and their intervals. A forest plot is a familiar way to arrange many such rows, and can be useful when rows are grouped by assumption family. Either format needs the same essentials: a labeled effect scale, point estimates, conditional confidence limits, the reference line or lines, and enough assumption detail to identify each row. A table of estimates may accompany the graphic when exact values or long assumption descriptions are important.
State the treatment-effect scale, sign convention, units, and the estimand being estimated.
Use a transparent, prespecified selection of scenarios and give each row a clear label.
Place \(\hat{\theta}(a)\) at its coordinate and draw the interval \([L(a),U(a)]\) on a common scale.
Mark zero and any prespecified clinical threshold, then label their meanings and directions.
Distinguish substantive plausibility categories from interval width, and say explicitly that the intervals are conditional on their listed assumptions.
Worked Example: A Forest Plot With Plausibility Groups
Worked Example: A Forest Plot With Plausibility Groups
Suppose a hypothetical trial estimates a treatment-minus-control risk difference in percentage points. Negative values favor treatment, and the prespecified clinical-benefit threshold is \(-5\) percentage points. The analysts report 95% normal-approximation intervals conditional on each assumption set, using \(\hat{\theta}(a)\pm1.96SE\). Their estimates and standard errors are:
| Assumption set | Substantive category | \(\hat{\theta}(a)\), pp | \(SE\), pp |
|---|---|---|---|
| Primary: specified arm-specific shifts | Primary | -7.0 | 2.5 |
| Alternative shift, mild | Also considered plausible | -6.0 | 2.6 |
| Alternative shift, moderate | Also considered plausible | -4.0 | 2.7 |
| Stress test, large shift | Deliberate stress test | -1.5 | 2.8 |
| Stress test, reversed direction | Deliberate stress test | 1.0 | 3.0 |
For the primary row, the interval is \( -7.0\pm1.96(2.5)=-7.0\pm4.9=[-11.9,-2.1]\) percentage points. For the mild alternative it is \( -6.0\pm1.96(2.6)=-6.0\pm5.096=[-11.096,-0.904]\), or approximately \([-11.1,-0.9]\). For the moderate alternative it is \( -4.0\pm1.96(2.7)=-4.0\pm5.292=[-9.292,1.292]\), or approximately \([-9.3,1.3]\). The two stress-test intervals are, respectively, \( -1.5\pm1.96(2.8)=[-6.988,3.988]\), approximately \([-7.0,4.0]\), and \( 1.0\pm1.96(3.0)=[-4.88,6.88]\), approximately \([-4.9,6.9]\).
To construct the plot, put all five rows on one risk-difference axis, for example from \(-15\) to \(+10\) percentage points. Mark zero and \(-5\). Put a point at each estimate and draw its interval to the calculated endpoints. Label the primary row and separate the two stress-test rows from the alternatives with a clearly named group label. The categories in this example are substantive judgments supplied by the hypothetical analysts; they are not calculated from the standard errors or confidence intervals.
The primary point estimate exceeds the clinical-benefit threshold in the beneficial direction, but its interval includes effects smaller than that threshold; it does not include zero or effects on the other side of zero. The moderate alternative interval includes zero, while the last stress-test estimate is on the unfavorable side of zero. These are conditional descriptions of the plotted rows, not probabilities that a particular scenario is true.
Why the Plot Is Conditional: A Short Justification
Suppose the interval procedure has 95% coverage when a particular assumption set \(a\) holds and its sampling model is appropriate. This means that across repeated samples generated under that same \(a\), the interval procedure covers \(\theta(a)\) about 95% of the time. It does not mean that the interval covers effects under all possible assumptions with 95% probability, or that 95% of the displayed rows are plausible.
The distinction follows directly from what the procedure targets. For fixed \(a\), the estimator and its standard error are calculated under \(a\), and the interval is formed around \(\theta(a)\). If \(a\) changes, both the target and potentially the estimate and standard error change. Thus the collection \(\{[L(a),U(a)]:a\in\mathcal{A}\}\) is a collection of assumption-conditional intervals, not a single interval procedure for a parameter that averages over \(\mathcal{A}\). An interval envelope over a sensitivity set is a different summary, with a different interpretation.
Worked Example: Constructing a Conditional Interval Plot
Consider a second hypothetical analysis on the same risk-difference scale, using the same 95% normal-approximation method. The primary assumption set gives \(\hat{\theta}=-6.4\) percentage points and \(SE=2.2\) percentage points. An alternative set gives \(\hat{\theta}=-2.0\) and \(SE=2.4\). The assumptions are not assigned probabilities.
For the primary set, the margin is \(1.96(2.2)=4.312\) percentage points, so the interval is \([-6.4-4.312,\,-6.4+4.312]=[-10.712,-2.088]\), approximately \([-10.7,-2.1]\). For the alternative, the margin is \(1.96(2.4)=4.704\), giving \([-2.0-4.704,\,-2.0+4.704]=[-6.704,2.704]\), approximately \([-6.7,2.7]\).
An interval plot puts the two points at \(-6.4\) and \(-2.0\), with whiskers reaching their respective limits. A zero line shows that the primary interval lies below zero while the alternative interval crosses zero. If a \(-5\)-point clinical threshold is relevant, mark it separately: the primary point estimate is beyond it, while the alternative point is not. These comparisons are descriptive and conditional; they do not show that the alternative is more or less likely because its interval is wider or crosses zero.
The assumptions behind the plotted intervals must be checked for the chosen method. In this example, the calculation assumes the stated normal-approximation interval is suitable for the estimates and standard errors under each fixed scenario. If an estimate is near a boundary, a normal approximation is poor, or the standard error omits important uncertainty, the displayed interval may not have the advertised conditional coverage. The plot cannot repair a deficient interval procedure.
Scale, Labels, and Diagnostics
Worked Example: Detecting a Misleading Axis
A third hypothetical report gives two risk-difference estimates, both in percentage points, with 95% normal-approximation intervals. Scenario A has estimate \(-3.0\) and \(SE=2.55\), so its interval is \(-3.0\pm1.96(2.55)=-3.0\pm4.998=[-7.998,1.998]\), approximately \([-8.0,2.0]\). Scenario B has estimate \(-5.5\) and \(SE=2.3\), so its interval is \(-5.5\pm1.96(2.3)=-5.5\pm4.508=[-10.008,-0.992]\), approximately \([-10.0,-1.0]\).
If the graph shows only an axis from \(-11\) to \(-6\), both intervals are cut off and zero is absent. The points may look close to a favorable boundary, while the missing parts hide that Scenario A's interval crosses zero and both intervals extend beyond the displayed range. The correction is to show a scale covering the full intervals and zero, or to use a clearly indicated axis break only when necessary and never to hide the omitted values. If a clinical threshold is relevant, include it as a labeled reference line rather than using it as an unexplained axis limit.
Several diagnostic questions help identify a plot that may mislead:
- Can a reader identify the precise assumption set for every row, including which arm-specific assumptions were used?
- Do all rows use the same effect scale, units, and horizontal scale? If not, are separate panels clearly labeled?
- Are interval endpoints visible, and are zero and relevant clinical thresholds shown without being confused?
- Are plausibility categories based on stated substantive rationale rather than on which rows produce favorable results?
- Are all displayed intervals conditional on a specified procedure and assumption set, rather than mislabeled as an envelope or an unconditional range?
Alternatives and Limitations
| Display | Useful for | Important limitation |
|---|---|---|
| Forest or interval plot | Comparing conditional point estimates and intervals across a manageable set of assumptions | Can visually overemphasize row-to-row changes or imply equal plausibility unless assumptions are explained |
| Scenario table | Showing exact assumption descriptions and numerical results | Less immediate for comparing the direction, spread, and threshold crossings of intervals |
| Confidence envelope over a sensitivity set | Summarizing the range covered by conditional confidence intervals over a specified set | Does not show each scenario's result as clearly and is not the same as a list of equally likely effects |
| Effect surface or faceted display | Showing how estimates vary with one or more continuously indexed sensitivity parameters | Can become difficult to read and still requires explicit assumptions, uncertainty, and scale labeling |
A plot is a communication device, not a validation tool. A tight interval under an implausible assumption does not make that assumption credible, and a model's fit to observed data cannot by itself verify an unobserved-outcome assumption. A graph also does not resolve which scenarios should be considered, establish their probabilities, or justify a pooled estimate. If there are many scenarios, display the prespecified set transparently and use an accompanying table or panel structure where needed rather than silently omitting inconvenient rows.
Common Pitfall
Correction: Each interval has a conditional coverage interpretation under its own assumption set and interval procedure. Row position does not encode plausibility, and the middle row is not a principled overall summary. Label assumptions and plausibility judgments separately; use a different, explicitly defined summary if an envelope or aggregate is required.
Check Your Understanding
Answer each question before reading the short solutions.
- For a treatment-minus-control risk difference, what does the zero reference line mean, and what must be stated about the sign convention?
- Why should a wider interval not be used to label an assumption set as less plausible?
- A conditional interval is \([-9,-2]\) percentage points and the clinical threshold is \(-5\). Does the interval cross zero, and does it contain effects that fall short of the threshold in the beneficial direction?
- What is wrong with interpreting the union of several conditional intervals as one 95% confidence interval without further justification?
- Name two pieces of information that should accompany an interval plot across assumptions.
Solutions. (1) Zero is no treatment-control risk difference; state which direction favors treatment and the effect units. (2) Width reflects sampling uncertainty under the specified procedure and assumptions, not the probability that those assumptions are true. (3) It does not cross zero; it contains values from \(-5\) to \(-2\), which are beneficial but smaller in magnitude than the threshold benefit. (4) Each interval targets an effect conditional on its own assumptions, so their union is not automatically a single interval with 95% coverage for one parameter. (5) Appropriate answers include the full assumption labels, interval level and method, effect scale and units, sign convention, and clearly labeled zero or clinical-threshold lines.