Common Misreadings of Sensitivity and Uncertainty Displays

What a Display Shows—and What It Does Not

A sensitivity or uncertainty display can make a decision easier to inspect, but its visual simplicity can invite overinterpretation. A long bar in a tornado plot may look like evidence that an input is uncertain or causally important. A wide interval may look like a probability statement about a fixed effect. A probability-of-success value may look like a guarantee that a development decision will succeed. Each conclusion requires information beyond the graphic itself.

The first task is to identify what varies in the display. Is it an input changed across selected scenarios, an estimated quantity with sampling or model uncertainty, or a future outcome generated under a predictive model? Those are different questions. A display should also specify the outcome, units, reference case, ranges or probability model, and decision event to which it refers.

Reading a Tornado Plot Carefully

A common tornado plot summarizes one-at-a-time sensitivity analysis. Starting from a reference case, the analyst changes one input over a stated low-to-high range while holding the other inputs fixed. The plotted bar shows the resulting output values, often with the largest output range at the top. The output might be a predicted treatment effect, a probability of meeting a target, or a decision score. The axis and units must identify which.

For an input \(t\) varied over an interval \(I\), write the output as \(g(t)\), with all other inputs held at their reference values. If \(g\) has attained maximum and minimum values over \(I\), the range length is the maximum output minus the minimum output. For a continuous function on a closed, bounded interval, the extreme value theorem guarantees that both extrema are attained. For a general interval, they need not be: if \(g(t)=t\) on \((0,1)\), there is no attained maximum or minimum. When the output is bounded but extrema are not attained, describe the range width using \(\sup_{t\in I}g(t)-\inf_{t\in I}g(t)\), rather than an undefined maximum-minus-minimum.

A bar’s width therefore reflects both the model’s response and the particular input range chosen. It is not a measure of the input’s probability distribution. Nor does a large bar establish that the input is poorly known, that it is the main source of uncertainty, or that intervening on it would change the outcome in the same way. One-at-a-time analysis also holds other inputs fixed, so it may miss interactions or combinations of inputs that cannot plausibly occur together.

Interpreting Uncertainty Intervals

An interval cannot be interpreted correctly from its endpoints and confidence level alone. The reader needs to know what quantity the interval concerns and what procedure produced it. A frequentist confidence interval is constructed by a method that, under its assumptions and in repeated samples, covers the fixed parameter at its stated rate. A Bayesian credible interval has a posterior probability interpretation conditional on the model, prior, and observed data. A prediction interval instead concerns a future observation or outcome and includes the relevant variability in that prediction.

These intervals are not interchangeable. In particular, a 95% confidence interval is not ordinarily a statement that there is a 95% probability that the fixed parameter lies in the computed interval. Nor does a 95% prediction interval mean that 95% of future outcomes are guaranteed to fall within it. Its stated predictive coverage depends on the model and conditions used to construct it.

A useful diagnostic is to ask whether the plotted interval is for a parameter, a model prediction, or a future clinical outcome. Then check whether it is a confidence, credible, or prediction interval, what population and time horizon it refers to, and what assumptions its construction requires. If a graphic labels only “95% interval,” its interpretation is incomplete.

Probability of Success Is Conditional on a Defined Event

A probability-of-success display is meaningful only after “success” has been defined. It might mean that a treatment effect exceeds a clinical threshold, that a future trial meets a statistical criterion, or that efficacy and safety conditions are both met. These events are not equivalent. A model-based probability is conditional on the data, model structure, parameter uncertainty, assumptions about future observations, and the stated event.

A Short Justification for the Range-Width Reading

The width of a tornado bar is a summary of output variation over the specified input set, not a standalone sensitivity property of an input. For example, take \(g(t)=2t+3\) over the closed interval \([1,4]\). Because this function is continuous on a closed, bounded interval, it attains its minimum and maximum. Its endpoint outputs are \(g(1)=5\) and \(g(4)=11\), so its output range width is \(11-5=6\). If the input interval were changed, the width would change even though the function did not.

For a non-monotone function, endpoints alone may not give the extrema. For example, \(g(t)=-(t-2)^2\) on \([0,4]\) has maximum \(0\) at \(t=2\) and minimum \(-4\) at both endpoints, giving width \(0-(-4)=4\). Thus, a plot based only on low and high endpoint evaluations is adequate only when those evaluations capture the extremes, as they do for a monotone function on the interval. Otherwise, the analysis must search the interior or accurately label the display as showing endpoint scenarios rather than the full range.

Common Misreadings and Better Checks

• “The longest tornado bar identifies the most uncertain input.” Not necessarily. It identifies the greatest output span under the chosen input ranges and fixed settings. Show the ranges, their rationale, and whether uncertainty distributions were used.
• “The bar proves this input causes the output to change.” No. It reports a model response to a changed input. A causal claim needs an appropriate causal question and supporting assumptions and evidence.
• “A 95% interval gives a 95% chance that the fixed effect is inside.” That is not the standard frequentist confidence-interval interpretation. Name the interval type and explain its target and construction.
• “A 72% probability of success means the decision has a 72% chance of being correct.” Only if “correct” has been defined as the modeled event—and even then, the probability is conditional on the analysis assumptions, not a guarantee.
• “The inputs were varied one at a time, so the analysis covers their joint behavior.” It does not. One-at-a-time plots may miss interactions, dependence, or combinations that matter jointly.

Association requires particular care. If a model output changes when a covariate value is altered, that is a sensitivity result under the model. It is not, by itself, evidence that intervening to change the covariate would change the clinical outcome. The model may use the covariate as a predictor, and predictive association alone does not establish a causal effect.

Choosing and Checking a Display

The appropriate display depends on the question. A tornado plot is useful for comparing output changes over explicitly selected input ranges, particularly as an initial screening tool. A probabilistic uncertainty analysis is more appropriate when the question concerns variation across distributions for several uncertain inputs, with their dependence represented as needed. Scenario or stress-test comparisons are useful for examining coherent combinations of assumptions, including extreme but plausible conditions. These approaches answer related but different questions; none automatically repairs an unclear outcome definition or an unsupported model.

Before communicating a display, verify the reference case, axes, units, input ranges, interval type, event definition, and population and time horizon. Check whether the result is stable to reasonable alternatives and whether the display separates simulation error from model and parameter uncertainty. If the display invites a claim stronger than the analysis supports, revise the label or add the missing qualification.