Pre-Specifying Sensitivity Result Summaries

Why Prespecify the Summary?

A sensitivity analysis can produce many treatment-effect estimates, each conditional on a different assumption about unobserved outcomes. If the reporting choices are made after those estimates are seen, it becomes easy—deliberately or not—to emphasize a favorable scenario, an appealing scale, or a convenient uncertainty display. Prespecifying the summary plan makes clear what will be reported regardless of which way the results turn out.

The plan is not a promise that every scenario will support one conclusion. It specifies the assumptions to examine, the quantities to summarize, and the rule for describing how conclusions vary. The results can then be read against decisions made before they were known. This is distinct from choosing a single “best” assumption: a sensitivity analysis describes how inferences depend on assumptions that cannot be settled from the observed data alone.

The estimand, its scale, and its direction must be explicit. For example, suppose the effect is defined as mean outcome under treatment minus mean outcome under control, measured in points, and higher values are better. Then positive effects favor treatment. If instead the outcome is time to an event, a risk difference, or a ratio, the direction and interpretation may differ. The plan should name the treatment contrast, outcome and time point, target population, scale, units, and direction considered beneficial. It should also state whether the sensitivity configuration changes assumptions about missing outcomes, the analysis model, or both.

What the Plan Should Fix

A useful plan makes the set of analyses reproducible, not just the final sentence of the report. Specify which assumptions vary, their plausible values or ranges, and any combinations that will be examined. If parameters vary independently, say so; if their values are constrained to move together, define that constraint. A grid of selected values is a finite set, whereas a range such as \(a\in[0,1]\) describes infinitely many configurations. The distinction matters when checking a rule that must hold across every planned configuration.

Next, specify the same core summary for each configuration: the effect estimate, uncertainty interval and confidence level, and—when useful—the standard error or other supporting quantity. State the method used to form the intervals and the assumptions needed for that method. For example, a normal-approximation interval \(\hat\theta(a)\pm1.96SE(a)\) relies on an adequate normal approximation for the estimator under configuration \(a\). If that approximation is not justified, the plan should name a more appropriate interval procedure. Reporting a confidence interval does not remove the need to state what it is conditional on: it is conditional on the configuration and the statistical model used to calculate it.

The plan should also fix how results will be displayed. A table can show every prespecified scenario and its estimate and interval; a plot can show the same quantities across an ordered sensitivity parameter. If a concise headline summary is needed, state in advance how it will be obtained and which result remains available to readers. A single representative scenario should not silently replace the full planned set.

Define Robustness as a Rule, Not a Reaction

A conclusion should be stated in relation to a prespecified criterion. Suppose higher values are beneficial and \(\delta\) is a clinically important benefit threshold, in the same units as \(\theta(a)\). One possible rule is to call evidence of clinically important benefit robust across the planned set only when the lower confidence limit exceeds the threshold for every planned configuration:

This is a universal condition. For a finite set, it can be checked by comparing the smallest lower limit with \(\delta\). But a range may contain infinitely many configurations, and a smallest lower limit may not exist. For an arbitrary set, do not replace the universal condition with a claim about a “smallest” lower limit unless a minimum is known to be attained. A strict bound \(\inf_{a\in\mathcal A}L(a)>\delta\) is sufficient for the universal condition, but it is not necessary: every \(L(a)\) can exceed \(\delta\) even when the infimum equals \(\delta\).

The threshold and direction must fit the outcome. A rule for clinically important benefit is not the same as a rule for any positive effect, absence of harm, or equivalence. For a harmful direction, state the corresponding rule with the inequality reversed. Do not define robustness as “the result is statistically significant” without specifying the effect scale and substantive criterion. Statistical significance and clinical importance answer different questions.

Also state what uncertainty display supports the rule. Intervals computed separately at each configuration are ordinarily pointwise intervals: each has its stated coverage under that configuration and model. They do not, merely by being displayed together, guarantee simultaneous coverage over all configurations. If the intended claim requires joint coverage across the whole set, the plan must specify an appropriate simultaneous procedure. Otherwise, describe the rule accurately as one applied to the planned set of pointwise intervals, rather than implying a simultaneous confidence guarantee.

Checking a Range of Assumptions

For a continuous sensitivity range, the plan should state how the universal rule will be checked. If \(L(a)\) is continuous on a closed, bounded set \(\mathcal A\), then it attains a minimum there. In that special case, checking that this attained minimum exceeds \(\delta\) is equivalent to checking \(L(a)>\delta\) for every \(a\in\mathcal A\). Without conditions that guarantee attainment, the universal statement remains the safest formulation. Numerical grids can help explore a range, but checking only a finite grid does not establish the rule for every point between grid values unless additional reasoning bounds what can happen between them.

Diagnostics, Alternatives, and Limits

Several warning signs suggest that the summary plan is not doing its job. The reported scenarios do not match the prespecified set; the effect scale changes between scenarios; an interval method or confidence level changes without explanation; or the headline conclusion is based on a favorable scenario while other planned results are omitted. A display that contains only a p-value can also obscure whether effects are clinically important and how uncertain they are. These are reasons to check the reporting against the plan, not reasons to redefine the plan after seeing the estimates.

A complete table is often the clearest primary display for a finite scenario set. A plot is useful when the sensitivity parameter has a meaningful order or continuous range, because it can reveal trends, crossings of a clinical threshold, or abrupt changes. A concise summary can accompany either display, but should not replace it. A worst-case summary can be useful when its direction and criterion are explicit; for an infinite set, its calculation needs justification rather than a grid-based assertion. These displays answer complementary questions and are not interchangeable with a single “preferred” scenario.

Prespecification improves transparency, but it cannot make implausible assumptions plausible or identify unobserved outcomes from observed data alone. Robustness is always relative to the assumptions included, effect scale, uncertainty procedure, and criterion chosen. Results outside the planned set may be informative, but they should be identified as exploratory and distinguished from planned analyses. Finally, a pointwise interval-based robustness rule does not by itself establish simultaneous confidence coverage over many scenarios, nor does robustness under the chosen set prove that the assumptions encompass every credible possibility.