Prespecified And Exploratory Sensitivity Scenarios

Why Scenario Status Matters

A sensitivity analysis asks how a treatment-effect conclusion changes under assumptions about unobserved outcomes. Its numerical results depend on those assumptions; they do not establish which assumptions are correct. Readers therefore need more than a table of effects indexed by scenarios. They also need to know when each scenario was chosen and what information was available when it was chosen.

That distinction is the point of labeling a scenario prespecified or exploratory. A prespecified scenario was documented before the relevant outcome results could influence its selection. An exploratory scenario was developed or selected after such results were available, or was not documented in advance. Exploratory analyses can be scientifically useful: they can reveal sensitivity, probe a concerning pattern, or motivate future work. But their status affects how much protection they provide against choosing assumptions that favor a desired conclusion.

Throughout, let \(\theta\) denote a set of sensitivity assumptions and let \(\widehat{\Delta}(\theta)\) be the estimated treatment effect under those assumptions. For examples, define a positive effect as favoring treatment. An interval calculated under \(\theta\) is conditional on that scenario; it does not incorporate uncertainty about whether \(\theta\) is appropriate.

What Should Be Prespecified?

A label is most informative when readers can tell what was fixed in advance. A broad statement such as “we will conduct sensitivity analyses” is not enough to establish that a particular scenario was prespecified. The record should say, as applicable, which assumptions or parameter values will be examined, which treatment-effect summary will be reported, how intervals will be calculated, and how results will be interpreted against any clinical threshold.

The timing should also be clear. Ideally, scenarios are set out in the protocol or a dated analysis plan before the trial begins. A plan written later can still be prespecified relative to the outcome information if the relevant results have not been examined, but the report should state when it was finalized and what information was available. If analysts saw outcome results before choosing a scenario, the scenario should not be described as prespecified merely because it was written down before the final report.

• Record the assumptions. Give enough detail to reproduce the scenario, including any arm-specific departures or parameter values.
• Give the rationale. Explain why the scenario was included, such as a clinically motivated departure from the primary assumption.
• Record the analysis plan. State the effect scale, interval procedure, and any decision summaries specified in advance.
• Preserve the timeline. Identify the dated protocol or analysis-plan version and disclose material changes.

Prespecification does not make a scenario automatically credible. A prespecified assumption may still be implausible, poorly justified, or too narrow to be informative. Conversely, an exploratory scenario can be carefully motivated and valuable. The distinction concerns the opportunity for outcome-informed selection, not the scientific quality of an assumption by itself.

Worked Results: Keep the Assumption and the Status Together

A useful report pairs each estimate with both its scenario and its status. The following examples use hypothetical data. In each, treatment effects and confidence intervals are conditional on the stated sensitivity assumptions.

Why Prespecification Changes Credibility

The credibility difference follows from a simple selection argument. Suppose an analyst examines several candidate scenarios after seeing the treatment results and then emphasizes the one with the most favorable estimate. Even if every calculation is correct, the reported scenario has been selected partly because of its result. Its apparent support for the conclusion is therefore less persuasive than it would be if the scenario had been fixed independently of those results.

Prespecification reduces this particular concern: the scenario cannot have been chosen because its result looked favorable if it was genuinely fixed before the relevant results were available. But it does not remove other uncertainties. The chosen assumptions may not represent the data-generating process, and the interval still reflects sampling uncertainty conditional on those assumptions rather than uncertainty over all possible assumptions.

A practical comparison is:

Worked Example: Reporting a Change in Direction Honestly

Imagine a hypothetical report with a prespecified scenario yielding an estimated treatment benefit of 0.30 points, with a 95% confidence interval from −0.19 to 0.79 points. During follow-up analysis, the team considers a more severe departure and obtains an exploratory estimate of −0.30 points, with a 95% confidence interval from −0.849 to 0.249 points.

A transparent report separates the two statuses and the conditional interpretations: “The prespecified scenario estimated a treatment-minus-control difference of 0.30 points (95% confidence interval −0.19 to 0.79). An additional, exploratory scenario based on more severe arm-specific departures estimated −0.30 points (95% confidence interval −0.849 to 0.249). These results indicate that the estimated direction depends on the assumptions; neither interval identifies which assumptions are correct.” The arithmetic and units are retained, and the exploratory result is not quietly folded into the prespecified analysis.

This wording does not claim that the treatment is beneficial under one scenario and harmful under another with certainty. Both intervals include zero, and both estimates are conditional on their respective assumptions. Nor does the exploratory result prove that the prespecified scenario was wrong. It identifies a sensitivity worth considering and, if appropriate, planning to examine in future work.

Diagnostics, Pitfalls, and Limits

The most useful diagnostics for this topic concern the analysis record and the internal consistency of the report. Check that the stated scenario matches the dated plan, that any changes are disclosed, and that the same label is used in the text, tables, and figures. A scenario described as prespecified should have enough detail in the prior record to distinguish it from alternatives that were considered later. If the record is incomplete, say so rather than implying certainty about its status.

A frequent mistake is to reason, “This scenario is clinically reasonable, so it can be called prespecified.” That is incorrect: plausibility and timing are different properties. The correction is to label the scenario according to when and how it was chosen, then separately explain its clinical rationale. Another mistake is to treat an exploratory result as worthless. The correction is to report it openly as exploratory and interpret it as evidence about sensitivity, not as a preplanned confirmatory result.

The main alternative is not to choose between prespecified and exploratory analyses, but to use both for distinct purposes: plan a defensible set of scenarios in advance, then use additional exploration to investigate limitations or unexpected patterns. This combination is informative only when the boundary is visible. If a prespecified scenario is changed after results are seen, report the original plan and the change; do not retroactively describe the modified scenario as the one planned all along.

Finally, neither label substitutes for a justification of the assumptions. Prespecification helps readers assess how the analysis was selected; it does not establish the truth of a missingness assumption or the adequacy of a model. The treatment-effect conclusion remains conditional on the scenario, and claims should be correspondingly qualified.