Why Multiplicity Matters in Sensitivity Analysis
A treatment-effect sensitivity analysis may calculate an estimate under several assumptions about missing outcomes, model structure, or other uncertain features of the analysis. The purpose is to examine how conclusions depend on those assumptions. A risk arises when many scenarios are examined but only the most favorable estimate, interval, or p-value receives emphasis. Readers may then see an apparently strong result without seeing how much the reported conclusion depended on selecting that scenario.
This is a problem of selective reporting, and it is related to—but not identical to—the usual multiple-testing problem. Sensitivity scenarios often use the same participants and observed data, so their estimates are dependent. More importantly, each scenario can represent a different assumption-conditional analysis, rather than a separate test of a distinct scientific hypothesis. There is therefore no automatic rule that every sensitivity analysis must apply a multiple-comparison correction. The key questions are what claim is being made, how the scenario set was formed, and whether a favorable result was selected after examining the results.
A scenario-specific interval answers a conditional question: what range is compatible with the data and the stated inferential procedure if that scenario’s assumptions and model are used? It does not, by itself, account for choosing that scenario because its result looked attractive. Earlier material on confidence intervals conditional on each assumption and confidence envelopes over sensitivity sets distinguishes these inferential summaries; the reporting issue here is whether the full set and the basis for emphasis are visible.
When the Familiar Multiplicity Calculation Applies
The standard family-wise error rate (FWER) is the probability of making at least one false rejection among a specified family of null hypotheses. If \(K\) tests are independent and each is conducted at level \(\alpha\), and all \(K\) null hypotheses are true, then the probability of no false rejection is \((1-\alpha)^K\). Thus the probability of at least one false rejection is:
This calculation assumes independence and the same per-test type I error rate. Scenario results are often dependent, so it is not usually an exact description of their joint behavior. For any dependence structure, the union bound gives a conservative result: if each of \(K\) valid tests uses level \(\alpha/K\), the probability of one or more false rejections is at most \(\alpha\). This is the Bonferroni procedure. It controls FWER for the stated family when the individual p-values are valid for their respective null hypotheses.
The distinction matters in sensitivity analysis. If investigators make a confirmatory claim such as “at least one of these prespecified treatment-effect claims is statistically significant,” a family-wise procedure may be relevant. If they instead show how one treatment-effect estimate changes across assumption-indexed scenarios, applying a correction mechanically to every scenario p-value may answer no useful scientific question. The scenarios are not necessarily independent replications, and their p-values need not correspond to a single family of claims the analysis was designed to test.
Worked Example: The Chance of at Least One Small P-Value
Suppose, purely for illustration, that eight independent tests are each conducted at \(\alpha=0.05\), and all eight null hypotheses are true. Find the probability of at least one false rejection. Then find the Bonferroni per-test threshold for controlling the FWER at 0.05.
Under the stated independence and global-null assumptions, each test has probability \(1-0.05=0.95\) of not rejecting. The probability that all eight tests do not reject is \(0.95^8\). Therefore:
The chance of at least one false rejection is about 33.7%, not 5%, under this particular set of assumptions. For Bonferroni control with \(K=8\), use:
This threshold guarantees FWER no greater than 0.05 for eight valid tests, whether or not the tests are independent. It does not follow that eight sensitivity scenarios should automatically be treated as these eight tests. The calculation illustrates the consequence of selecting among multiple tests when making a family-wise claim; actual scenario analyses may be dependent and may not define such a claim.
How Selection Can Change the Story
Suppose an analysis includes scenarios that differ in assumptions about outcomes after dropout. If the analyst reports only a scenario with a larger estimated benefit and a small p-value, the reader cannot tell whether that scenario was chosen in advance, whether it was one among many, or whether the treatment effect was similar across the remaining scenarios. A favorable scenario may be scientifically informative, but it should not silently replace the broader sensitivity analysis.
Selection can affect both the reported estimate and the apparent strength of evidence. Even if every scenario-specific estimate is unbiased under its own assumptions, choosing the largest estimate after observing the results tends to make that selected estimate larger than the unselected estimates. Similarly, emphasizing the smallest p-value from many tests tends to make the displayed p-value smaller than it would be for a prespecified single test. The precise amount of this effect depends on the joint distribution and the selection rule; it cannot be inferred from the number of scenarios alone.
Worked Example: A Favorable Scenario Among Several
Consider a hypothetical trial reporting treatment effects in percentage points, where positive values favor treatment. The analysis uses four stated missing-outcome scenarios. The following values are illustrative; each interval is a 95% confidence interval calculated under its corresponding scenario.
| Scenario | Estimated effect | 95% confidence interval | Two-sided p-value |
|---|---|---|---|
| A | 2.0 percentage points | −1.5 to 5.5 | 0.26 |
| B | 3.1 percentage points | −0.4 to 6.6 | 0.082 |
| C | 4.2 percentage points | 0.6 to 7.8 | 0.022 |
| D | 1.4 percentage points | −2.0 to 4.8 | 0.42 |
Scenario C has the largest estimate and the only interval in this table that excludes zero. If a report states only that “the treatment effect was statistically significant, \(4.2\) percentage points, \(p=0.022\),” that statement omits the fact that the analysis also produced three less favorable results under other assumptions.
A more informative account reports all four estimates and intervals, identifies the assumptions defining each scenario, and states which scenario or scenarios were specified before examining results. It can then say, for example, that the estimated benefit ranged from 1.4 to 4.2 percentage points, and that the interval excluded zero only under scenario C. This description does not declare that scenario C is invalid; it makes the dependence of the conclusion on the assumptions explicit.
These figures do not establish that C is “the correct” scenario, nor do the three intervals containing zero demonstrate no effect in their respective scenarios. They show that the estimated effects and statistical conclusions differ across the stated analyses. Whether the differences are scientifically important depends on the context, the plausibility of the assumptions, and the clinical threshold relevant to the outcome.
A Reporting Standard That Makes Selection Visible
Transparent reporting lets a reader reconstruct the relationship between the complete scenario set and the claims made from it. A useful report does not need to imply that every scenario is equally plausible or equally important. It should, however, distinguish prespecified analyses from exploratory analyses and make the reason for any prioritization clear.
State the number of scenarios, the assumptions that differ, and the outcome and treatment-effect scale being reported.
Say which scenarios and summaries were specified before results were examined, and identify additions or changes made during analysis.
Report estimates and uncertainty measures in a table or figure that includes the complete stated set, not only favorable rows.
Describe which conclusions hold across the set and which depend on particular assumptions; do not present a selected scenario as an unconditional result.
If formal claims are made across a family of tests, define the family, state the procedure, and report adjusted results where applicable. Explain why the procedure fits the claim.
A table is often more transparent than a single highlighted p-value. A graph can help show patterns across scenarios, but it should not obscure omitted results or leave scenario definitions unclear. If a figure emphasizes a primary scenario, the remaining results should still be accessible and the reason for that emphasis should be stated. This complements graphical summaries of assumption-indexed effects: the visualization communicates the pattern, while reporting practices make the analysis set and selection process inspectable.
Worked Example: Choosing an Appropriate Multiplicity Statement
Suppose a hypothetical report evaluates five prespecified sensitivity scenarios. Investigators calculate a p-value for each and obtain \(0.11, 0.08, 0.04, 0.03,\) and \(0.01\). They want to make one claim: “At least one scenario gives evidence of a nonzero treatment effect.” For this illustration, assume each p-value is valid for its scenario-specific null hypothesis.
If the claim is treated as a family-wise claim across all five tests, a Bonferroni threshold at \(\alpha=0.05\) is:
The smallest p-value is exactly \(0.01\), so it meets the Bonferroni rejection rule \(p\leq 0.01\). The correct interpretation is that, under the specified family and validity assumptions, the Bonferroni procedure rejects the corresponding null at a family-wise level no greater than 0.05. It is not a probability that the null is true, and it does not show that every sensitivity scenario supports a nonzero effect.
If the intended purpose is instead to describe how the estimate changes across assumptions, the p-values alone are not an adequate summary. The report should show the scenario-specific estimates, intervals, assumptions, and the pattern of results. Whether the five tests form a meaningful family for formal multiplicity control depends on the stated scientific claim; the calculation cannot decide that question.
Diagnostics, Alternatives, and Limits
Warning signs of selective reporting include a methods section that promises several scenarios but results that show only one; unexplained changes to the scenario set after results were available; a conclusion based on the smallest p-value without identifying the other tests; and different standards of emphasis for favorable and unfavorable scenarios. A reader should also be able to distinguish the primary analysis from sensitivity analyses and see whether the main conclusion changes as assumptions vary.
There are several complementary reporting choices, but they serve different purposes:
| Approach | What it clarifies | What it does not do |
|---|---|---|
| Complete scenario table or figure | Shows the pattern of estimates and uncertainty across the reported assumptions | Does not itself control a family-wise error rate |
| Bonferroni or another multiplicity procedure | Controls a stated error rate for a defined family of valid tests | Does not assess assumption plausibility or replace complete reporting |
| Confidence envelope over a sensitivity set | Summarizes uncertainty over a defined set of assumptions | Does not excuse hiding scenario results or explain post hoc selection |
No presentation can make an untestable sensitivity assumption empirically verified merely by reporting it. Conversely, the existence of multiple scenarios is not itself evidence that the analysis is invalid. Sensitivity analyses are intended to examine dependence on assumptions. Their value is reduced when results are selectively displayed, or when readers cannot tell whether the set was chosen before or after seeing the results.
Common mistake: “The smallest p-value is below 0.05, so the treatment effect is significant, and the other scenarios are just secondary.” This reasoning treats a result selected from the analysis set as though it had been the only planned test, without establishing that the selection was prespecified or addressing the full set. The correction is to report the complete scenario set, explain the status of the selected analysis, and match any formal multiplicity procedure to the actual family-wise claim. A favorable result can be discussed, but its dependence on its assumptions and its place among the analyses must remain clear.
Check Your Understanding
Answer the questions, then compare your reasoning with the brief solutions.
- Under independence, what is the probability of at least one false rejection among six tests, each conducted at \(\alpha=0.05\), when all six null hypotheses are true?
- What Bonferroni per-test threshold controls the FWER at 0.05 for a family of six valid tests?
- Why does reporting only the scenario with the smallest p-value create a concern even when each scenario-specific p-value is calculated correctly?
- Does a complete table of estimates and intervals automatically control the family-wise error rate? Explain.
- State two pieces of information that make a sensitivity analysis more transparent.
Solutions. (1) Under independence, \(1-(1-0.05)^6=1-0.95^6=0.2649\), or about 26.5%. (2) The threshold is \(0.05/6=0.00833\) to three significant figures. (3) Selecting the smallest observed p-value makes it appear as if a favorable result came from a single prespecified analysis; the selection and other results are hidden, so the displayed evidence may overstate how compelling the analysis set is. (4) No. A table supports transparent description, but it is not an error-rate procedure. (5) Examples include identifying the assumptions and number of scenarios, distinguishing prespecified from exploratory analyses, reporting all estimates and intervals, and explaining any formal multiplicity procedure.