Why Sensitivity Analysis Is Needed
A simulated operating characteristic is conditional on assumptions. A design may appear to meet its target probability of success when outcomes follow the assumed distribution, yet perform differently if outcomes are more variable, a nuisance parameter takes another plausible value, or missingness depends on unobserved outcomes. Sensitivity analysis asks whether the conclusions about trial performance persist across such alternatives.
The aim is not to find a single “correct” scenario by simulation. Rather, it is to expose which assumptions matter, how much they matter, and whether plausible departures change a design decision. The design, the analysis model, and the decision rule should be made explicit so that each comparison has a clear interpretation.
A scenario specifies the features of the data-generating process relevant to the question: for example, the true treatment effect, baseline event rate, outcome distribution, overdispersion, accrual and censoring pattern, or missingness mechanism. The trial design and decision rule determine what data would be collected and how the data lead to a decision. To isolate the effect of a particular assumption, hold the design, decision rule, and other assumptions fixed while changing that assumption. Then consider combinations of departures when those combinations are scientifically plausible.
This is a robustness exercise, not a replacement for specifying the primary assumptions. It also differs from averaging operating characteristics across scenarios: a scenario-specific sensitivity comparison reports how performance changes under each assumption, whereas an average combines scenarios using explicit weights. Worst-case summaries answer a different question again. The earlier material on held-out scenarios concerns checking calibration on scenarios not used to tune it; sensitivity analysis can include such scenarios, but its main purpose is to quantify dependence on assumptions.
Estimating and Comparing Operating Characteristics
For each scenario \(s\), simulate \(R\) independent trial data sets from \(P_s\), apply the same decision rule to each, and record \(I_{rs}=\delta(Y_{rs})\). The Monte Carlo estimate of the success probability is the fraction of simulated trials meeting the criterion.
Because each indicator is Bernoulli with probability \(q_s\), the estimated Monte Carlo standard error is approximately \(\sqrt{\hat q_s(1-\hat q_s)/R}\). This quantifies simulation noise conditional on the scenario; it does not quantify uncertainty about which scenario is realistic. Increasing \(R\) reduces Monte Carlo error but cannot repair an implausible data-generating model.
For a difference between scenarios simulated independently, estimate \(\Delta_s\) by \(\hat q_s-\hat q_{s_0}\). Its estimated Monte Carlo standard error is \(\sqrt{\hat q_s(1-\hat q_s)/R_s+\hat q_{s_0}(1-\hat q_{s_0})/R_{s_0}}\). If simulations are paired using common random numbers, calculate a difference for each replicate and estimate the standard error from the variability of those paired differences. Pairing can improve precision when it induces positive correlation between scenario outcomes, but it should be designed so that each scenario still has the intended marginal data-generating distribution.
This paired standard error follows from treating the replicate differences \(D_r=I_{rs}-I_{rs_0}\) as independent draws with variance estimated by \(s_D^2\). Since the variance of their average is \(\operatorname{Var}(D_r)/R\), its estimated standard error is \(s_D/\sqrt R\). Pairing is useful for precision, not a way to make the scenarios more alike or to remove genuine sensitivity.
Worked Example: Sensitivity to Overdispersion
Suppose a hypothetical two-arm binary-endpoint trial declares success when its prespecified Bayesian posterior decision rule is met. The rule and analysis model are held fixed. In the reference simulation, outcomes follow independent Bernoulli distributions within each arm, with true event probabilities 0.20 in control and 0.35 in treatment. An alternative simulation preserves those arm-specific means but generates beta-binomial counts with within-arm correlation 0.05, representing extra-binomial variation. The two scenarios are simulated independently for \(R=1{,}000\) trials each.
In this hypothetical run, 820 of 1,000 reference trials and 680 of 1,000 overdispersed trials meet the success rule. Thus \( \hat q_{\mathrm{ref}}=820/1{,}000=0.820 \) and \( \hat q_{\mathrm{over}}=680/1{,}000=0.680 \). The estimated change is \( \hat\Delta=0.680-0.820=-0.140 \), a decrease of 14.0 percentage points in success probability under the overdispersed scenario.
The Monte Carlo standard error for this independently simulated difference is \[ \sqrt{\frac{0.820(1-0.820)}{1{,}000}+ \frac{0.680(1-0.680)}{1{,}000}} = \sqrt{0.0001476+0.0002176} = 0.0191. \] The estimated change is much larger than this simulation error. In this run, the conclusion that the design has high success probability is sensitive to the distributional assumption. This comparison does not establish that overdispersion is present; it shows the consequence for trial performance if the stated alternative is relevant.
Choosing Scenarios and Interpreting Results
Scenario choices should be clinically and statistically defensible. Varying a parameter over an arbitrary range can create impressive-looking changes without answering a useful question. Identify the assumptions that are both uncertain and consequential, specify plausible values or mechanisms for them, and state whether they are varied separately or jointly. For parameters that cannot vary independently, construct coherent combinations rather than assembling incompatible extremes.
A one-at-a-time comparison is useful for identifying which assumption drives a change, but it can miss interactions. For example, a higher baseline event rate might have a different impact when the treatment effect is expressed as a risk difference rather than a relative effect. After examining individual factors, evaluate a smaller set of joint scenarios motivated by the clinical setting. Keep the target estimand and success rule fixed unless the purpose is explicitly to study a change in either.
Worked Example: Sensitivity to a Nuisance Parameter
Consider a hypothetical binary-endpoint trial evaluated using the same design, Bayesian analysis, and success decision rule in two scenarios. The treatment and control event probabilities differ by a fixed risk difference of 0.08: they are 0.10 and 0.18 in the first scenario, and 0.25 and 0.33 in the second. The baseline event probability is a nuisance parameter for this comparison. Independent simulation produces 1,510 successes among 2,000 trials in the first scenario and 1,190 among 2,000 in the second.
The estimated success probabilities are \(1{,}510/2{,}000=0.755\) and \(1{,}190/2{,}000=0.595\). The estimated contrast is \(0.595-0.755=-0.160\), or a reduction of 16.0 percentage points at the higher baseline event probability.
The estimated Monte Carlo standard error is \[ \sqrt{\frac{0.755(0.245)}{2{,}000}+ \frac{0.595(0.405)}{2{,}000}} = \sqrt{0.00009249+0.0001205} = 0.0146. \] The decrease is large relative to the simulation error. It is therefore important to report both baseline scenarios rather than describe the design by its performance at only one event rate. The result is conditional on the stipulated true risk differences and decision rule; it does not imply that baseline event rate alone must have the same effect in other designs.
Missingness deserves its own sensitivity analysis because the mechanism can matter even when the overall proportion missing is held constant. Under missing at random (MAR), missingness may depend on observed information but not on the unobserved outcome after conditioning on that information. Under missing not at random (MNAR), missingness can still depend on the unobserved outcome after conditioning. These assumptions imply different observed-data distributions and can lead to different performance when the trial analysis relies on a MAR model.
Worked Example: Sensitivity to the Missingness Mechanism
In a hypothetical continuous-outcome trial, the true treatment contrast and the overall missing-outcome proportion are fixed across scenarios. The analysis uses its prespecified MAR-based model and declares success if the posterior probability that the treatment contrast exceeds its target is above the required cutoff. One data-generating scenario makes missingness depend on observed covariates, consistent with MAR; an alternative makes missingness also depend on the unobserved outcome, an MNAR departure. Each scenario is simulated independently for 1,000 trials.
Suppose the MAR scenario meets the rule in 730 trials and the MNAR scenario in 590. The estimated probabilities are \(0.730\) and \(0.590\), giving a difference of \(0.590-0.730=-0.140\), or 14.0 percentage points lower under the MNAR mechanism. The Monte Carlo standard error is \[ \sqrt{\frac{0.730(0.270)}{1{,}000}+ \frac{0.590(0.410)}{1{,}000}} = \sqrt{0.0001971+0.0002419} = 0.0210. \] The difference is larger than the simulation error. It indicates that the success probability depends materially on the missingness assumption in this hypothetical setup. It is not evidence that the MNAR mechanism is the true one; that judgment requires substantive knowledge about why outcomes are missing.
Diagnostics, Alternatives, and Limitations
A sensitivity analysis is only as useful as its implementation. Before interpreting differences, check that each scenario produces the intended data. Compare generated event rates, outcome variances, missingness proportions, and relevant covariate distributions with their specified values. For time-to-event settings, check event counts and censoring patterns. Confirm that the same analysis and decision rule are applied in every scenario when the purpose is to assess data-generating assumptions.
Also check simulation precision. If a reported difference is comparable to its Monte Carlo standard error, increase the number of replicates before drawing a firm conclusion. With a small number of successes or failures, normal approximations for Monte Carlo uncertainty can be poor; report the counts and use an appropriate binomial interval or more simulations. Do not interpret Monte Carlo uncertainty as uncertainty about the clinical truth or the scenario assumptions.
Several common approaches answer related but distinct questions. One-at-a-time sensitivity checks help attribute changes to individual assumptions. Joint scenario analysis evaluates combinations and interactions. Scenario averaging requires explicit weights and answers a weighted-average question; a worst-case summary emphasizes the least favorable scenario in the set. None of these approaches proves robustness outside the scenarios actually examined. Sensitivity analysis also cannot compensate for a misspecified decision rule or an incomplete set of plausible mechanisms.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A reference scenario has estimated success probability 0.80 from 1,000 simulations. An alternative has estimate 0.74 from 1,000 independent simulations. What is the estimated change, and what is its approximate Monte Carlo standard error?
- Why should the design and decision rule usually be held fixed when testing sensitivity to an outcome distribution?
- Two missingness scenarios have the same proportion of missing outcomes but different mechanisms. What is the main reason their operating characteristics might differ?
- When can paired simulation reduce Monte Carlo error for a scenario contrast, and what quantity should be used to estimate that error?
- Does a precise estimate under a single scenario establish that the design is robust? Explain.
Solutions. (1) The estimated change is \(0.74-0.80=-0.06\). Its approximate standard error is \(\sqrt{0.80(0.20)/1{,}000+0.74(0.26)/1{,}000} =\sqrt{0.0003524}=0.0188\). (2) Holding them fixed isolates the effect of the distributional change; changing the rule at the same time would confound the comparison. (3) The relationship between missingness and outcomes or observed covariates differs, changing which outcomes are observed and how well the analysis model handles them. (4) Pairing can help when common random numbers induce positive correlation between scenario outcomes while preserving each scenario’s marginal distribution. Estimate the standard error from the replicate-level differences, \(s_D/\sqrt R\). (5) No. Precision describes Monte Carlo error for that scenario, not performance under unexamined plausible assumptions; robustness requires comparisons across relevant alternatives.