Sensitivity Analysis for Nuisance Parameters

Why Examine Nuisance-Parameter Sensitivity?

A simulation estimates how a proposed trial design behaves under specified inputs. Some inputs define the scientific target, such as the treatment effect of interest. Others affect the distribution of the data, and therefore how often the design meets its objectives, without being the target themselves. These latter inputs are nuisance parameters. Examples include outcome variance, a control-group event rate, and correlation between a baseline measure and an outcome.

A design can appear satisfactory at one assumed value and perform differently at another plausible value. Sensitivity analysis makes this dependence visible: it asks how operating characteristics change when nuisance inputs are varied while the design and target effect are held fixed. The purpose is not to predict which nuisance value will occur, but to show which design conclusions depend on the assumed value.

Operating characteristics may include power, Type I error, the probability of stopping early, or expected sample size. For example, power is the probability that a specified analysis rejects \(H_0\) under a specified alternative. A sensitivity analysis needs a clear estimand, decision rule, and set of nuisance scenarios: changing an input without saying what else is held constant makes the result hard to interpret.

Specify the Sensitivity Analysis

Begin with a reference scenario and define a plausible range for each nuisance parameter. The reference scenario is a clearly stated set of assumptions, not necessarily the most likely truth. Values in the range should be clinically and scientifically credible, and consistent with the data model. Then choose which operating characteristics to estimate and how much simulation precision is needed.

A one-at-a-time analysis changes one nuisance parameter while holding the others at their reference values. It is useful for showing the separate effect of each input. It does not reveal whether parameters interact. A joint scenario grid evaluates combinations of nuisance values, so that, for example, low event rates can be considered together with high outcome variability. The earlier material on constructing null and alternative scenario grids establishes how to specify those combinations; here the emphasis is on the nuisance inputs within them.

For each scenario, simulate trial data under the stated model, apply the planned design and analysis, and record the outcome of interest. If \(R\) simulated trials produce \(K\) rejections under an alternative, the estimated power is \(\hat{p}=K/R\). Its Monte Carlo standard error is approximately \(\sqrt{\hat{p}(1-\hat{p})/R}\). A difference between scenarios should be interpreted alongside this simulation error; the earlier material on confidence intervals and Monte Carlo error gives tools for assessing its precision.

Outcome Variance and Power

For a simple illustration, suppose a two-arm trial has \(n\) independent participants per arm, a continuous outcome with common known standard deviation \(\sigma\), and a treatment effect \(\delta\). Assume normal outcomes, equal variances, complete outcome data, and a two-sided \(z\)-test at \(\alpha=0.05\). Under the alternative, the estimated difference in means has standard error \(\sigma\sqrt{2/n}\). With the normal approximation, power is the probability that this difference falls outside the two-sided null acceptance region.

Here \(\lambda\) is the standardized treatment effect under the alternative, and \(\Phi\) is the standard normal cumulative distribution function. The approximation treats the standard error as known; an actual analysis using an estimated variance and a \(t\)-test may have slightly different power.

Baseline Event Rate and Correlation

For binary outcomes, the baseline event rate affects the variability of the observed proportions. Its effect on power depends on the effect measure and the analysis, so it should not be assumed that a lower event rate always reduces power. The following approximate calculation illustrates a risk-difference analysis. Assume independent participants, equal group sizes, a two-sided normal-approximation test at \(\alpha=0.05\), and no continuity correction. For control event rate \(p_0\) and treatment event rate \(p_1\), the estimated risk difference has alternative standard error \(SE_A=\sqrt{\{p_0(1-p_0)+p_1(1-p_1)\}/n}\). Under the null, the pooled rate is \(\bar{p}=(p_0+p_1)/2\), giving \(SE_0=\sqrt{2\bar{p}(1-\bar{p})/n}\) and approximate rejection cutoff \(1.96SE_0\).

Correlation can also be a nuisance input when the analysis adjusts for a baseline outcome. In a simple normal model with a common baseline–outcome correlation \(\rho\), the adjusted residual standard deviation is \(\sigma\sqrt{1-\rho^2}\). For equal group sizes and otherwise the assumptions of the earlier normal example, the treatment-effect standard error is \(\sigma\sqrt{1-\rho^2}\sqrt{2/n}\). This formula assumes a correctly specified linear adjustment and a common correlation structure; it is not a general formula for every adaptive analysis.

Interpreting Results and Checking the Analysis

One-at-a-time results are easy to communicate, but can miss joint effects. If high variance and low correlation are both plausible, their combined effect need not equal the sum of their separate effects. A joint grid can reveal such patterns, provided every combination is meaningful under the model. For example, rates must remain between 0 and 1, and a correlation must remain between \(-1\) and \(1\). Dependence between nuisance inputs may also make some combinations implausible.

A useful report gives the reference value, the varied values, the operating-characteristic estimate in each scenario, and its Monte Carlo uncertainty. When the operating characteristic is estimated as a proportion, the Monte Carlo standard error is about \(\sqrt{\hat{p}(1-\hat{p})/R}\); it quantifies simulation noise, not uncertainty about which nuisance value will occur. If two scenario estimates differ by less than their simulation error would readily explain, the apparent change is not compelling evidence of sensitivity. Increasing \(R\) can clarify a comparison but cannot resolve uncertainty about the appropriate nuisance inputs.

Diagnostics include checking that inputs are within their allowed ranges, confirming that the same design and analysis were applied in every scenario, and examining whether results change smoothly when a parameter is varied gradually. Unexpected jumps may reflect a coding or implementation issue, a decision rule that changes discretely, or a genuinely nonlinear response. Do not assume monotonicity: changing an event rate or correlation can have different consequences under different effect measures and analysis procedures. When feasible, compare simulation estimates with a suitable analytic approximation in a simple case.

Sensitivity analysis differs from an analytic power calculation in emphasis: a formula can efficiently show the effect of an input under simple assumptions, while simulation can evaluate the full planned design when its operating characteristics cannot be calculated conveniently. One-at-a-time analysis isolates changes but can obscure interactions; a joint grid examines combinations but requires more scenarios and simulation. Neither approach makes implausible input values informative. The analysis is limited by the model used to generate data and by the plausibility of the selected ranges; conclusions should therefore be stated conditionally on those assumptions.

Common Mistake