Sensitivity Analysis for Nuisance Parameters in MAMS Trials

Why Vary Nuisance Parameters?

A MAMS design is usually evaluated under specified treatment effects and nuisance parameters. The treatment effects describe the benefits of interest; nuisance parameters help determine the data’s variability or timing but are not themselves the primary treatment effects. Examples include the outcome variance for a continuous endpoint, the control response rate for a binary endpoint, and the rate at which participants are accrued.

In practice, these nuisance parameters may not be known precisely at the design stage. A design that meets its targets at one assumed value may perform less well at another plausible value. Sensitivity analysis asks how operating characteristics change when nuisance parameters vary while the design and the scientific question are held appropriately fixed. The earlier material on scenario grids provides a way to organize candidate scenarios; the emphasis here is how to choose and interpret nuisance-parameter variations.

Operating characteristics may include familywise error rate, pairwise or disjunctive power, expected sample size, and trial duration. The parameter being varied should be relevant to the characteristic under study. For example, an outcome variance can affect power for a fixed raw treatment difference, while an accrual rate primarily affects calendar duration when the required sample size and follow-up are unchanged.

Set Up an Interpretable Comparison

Before simulating, state the data-generating model, the nuisance-parameter values, and what remains fixed. For instance, a continuous outcome might follow a normal model with common variance \(\sigma^2\) in each arm. A binary outcome might have control response probability \(p_0\) and treatment response probability \(p_1\). Accrual might be modeled at a constant average rate \(r\) participants per month. These are modeling assumptions, not facts established by the simulation.

Keep the MAMS design fixed across scenarios: retain the same allocation, interim analysis rules, stopping boundaries, and maximum sample size unless the question explicitly concerns redesign. Also specify whether treatment effects are fixed on a raw scale, a standardized scale, a risk-difference scale, or another scale. Otherwise, changing a nuisance parameter can unintentionally change the treatment effect being evaluated.

For a binary operating-characteristic outcome, let \(Y_b\) equal 1 if replicate \(b\) meets the event of interest and 0 otherwise. With \(R\) independent replicates at one scenario, the estimated probability is

If two scenarios use independent simulation runs, the estimated difference is \(\hat{\theta}_2-\hat{\theta}_1\), with estimated Monte Carlo standard error

This quantifies simulation noise in the contrast; it does not measure uncertainty about which nuisance-parameter values are scientifically plausible. That uncertainty is represented by the chosen range or scenarios.

Worked Sensitivity Analyses

Justifying the Simulation Contrast

The central interpretation can be stated in terms of a simulated event. At nuisance setting \(\eta\), let \(Y_b(\eta)\) be the indicator that replicate \(b\) meets the operating-characteristic definition. Under the specified simulation model, the target is \(\theta(\eta)=P\{Y_b(\eta)=1\}\). Independent replicates have a sample mean whose expectation is \(\theta(\eta)\), because

Since each \(Y_b(\eta)\) is Bernoulli with variance \(\theta(\eta)\{1-\theta(\eta)\}\), independence gives

Replacing the unknown probability by \(\hat{\theta}(\eta)\) yields the Monte Carlo standard-error estimate used above. The same reasoning does not make a scenario comparison causal or guarantee generality: it estimates differences under the chosen data-generating assumptions. A sensitivity contrast is meaningful only when the simulated scenarios differ in the intended nuisance parameter and keep other design choices aligned.

Diagnostics, Alternatives, and Limitations

Check that each scenario is implemented as specified. For a continuous endpoint, verify the simulated variance against the target and confirm that the treatment effect remains on the intended scale. For a binary endpoint, verify the generated control and treatment probabilities and ensure they lie between 0 and 1. For accrual, compare simulated recruitment totals and durations with the assumed rate and follow-up rule. Unexpectedly identical results across variance settings, for example, may indicate that the variance input is not reaching the outcome generator.

Inspect the operating characteristic alongside the relevant intermediate quantities. A change in power may reflect a changed standardized effect, a changed number of evaluable outcomes at an interim analysis, or a change in follow-up—not necessarily a defect in the decision rule. Also check whether the observed contrast is large relative to its Monte Carlo standard error. More replicates reduce simulation error, but they cannot repair an implausible model or an uninformative range of nuisance values.

A common mistake is to say, “Power is 0.82 at the assumed variance, so variance uncertainty is unimportant.” That single value says nothing about sensitivity. The correction is to simulate a defensible range of variances, keep the treatment effect scale explicit, and report the change in power with its Monte Carlo precision. Another mistake is to interpret a small change as proof of robustness when the simulation runs are too few to resolve it. In that case, increase replication or report that the contrast remains imprecise.

Sensitivity conclusions are limited by the realism of the assumed ranges and data-generating models. A finite set of scenarios does not cover every possible value, and an average over scenarios is not a worst-case guarantee. Nor does sensitivity analysis by itself select a design: investigators must judge which operating characteristics matter and what degree of variation is acceptable.