Sensitivity to Treatment-Effect Magnitude

Why Vary Treatment-Effect Magnitude?

A response-adaptive design changes assignments as outcomes accumulate. Its allocation and operating characteristics can therefore depend on how large the true treatment effect is. If an analysis reports results only at one assumed effect, it may miss important differences between a barely worthwhile effect and a substantial one. Sensitivity analysis makes those differences visible by repeating the same design evaluation over a set of clinically plausible effects.

The point is not to select the effect that makes the design look best. Instead, the scenarios should represent a defensible range, chosen from clinical judgment, prior evidence, or a prespecified decision threshold. The null case is useful as a calibration check, but nonzero effects are needed to see how adaptation responds when treatment truly helps.

Specify the Model and the Comparison

Consider a hypothetical two-arm trial with a continuous outcome. For this tutorial, assume that patient \(i\)'s outcome under treatment \(a\) follows a normal distribution with mean \(\mu_a\) and variance \(\sigma^2=1\), for \(a\in\{0,1\}\). Outcomes are independent across patients conditional on their assigned treatments, the treatment effect is constant across patients, and outcomes are available before the next assignment. There are no time trends, missing outcomes, or treatment-by-subgroup effects in this illustrative model. These assumptions make the example transparent; an applied analysis should use a model and data-generating scenarios suitable for its endpoint and trial.

Suppose the response-adaptive rule begins with 10 patients assigned to each arm. After that, it assigns the next patient to treatment 1 with probability \(q_t=P(\mu_1>\mu_0\mid\text{data available at time }t)\), and to treatment 0 otherwise. For a concrete Bayesian implementation, take independent priors \(\mu_0,\mu_1\sim N(0,4)\), with known outcome variance 1. After observing \(n_a\) outcomes with sum \(S_a\) on arm \(a\), the posterior variance and mean are

Here \(\Phi\) is the standard normal cumulative distribution function. This rule is specified to make the sensitivity exercise reproducible; the goal is not to compare allocation algorithms. In every scenario, the trial size, prior, initial assignments, update rule, outcome variance, and final testing procedure must remain the same. Only the generating effect \(\delta\) changes.

For a finite trial, useful operating characteristics include the expected fraction assigned to treatment 1, the expected patient benefit relative to assigning everyone to control, and power. If \(N_1\) is the number assigned to treatment 1 among \(N\) patients, then under a constant additive effect the expected treatment benefit per randomized patient is

This is an average outcome difference per patient, in outcome units. It is not a claim that every patient benefits by that amount. Power at a scenario value \(\delta\) is the probability that the prespecified final test rejects \(H_0\) at significance level \(\alpha\), when that value is the true effect. Because assignments depend on observed outcomes, the final test must have its type I error evaluated under the actual adaptive design; an ordinary fixed-allocation reference distribution should not be assumed valid without justification.

Construct a Scenario Grid

A practical grid includes the null, an effect near the smallest clinically important difference, and one or more larger effects that remain credible. For example, if a difference of 0.3 outcome units is considered clinically important, a grid might include \(\delta=0,\ 0.2,\ 0.3,\ 0.5,\ 0.8\). The precise values depend on context. Include values on both sides of the decision threshold when that helps distinguish a design that performs adequately only under optimistic assumptions from one that performs adequately across a broader range.

The analysis should define the scenarios before looking at the simulated results. If the effect is reported in standardized units, state the standard deviation used; if reported in natural units, retain those units in the table and interpretation. A scenario grid is not a probability distribution over effects. It describes separate conditional operating characteristics, unless a probability model over scenarios is explicitly added.

Worked Sensitivity Analysis

Why the Design Can Change Across Scenarios

The effect magnitude influences two linked processes. First, it changes the distribution of outcomes observed on the two arms. Second, those outcomes change the posterior probability used to choose later assignments. With a positive effect, stronger separation in observed outcomes will generally make the rule favor treatment 1 more often. That can increase the proportion of patients receiving the better treatment, and therefore increase expected benefit.

But assigning more patients to one arm also reduces the number available on the other arm. Unequal allocation can make a treatment comparison less precise than a balanced allocation with the same total sample size. Consequently, the direction of change in power is not determined by the direction of change in patient benefit. Power may rise with effect magnitude while remaining below a fixed-allocation benchmark; power can also change irregularly because each simulated allocation path is random. The quantities answer different questions and should be reported side by side rather than collapsed into one score.

Justifying the Benefit Calculation

The expression for average benefit follows directly from the constant-effect model. If patient \(i\) receives treatment 1, the expected outcome improvement relative to control is \(\delta\); if the patient receives control, that improvement is zero. Write \(A_i=1\) for assignment to treatment 1 and \(A_i=0\) otherwise. Then the expected total improvement relative to assigning all \(N\) patients to control is

Dividing by \(N\) gives \(B(\delta)=\delta E(N_1/N)\). This argument needs the stated constant additive effect and a meaningful control comparison. If treatment effects vary across patients, the benefit depends on which patients are assigned to treatment, and the simple product may not apply. It also describes mean outcome benefit under the model, not a complete accounting of harms, costs, or downstream consequences.

Diagnostics, Alternatives, and Limitations

A sensitivity analysis is only informative if the simulation represents the stated design and the comparisons are stable enough to interpret. Check that the null scenario has a rejection rate near the target \(\alpha\), especially when the final test is calibrated by simulation. Report the number of replicates and Monte Carlo standard errors. For a simulated rejection proportion \(\hat{\pi}\), a standard estimate is \(\sqrt{\hat{\pi}(1-\hat{\pi})/R}\). For an average allocation fraction, report its replicate-to-replicate standard deviation divided by \(\sqrt{R}\), or provide an appropriate interval.

Also inspect the distribution of allocation fractions across simulated trials, not only its mean. A mean near 0.7 could arise from most trials assigning roughly 70% to treatment 1, or from a mixture of trials assigning nearly everyone to one arm or the other. Those patterns have different implications for precision and reproducibility. Review the simulated allocation histories and final arm sizes for coding errors, unexpected early lock-in, or large run-to-run variability.

Three approaches are often contrasted:

A grid is usually more transparent than presenting only a favorable single scenario. A fixed-allocation benchmark helps show what is specific to adaptation, but comparisons should hold trial size, endpoint model, significance level, and inferential goal as similar as possible. Averaging results over a distribution of effects is a different analysis: it requires choosing and justifying that distribution, rather than treating a list of scenarios as if they were equally likely.

Do not require every estimated operating characteristic to change monotonically with \(\delta\). In expectation, the specified rule may allocate more often to treatment 1 as evidence of benefit strengthens, but finite simulations are noisy and adaptive paths can be irregular. If a pattern looks implausible, increase the number of replicates, check the update and randomization code, examine the scenario-specific allocation distributions, and verify that only the intended parameter changed. A non-monotone result can be real under a complex rule, but it should not be dismissed as simulation noise without checking.

Finally, the analysis is conditional on its model. A constant effect, known variance, prompt outcomes, and a particular prior are simplifying assumptions here. If plausible variance, response, or outcome-delay assumptions also matter, assess those separately or in a clearly labelled joint sensitivity analysis. A scenario grid cannot guarantee performance for effects outside its range, and expected benefit does not capture all clinical trade-offs.