Sensitivity Analysis Across True Treatment Effects

Why Vary the True Treatment Effect?

A sample-size re-estimation rule is designed using planning assumptions, but its operating characteristics depend on the effect that actually holds. A rule that often increases enrollment when the treatment effect is modest may rarely increase it when the effect is large. Conversely, power may remain low if the true effect is smaller than the planning value, even when the rule frequently uses its maximum sample size.

A sensitivity analysis across true treatment effects holds the design, decision rule, and analysis method fixed, then recalculates key operating characteristics over a prespecified grid of possible effects. The resulting curves answer related but distinct questions: how often does the rule increase the sample size, how many participants does it use on average, and what is the probability of rejecting \(H_0\)? Those quantities should be reported separately.

The calculations below use a hypothetical normal-endpoint design to make the dependence on the true effect explicit. They are an illustration of sensitivity analysis, not a recommendation to use an unadjusted final test after data-dependent sample-size adaptation.

Design and Assumptions for the Illustration

Let \(\delta\) denote the true standardized treatment effect, with positive values favoring the treatment. The total sample size at the interim analysis is 100; the planned final total is 200, and the permitted increase raises the final total to 300. Allocation is equal between groups, observations are independent, and the endpoint is normally distributed with known common variance. Information is proportional to total sample size. These assumptions give the standardized treatment-effect statistic at total sample size \(N\) a normal distribution with mean \(\delta\sqrt{N/4}\) and variance 1.

At the interim, let \(Z_I\) be the standardized statistic. Under a true effect \(\delta\), \(Z_I\sim N(5\delta,1)\). The fixed rule increases the total sample size from 200 to 300 if \(0<Z_I<1.5\); otherwise, it completes the trial at 200. There is no early stopping. The final rejection criterion used for this numerical illustration is a one-sided statistic exceeding 1.96.

The known-variance, equal-allocation model is deliberately simple. In an actual trial, the chosen endpoint model, variance assumptions, rounding of sample sizes, and confirmatory analysis procedure must match the design being evaluated. In particular, using a fixed 1.96 final cutoff after selecting sample size based on an unblinded interim statistic does not by itself establish type I error control.

Increase Probability and Expected Enrollment

Because \(Z_I\sim N(5\delta,1)\), the probability of entering the increase region is the difference between two normal cumulative probabilities:

The first term is the probability that \(Z_I\) is below 1.5; subtracting the probability that it is at or below 0 leaves the probability between the cutoffs. Expected sample size follows by weighting the two possible totals by their probabilities:

This expectation describes average enrollment over repeated trials at the specified true effect. It is not the sample size of a particular trial, which is either 200 or 300 under this rule.

Power Across the Increase and No-Increase Regions

Power must account for the interim statistic and for the final statistic’s dependence on it. The relevant normal model has independent increments: information accumulated after the interim is independent of the interim data, while the final statistic includes both interim and subsequent information. Consequently, it is not correct to calculate fixed-design power and then add the probability of an increase.

For total \(N=200\), the interim contains half of the planned final information. Conditional on \(Z_I=z\), the final standardized statistic has mean \(\sqrt{0.5}\,z+\sqrt{0.5}(5\delta)\) and variance 0.5. Therefore its conditional rejection probability is

For total \(N=300\), the interim contains one third of the final information. Conditional on \(Z_I=z\), the final statistic has mean \(z/\sqrt{3}+\delta\sqrt{100/3}\) and variance \(2/3\). Its conditional rejection probability is

Writing the interim density as \(\phi(z-5\delta)\), the power is the sum of the rejection probabilities integrated over the two disjoint decision regions:

The first integral is the contribution to power from trials that do not increase sample size; the second is the contribution from trials that do. The power curve can be evaluated at each grid value by numerical integration. The fixed-design power, which provides a useful reference but is not the adaptive design’s power, is \(\Phi(\delta\sqrt{50}-1.96)\).

Reading a Sensitivity Grid

The following grid reports the probability of increase, expected total sample size, and fixed-design reference power. The adaptive power at any row is obtained from the two integrals above; the value at \(\delta=0.20\) is shown separately because it illustrates the full regional calculation.

The increase probability is not generally increasing with the true effect. Here it is largest near \(\delta=0.15\), because the mean interim statistic is then centered in the increase interval from 0 to 1.5. As the true effect moves well above that interval, interim values increasingly exceed its upper cutoff, so the rule often returns to the planned sample size. Meanwhile, fixed-design power increases across the displayed positive effects because the final statistic’s mean moves farther above the rejection threshold.

A useful sensitivity report therefore does not summarize the design with a single “expected sample size.” It shows how the burden and power change over effects that are scientifically plausible, and identifies the effect range used for planning. The same design can have lower expected enrollment at a large effect precisely because it is less likely to trigger an increase, while its power is higher.

Assumptions, Diagnostics, and Limitations

A sensitivity grid is only as credible as the model and the rule used to produce it. Check that the interim and final information fractions match the planned sample sizes, that the independent-increments representation is appropriate, and that the effect scale and variance used in the calculation are consistent throughout. If the variance is estimated, outcomes are binary, allocation is unequal, sample sizes are rounded, or there are multiple interim looks, the operating-characteristic calculations must reflect those features rather than reuse this known-variance illustration unchanged.

Also inspect the rule itself. The increase probability should lie between 0 and 1, expected enrollment should lie between the minimum and maximum totals, and the two power integrals should cover disjoint regions whose union includes every possible interim value. An increase probability that rises steadily across all effects would be a warning in this particular interval rule: once the interim mean moves above 1.5, the probability of being inside the promising zone falls.

The comparison with fixed-design power is descriptive, not proof that the adaptive design controls the type I error rate. Because the sample size is chosen using an unblinded interim statistic, the final statistic and sample-size decision are dependent. A valid confirmatory analysis may require an adaptive combination test, a conditional-error approach, or another prespecified method that preserves the required error rate. Its rejection event must replace the illustrative 1.96 cutoff when calculating the design’s actual power.

A further limitation is that a grid cannot represent uncertainty about which effect is true. It describes conditional operating characteristics at selected values; it does not assign probabilities to those values. Include effects below, near, and above the planning effect, use a sufficiently fine grid around decision-region transitions, and report the range rather than selecting only favorable points.

Common Mistake