Stress-Testing Adaptation Triggers Near Thresholds

Why Probe the Neighborhood of a Trigger?

An adaptive trial may change its sample size, allocation, or another design feature when interim information crosses a prespecified threshold. Such a rule can be implemented exactly as written and still behave awkwardly near its cutoff: small changes in the interim data, or in the underlying scenario, may lead to different actions. A simulation at only a few widely spaced scenarios can miss this behavior.

A near-threshold stress test deliberately examines scenarios just below and just above the cutoff. Its purpose is not to revise the rule after seeing attractive results. It is to find out how often the rule changes action in this neighborhood and whether those changes materially affect operating characteristics such as expected sample size or probability of success.

The earlier material on scenario grids and simulation precision provides the tools for choosing inputs and assessing Monte Carlo error. Here the key design choice is to make the grid locally dense around the trigger, while holding other assumptions fixed. This isolates sensitivity to the trigger rather than mixing it with changes in the data-generating model.

Formalizing the Trigger and the Stress Test

The notation \(\theta\) stands for the scenario input being varied, such as a true response rate or treatment effect. A stress test should identify exactly what is held fixed as \(\theta\) changes. It should also specify whether equality triggers the action: \(T \geq c\) and \(T > c\) can produce different results, especially when \(T\) is discrete.

For each scenario, simulate \(M\) independent trial replicates and record whether the rule triggers. If \(I_m=1\) when replicate \(m\) triggers and \(I_m=0\) otherwise, estimate the trigger probability by

Under independent replicates generated from the specified scenario, the Monte Carlo standard error of this estimated probability is estimated by

This standard error describes simulation noise in estimating the trigger probability; it does not describe uncertainty about whether the trial’s assumptions or the chosen threshold are appropriate. A change in \(\hat{q}\) between nearby scenarios should be judged against its Monte Carlo error, but a small Monte Carlo error does not make a sensitive rule less sensitive.

A Simple Sensitivity Result

A simple continuous example explains why trigger probabilities can change quickly near a threshold. Suppose the interim statistic follows \(T\sim N(\theta,1)\), where \(\theta\) is its scenario-specific mean and the variance is one. The rule triggers when \(T\geq c\). Standardizing gives

where \(\Phi\) is the standard normal cumulative distribution function. Differentiating with respect to \(\theta\) yields

where \(\phi\) is the standard normal density. This derivative is largest at \(\theta=c\), with value \(1/\sqrt{2\pi}\). Thus, in this model, a small change in the scenario mean changes the trigger probability most rapidly at the boundary. The rule itself is binary for any single trial; the probability of applying it varies smoothly across scenarios.

This calculation is a justification for concentrating simulation points near the cutoff, not a universal formula for every trial. With discrete interim data, the realized trigger decision jumps as the observed statistic changes, but the trigger probability is generally continuous in a continuous scenario parameter under regular models, such as a binomial model with fixed \(n\). A finite-simulation estimate may appear stepwise; a genuine discontinuity requires something beyond the discrete statistic alone, such as a discontinuous scenario model. In either case, simulations should follow the actual rule, including its rounding, equality convention, and any minimum or maximum limits.

Translate Trigger Sensitivity Into Trial Performance

A trigger rate matters because the rule selects between actions. If the trial uses sample size \(N_0\) without adaptation and \(N_1\) after triggering, then its expected sample size under scenario \(\theta\) is

This identity follows by averaging the two possible sample sizes, weighted by their probabilities. More generally, if an outcome has conditional mean \(g_0(\theta)\) without adaptation and \(g_1(\theta)\) after adaptation, its overall mean is

In a real adaptive trial, the performance calculation should come from the full simulated trial process; conditional outcome probabilities may depend on the same interim data that determine the action. The weighted expression is exact when \(g_0\) and \(g_1\) are the corresponding conditional expectations given each action. It is not a license to combine unrelated marginal summaries.

Measure Decision Instability and Check the Rule

One way to express the instability of a binary trigger is to imagine two independent interim realizations under the same scenario and ask how often they lead to different actions. If each triggers with probability \(q\), then

The expression is largest at \(q=0.5\), where it equals 0.50. This is a useful description of repeated-trial decision variability, not a recommendation to change the rule. In the normal example, at \(\theta=c\), \(q=0.5\); at \(\theta=c\pm0.10\), \(q\) is about 0.4602 or 0.5398, giving \(2q(1-q)\approx0.4968\). Both are close to the maximum.

For a near-threshold assessment, plot or tabulate at least the scenario value, estimated trigger probability, its Monte Carlo standard error, and the relevant performance measures. A locally dense grid can reveal curvature that a coarse grid hides; finite-simulation estimates may appear stepwise, while a genuine discontinuity requires something beyond the discrete statistic alone, such as a discontinuous scenario model. Check that:

• the simulated rule applies the intended cutoff and the intended equality convention;
• the trigger frequency is plausible under known or analytically tractable cases;
• the pattern is stable when simulation size is increased, within Monte Carlo error;
• performance changes are computed from the complete trial outcomes, not inferred solely from trigger rates.

For example, if the interim count follows a binomial model with fixed sample size and the trigger is “at least 12 responses,” the realized decision jumps as the observed count crosses the cutoff, but the trigger probability is continuous in the underlying response probability. Its rate of change need not be uniform, and a finite-simulation estimate may appear stepwise. A smooth-looking curve created by interpolation does not change the discrete behavior of the implemented rule.

How This Stress Test Differs From Alternatives

Analytic calculations are valuable checks when available, but simulation is generally needed to evaluate the consequences of the decision in a full adaptive design. Conversely, simulation does not excuse a failure to verify the rule’s implementation. A targeted simulation and an analytic check answer related but distinct questions.

Common Pitfalls and Limitations

Common mistake: “The estimated trigger rates are 0.46 and 0.54, so the threshold rule is unreliable and should be changed.” This reasoning confuses a frequent change of action across hypothetical repeated trials with an implementation error or an unacceptable design. A rule can intentionally make a binary choice using noisy interim data. The correction is to report the trigger probabilities and quantify what the two actions imply for the design’s operating characteristics, then judge whether those consequences meet prespecified requirements.

Another mistake is treating a difference smaller than the Monte Carlo error as evidence that the scenarios behave identically. It means the simulation has not estimated the difference precisely enough to distinguish it from simulation noise at that run size; it does not prove equality. Increase simulation effort if the comparison matters, and report uncertainty alongside estimates.

A local stress test has limits. It does not establish performance over the entire scenario space, validate the underlying model, or account for operational complications that were held fixed. Nor should the threshold be selected solely because one locally simulated version looks favorable. Keep the rule and its decision criteria specified independently of the results used to evaluate them, and document the exact threshold, comparison convention, and scenario inputs.