Failure Modes in Sequential Redesign

Why Sequential Redesign Can Become Unstable

Sequential redesign uses interim data to update a design decision. That flexibility can improve a study when the interim data are informative and useful options remain available. But a decision recalculated several times can also react to random fluctuations, overvalue the apparent winner, or become constrained by choices already made. A design that looks optimal at one interim analysis is not necessarily a reliable choice for the remaining study.

Here, instability means that small changes in interim data, the assumed parameter range, or the set of feasible options can cause large changes in the selected design or its expected performance. The focus is on diagnosing that instability, not on a single redesign algorithm. The earlier material on updating parameter uncertainty, conditional optimization, and diagnostic plots supplies the basic setting: at an interim analysis, data inform uncertain model parameters, and a criterion is used to compare feasible designs.

The examples use information as a design criterion, but the same reasoning applies to other prespecified criteria. Any numerical information score below is illustrative, not a universal measure of clinical value. Before applying a rule, specify the model, the criterion, the interim data used, and the feasible choice set. In particular, a local information calculation depends on the model and parameter values at which it is evaluated; it does not by itself guarantee good estimation over a wider parameter range.

Noise Can Reverse the Ranking of Designs

An interim estimate is uncertain. If two candidate designs have similar true scores, even moderate estimation error can change which one appears better. A single observed reversal is not proof that a redesign rule is defective; the concern is a high probability of reversals, especially when the consequences of choosing the lower-performing design are substantial.

A practical diagnostic is to examine how often the chosen design changes under plausible perturbations of the interim estimate, such as draws from its estimated sampling distribution or bootstrap resamples. Also inspect the score gap between the top candidates. A very small gap coupled with frequent switches indicates a fragile ranking, even if the reported best score itself seems favorable.

Repeated Optimization and the Winner's Curse

Comparing many candidates creates another problem: the highest estimated score tends to be optimistic because it is selected for being high. Repeating the comparison at multiple interim analyses provides further opportunities to respond to random high estimates. This is a selection effect, not necessarily a bias in each candidate's score estimator considered separately.

At successive interim analyses, the same principle applies: each re-optimization can seize on a new favorable fluctuation. A useful diagnostic is to simulate the entire decision process, including the interim-data updates, candidate comparisons, and any stopping or redesign triggers. Reporting only the average score of the design chosen at one isolated interim analysis can conceal selection optimism and unstable trajectories.

Parameter Support Must Cover Plausible Cases

A redesign can be well optimized for the parameter values it considers and still perform poorly when those values fail to represent the truth. “Support” here means the range or distribution of parameter values used to evaluate candidate designs. A narrow support can exclude scientifically plausible cases, particularly when early data are weak or the model is only locally informative.

Consider a one-compartment concentration mean \(C(t)=e^{-kt}\), with time \(t\) in hours and elimination rate \(k\) in inverse hours. Assume one observation has independent additive normal error with variance 1 concentration-unit squared, and normalize the concentration scale to 1. The expected Fisher information in that observation about \(k\) is the squared derivative of the mean with respect to \(k\), divided by the error variance:

This formula is conditional on the stated mean model and error assumptions. For a fixed positive \(k\), the continuous-time optimum occurs at \(t=1/k\): differentiating \(\log I(t;k)=2\log t-2kt\) gives \(2/t-2k=0\). This local optimum shifts earlier as elimination becomes faster.

Diagnostics should include the support boundaries, the scientific basis for them, and sensitivity to plausible values beyond them. Check whether the interim data can actually distinguish values near those boundaries. A concentrated posterior or estimated parameter range is not automatically reassuring if it is concentrated because the model, prior, or computational search excluded alternatives.

Shrinking Options Can Create a Hidden Performance Cost

Sequential decisions may consume or rule out options: a sampling time may no longer be feasible, a remaining visit may have a fixed slot, or a budget may prevent restoring an earlier choice. Redesign must therefore account for the feasible set at the time of decision. A criterion optimized over the current options cannot recover an option that has already disappeared.

Track the feasible set explicitly at each stage and compare the selected score with a relevant benchmark, such as the best option before constraints accumulated. That comparison is diagnostic, not a claim that the original option could still have been used. Where feasible, retaining a reserve option or delaying irreversible allocation can protect against premature loss of useful choices.

Safeguards, Assumptions, and Limits

No safeguard eliminates all instability. The appropriate response depends on what is uncertain and what can still be changed. The following checks address different failure mechanisms:

• For noisy rankings: show uncertainty around the score difference and test whether modest perturbations change the selected option. When candidates are nearly tied, the apparent winner may not be meaningfully better.
• For repeated selection: evaluate the whole sequence of interim analyses in simulation. Include the actual number and timing of looks, the choice rule, and the dependence created by reusing data.
• For weak parameter support: assess performance over scientifically plausible parameter values, including boundary and stress-test cases. A robust criterion can reduce sensitivity to uncertain parameters, but only over the range it actually covers.
• For disappearing options: record how feasibility changes and quantify the score loss from constraints already incurred. Retain flexibility where the study's operational and ethical requirements permit.

These checks rely on assumptions that need examination. The normal approximation in the first example presumes an adequately characterized score-difference error distribution. The independent-score calculation in the second example is valid only under independence. The information calculations assume the stated concentration model and error structure. In an application, model misspecification, correlated observations, assay constraints, and finite-sample behavior may alter the rankings. Use a simulation model that represents these features as far as possible, and report sensitivity when they are uncertain.

Alternatives have different tradeoffs. A fixed design avoids data-driven switching but cannot react to interim information. A robust design can average performance over a parameter distribution or guard against poor performance across a specified range, but its protection is only as credible as that distribution or range. A plug-in redesign is simple but may treat a noisy interim estimate as if it were known. Stabilizing rules—such as requiring a meaningful score improvement before switching, limiting the number of redesigns, or retaining a fallback—can reduce reactions to noise, but may also delay a genuinely beneficial change. These are not interchangeable guarantees; compare them under the same plausible data-generating scenarios and operational constraints.