Stress-Test Response-Model Misspecification

Why Stress-Test the Allocation Model?

A response-adaptive design uses accumulating response information to change the probability of assigning later participants to each treatment. This can direct more participants to a treatment that appears beneficial, but it also makes the design depend on the model or prediction rule that turns response information into allocation probabilities. If that working response model is wrong, the trial may allocate more participants to a treatment that is not beneficial, miss important differences between patient subgroups, or produce inference with poor operating characteristics.

A stress test asks what happens when the response model driving allocation differs from the mechanism that generates outcomes. The aim is not to declare one model universally correct. It is to identify which conclusions about participant outcomes, allocation, and inference are sensitive to plausible departures from the model. The simulation principles established in the earlier material on building data-generating mechanisms and matching trial conditions apply: specify the true mechanism independently of the allocation rule, and change the feature being stress-tested while holding other conditions fixed.

Specify the Two Models Separately

For participant \(i\), let \(A_i\in\{0,1\}\) denote the assigned treatment, \(X_i\) the participant’s covariates, and \(Y_i\) the observed outcome. For a binary outcome, write \(\mu_a(X_i)=P(Y_i=1\mid A_i=a,X_i)\) for the true response probability under treatment \(a\). The working model supplies predictions \(m_a(X_i)\), which the allocation algorithm may use to choose the probability \(q_i=P(A_i=1\mid\mathcal H_{i-1},X_i)\), where \(\mathcal H_{i-1}\) denotes the trial history available before participant \(i\) is randomized.

In a simulation, the true outcome mechanism generates \(Y_i\), while the working model affects \(q_i\). These roles should not be conflated. A useful stress test specifies both explicitly, including the assumed covariate distribution, outcome distribution, treatment-effect pattern, and allocation rule. If the algorithm refits the working model as outcomes accrue, specify that updating procedure too: an initially wrong prediction may improve with data, remain biased, or become more extreme.

Misspecification can take several forms. A model might omit a treatment-by-subgroup interaction, assume a constant effect when effects vary with a covariate, use an incorrect link or functional form, or ignore a time trend. Choose departures that are plausible for the application and consequential for the rule’s decisions. A model can be wrong in a way that barely affects assignment, or only slightly wrong in prediction while repeatedly pushing assignments in an unhelpful direction.

Misspecification Can Hide in Subgroups

A working model may correctly predict an average response yet miss the treatment-effect pattern that matters for individual assignment. For example, a model with a common treatment effect may ignore that treatment 1 benefits one subgroup but harms another. The overall marginal effect alone does not reveal whether the model’s assignments are sensible for each subgroup.

Keep Allocation Error Separate from Inference

A wrong working response model does not automatically imply that every treatment-effect estimator is biased. The consequences depend on the allocation probabilities and the analysis procedure. For instance, under sequential randomization with known assignment probabilities, positivity, and well-defined potential outcomes, an inverse-probability estimator for the finite cohort average treatment effect is unbiased over the randomization distribution. Let \(Y_i(1)\) and \(Y_i(0)\) be the potential outcomes, and let \(q_i\) be the conditional probability of assignment to treatment 1 before \(A_i\) is drawn. The estimator is

Each observed outcome is weighted by the inverse of its treatment-assignment probability, together with the indicator for its assigned arm. This targets the average difference in potential outcomes for the \(N\) participants; it does not weight by the assignment probability itself. For a marginal effect in a broader target population, the estimator must also standardize to that target population.

Averaging this equality over participants gives \(E(\widehat{\tau}_{HT})=N^{-1}\sum_i\{Y_i(1)-Y_i(0)\}\). This argument requires the recorded \(q_i\) values to be the actual conditional assignment probabilities and both probabilities to be positive. It establishes unbiasedness, not good precision: very small assignment probabilities can give extreme weights. Nor does it justify an analysis that substitutes incorrect probabilities or relies on an outcome model that fails under the true mechanism.

Design the Stress Test and Read Its Results

For a fair comparison, use the same true outcome mechanism, enrollment process, sample size, outcome definition, analysis procedure, and stopping rule across designs. Compare the adaptive design using its working response model with an appropriate alternative, such as equal randomization. If both designs use adaptive allocation but differ only in their response model, define that contrast explicitly. When feasible, use paired replicates: generate shared underlying participant characteristics and potential outcomes for each replicate, then apply each design to that common replicate. This makes the comparison focus on the design difference rather than unrelated simulation randomness.

Useful diagnostics include checking the working model’s predictions against the true response probabilities by treatment and subgroup; plotting or tabulating allocation probabilities over time; and examining whether probabilities repeatedly approach the rule’s permitted limits. In simulated data, assess whether the analysis has the intended error rate or confidence-interval coverage under each true scenario. A model that looks adequate in aggregate can be systematically wrong in an important subgroup. Conversely, prediction error alone is not proof of consequential harm if it rarely changes the allocation decisions.

Consider this hypothetical paired simulation with 2,000 replicate pairs and a true marginal effect of 0.10. The correctly specified and misspecified designs have mean treatment-1 allocation fractions of 0.58 and 0.76, and mean participant success proportions of 0.612 and 0.584. Suppose nominal 95% intervals cover the true effect in 1,888 replicates under the correctly specified design and 1,817 under misspecification. The estimated coverage rates are \(1888/2000=0.944\) and \(1817/2000=0.9085\), a difference of 0.0355. If 80 pairs cover only under the correctly specified design and 9 only under misspecification, the paired Monte Carlo standard error of the coverage difference is

The 0.0355 difference is about 7.63 such Monte Carlo standard errors, indicating that simulation noise alone is an implausible explanation for the difference in this hypothetical run. It does not establish that the same loss will occur in every setting. Here the success proportion is a participant-outcome metric, the allocation fraction measures imbalance, and coverage assesses inference; none should be substituted for the others.

Alternatives, Failure Modes, and Limits

A nonadaptive design, often equal randomization, is a useful benchmark because its assignment does not depend on a response model. It may avoid harms caused by a wrong response ranking, although it does not guarantee the best participant outcomes or the most precise inference in every setting. A capped or probability-floor allocation rule can limit extreme assignments and preserve opportunities to observe outcomes in both arms. This reduces some risks but does not correct a wrongly ranked treatment. Robustness can also be evaluated across several plausible true mechanisms rather than relying on a single misspecification.

Watch for these signs that a stress test or its interpretation is going wrong:

• The truth and working model are accidentally identical. Verify the simulated response probabilities directly, especially across subgroups and over time.
• The analysis uses the wrong assignment probabilities. Store the probability actually used at each randomization; predictions from the response model are not a substitute for \(q_i\).
• Only the overall treatment effect is checked. Summaries can conceal subgroup harms or poor subgroup-specific inference.
• Extreme probabilities are ignored. When a probability approaches zero or one, inverse-probability weights can become large and make estimates unstable.
• A difference is reported without simulation uncertainty. A finite number of replicates gives noisy estimates of operating characteristics; report Monte Carlo uncertainty as well as the point estimate.

The main limitation is that a stress test only assesses the mechanisms and departures that were specified. It cannot show that an unexamined model failure is harmless, and a deliberately severe departure may not describe the eventual trial. Interpret results conditionally: under this true mechanism, this allocation rule and this analysis produced these operating characteristics. A useful conclusion is which design features are sensitive to the departure, not that one approach is universally superior.