Stress Testing Joint Design Assumptions

Why Stress Testing Is Different From Calibration

Calibration chooses or tunes an algorithm to meet prespecified goals under selected assumptions. Stress testing asks whether its performance remains acceptable when those assumptions are challenged. A design can be well calibrated for one dose-response curve and still be fragile if efficacy plateaus, efficacy and toxicity are dependent, outcomes arrive late, or the enrolled population differs from the one used in planning.

The design algorithm should be held fixed during this assessment: its eligibility rules, allocation decisions, stopping criteria, and final dose-selection rule are not changed from scenario to scenario. What changes is the data-generating situation used to create hypothetical trials. This separation makes the question interpretable: how would this specified design perform if the clinical truth were different from the assumptions that guided its construction?

Definition: A stress-test scenario is a fully specified hypothetical data-generating condition used to evaluate a fixed trial design. It can specify, for each dose and relevant subgroup, the outcome probabilities, dependence between outcomes, outcome-delay distributions, and enrollment composition. A scenario is a challenge to an assumption, not a claim that the scenario is the true clinical model.

For joint dose finding, let $E=1$ indicate the defined efficacy outcome and $T=1$ indicate toxicity for a patient. A scenario needs to specify the relevant probabilities for the pair $(E,T)$, not merely separate efficacy and toxicity rates, whenever the design uses their joint distribution. The clinical estimand and the endpoint observation windows must remain the same across scenarios unless the purpose of the stress test is explicitly to vary them.

A useful plan starts with a reference scenario and then varies one important feature at a time. That reveals which assumption drives a performance change. Afterward, combined scenarios can test whether several departures together create a more severe challenge. The tested scenarios should be clinically interpretable and plausible enough to inform design decisions, rather than chosen solely to make the design look good or fail.

Challenge the Shape of the Dose-Response Curves

A dose-response assumption concerns how efficacy and toxicity probabilities change across ordered dose levels. For instance, a model may assume increasing efficacy and increasing toxicity. A stress test can retain that general pattern but compare a gradual efficacy increase with a plateau, or include a less regular pattern when clinical knowledge makes it plausible. The point is not to decide which curve is correct from simulation; it is to learn whether the design’s recommendation is sensitive to the curve it was built around.

For a transparent comparison, suppose the trial’s final rule selects an eligible dose with the largest probability of efficacy without toxicity, $P(E=1,T=0)$. Eligibility requires the toxicity probability to be at most 0.25. These are hypothetical rules for the example, not universal dose-selection criteria.

Worked Example: A Plateau Challenges a Monotone Efficacy Assumption

Consider three doses and assume, for this example, that efficacy and toxicity are independent within each dose. At dose 1, the probabilities are $P(E=1)=0.35$ and $P(T=1)=0.05$. At dose 2, they are 0.60 and 0.15. Compare two possible dose-3 scenarios: a continuing increase, with probabilities 0.75 and 0.30, and a plateau, with probabilities 0.62 and 0.30.

Under independence, $P(E=1,T=0)=P(E=1)\{1-P(T=1)\}$. At dose 1 this is $0.35(1-0.05)=0.3325$. At dose 2 it is $0.60(1-0.15)=0.5100$. Dose 3 has toxicity probability 0.30, so it is ineligible under the stated rule in either scenario. Dose 2 is therefore the best eligible dose under both truths, although its advantage over dose 1 is smaller in the plateau scenario: $0.5100-0.3325=0.1775$ in either case here, while the plateau reduces the apparent incremental efficacy from dose 2 to dose 3 from 0.15 to 0.02.

Now suppose 500 hypothetical trials are simulated under each scenario using the same fixed adaptive algorithm. In the continuing-increase scenario, it selects dose 2 in 330 trials, dose 1 in 100, dose 3 in 50, and no dose in 20. The estimated probability of selecting the best eligible dose is $330/500=0.660$. Under the plateau scenario, suppose the counts are 280, 125, 65, and 30, respectively. The corresponding estimate is $280/500=0.560$. These hypothetical results show how a plausible change in curve shape can reduce correct selection even when the target dose under the stated rule remains dose 2.

The arithmetic in this example uses a simplifying independence assumption. In an actual stress test, if dependence is part of the concern, the joint probability should be specified directly rather than calculated by multiplying marginal probabilities. A useful comparison reports more than the most frequent selected dose: it also includes the probability of selecting an unsafe dose, selecting no dose, and the distribution of allocations across doses.

Vary Dependence Between Efficacy and Toxicity

Separate marginal rates do not determine the joint distribution. With binary outcomes, if $P(E=1)=p_E$, $P(T=1)=p_T$, and $q=P(E=1,T=1)$, then the four cell probabilities are $q$, $p_E-q$, $p_T-q$, and $1-p_E-p_T+q$. They must all be nonnegative. Equivalently, $q$ must lie between $\max(0,p_E+p_T-1)$ and $\min(p_E,p_T)$.

Thus, holding the two marginal rates fixed while changing $q$ is a direct test of dependence assumptions. Positive dependence means that joint occurrence of efficacy and toxicity is more frequent than under independence; negative dependence means it is less frequent. The effect on a design depends on which joint outcomes its rules use.

Worked Example: Same Margins, Different Joint Outcomes

Suppose at one dose $P(E=1)=0.60$ and $P(T=1)=0.20$. Under independence, $q=0.60(0.20)=0.12$, giving the cell probabilities $P(E=1,T=1)=0.12$, $P(E=1,T=0)=0.48$, $P(E=0,T=1)=0.08$, and $P(E=0,T=0)=0.32$. These sum to 1.

Consider instead a positively dependent alternative with $q=0.18$. This is valid because $0\leq0.18\leq0.20$. Its cells are 0.18, $0.60-0.18=0.42$, $0.20-0.18=0.02$, and $1-0.60-0.20+0.18=0.38$, again summing to 1. Both scenarios have the same efficacy rate and toxicity rate, but the probability of efficacy without toxicity falls from 0.48 to 0.42.

For 10 independent patients at this dose, if the target event is that a patient has both outcomes, the probability of at least one such event is $1-(1-q)^{10}$. Under independence of the two outcomes within a patient, this is $1-0.88^{10}\approx0.7215$. Under the alternative joint probability $q=0.18$, it is $1-0.82^{10}\approx0.8626$. The calculation assumes patients’ outcome pairs are independent of one another; it does not assume that efficacy and toxicity are independent within a patient in the alternative scenario.

When building a dependence stress test, use joint probabilities that are mathematically compatible with the margins and clinically defensible. Compare the design’s actual decision-relevant quantities, such as the probability of a favorable joint outcome or the chance of crossing a joint safety boundary. A comparison of marginal rates alone cannot reveal whether the algorithm is sensitive to dependence.

Delayed Outcomes and Subgroup Patterns

A delayed outcome changes what the algorithm can know at each decision time, even if the eventual outcome probabilities are unchanged. Stress scenarios should therefore specify when efficacy and toxicity become observable, whether their delays differ, and how much accrual occurs while outcomes are pending. The objective is to measure consequences for interim decisions and time to recommendation, not to treat an unobserved outcome as if it were a negative outcome.

Worked Example: Incomplete Outcome Availability at an Interim

In a hypothetical trial, 12 patients have been enrolled at the interim review. Efficacy is observed for 8 patients, with 5 observed responses; toxicity is observed for 9, with 2 observed toxicities. Only 6 patients have both outcomes observed, and 3 of those have efficacy without toxicity. The observed summaries are $5/8=0.625$ for efficacy, $2/9\approx0.222$ for toxicity, and $3/6=0.500$ for the favorable joint outcome among patients with both outcomes available.

These three fractions use different denominators, so they are not a complete joint outcome table for all 12 patients. For example, dividing the 3 favorable joint outcomes by 12 would give 0.250, but that treats the six patients with incomplete joint follow-up as if they did not have the favorable outcome. That is not a valid estimate of the eventual joint probability unless an explicit rule justifies that treatment.

A delay stress test would generate the eventual outcomes for all 12 patients, then apply the stipulated observation times and interim algorithm. Across simulated trials, it could compare the rate of correct dose selection, safety-rule triggers, and time to final recommendation with a scenario in which outcomes arrive promptly. The comparison isolates the impact of outcome availability while preserving the same eventual outcome probabilities.

Subgroup stress tests similarly ask whether a design’s overall recommendation changes when benefit-risk patterns or the composition of enrolled patients differ. Specify the subgroup-specific outcome distributions and the target population mix. If the target population itself is defined differently, that is an estimand change and should not be disguised as a mere modeling challenge.

Worked Example: A Changing Subgroup Mix Changes the Preferred Dose

Suppose there are two subgroups, A and B. Assume independence of efficacy and toxicity within each subgroup for this example. At dose 1, subgroup A has efficacy and toxicity probabilities 0.60 and 0.05, while subgroup B has 0.25 and 0.20. At dose 2, the probabilities are 0.80 and 0.10 for A, and 0.30 and 0.40 for B. Use the same rule as before: toxicity must be at most 0.25 in the target population, and among eligible doses choose the one with the largest population-average probability of efficacy without toxicity.

At dose 1, the subgroup-specific favorable probabilities are $0.60(0.95)=0.570$ for A and $0.25(0.80)=0.200$ for B. At dose 2 they are $0.80(0.90)=0.720$ and $0.30(0.60)=0.180$, respectively. If the target mix is 50% A and 50% B, dose 1 has favorable probability $0.5(0.570)+0.5(0.200)=0.385$ and toxicity probability $0.5(0.05)+0.5(0.20)=0.125$. Dose 2 has favorable probability $0.5(0.720)+0.5(0.180)=0.450$ and toxicity probability $0.5(0.10)+0.5(0.40)=0.250$. Both are eligible, and dose 2 is preferred.

Now change the target mix to 20% A and 80% B. Dose 1 has favorable probability $0.2(0.570)+0.8(0.200)=0.274$ and toxicity probability $0.2(0.05)+0.8(0.20)=0.170$. Dose 2 has favorable probability $0.2(0.720)+0.8(0.180)=0.288$, but toxicity probability $0.2(0.10)+0.8(0.40)=0.340$, exceeding the 0.25 limit. Dose 1 is therefore preferred under the changed mix. This is a stress test of population composition and subgroup pattern, not evidence that either mix is correct.

Summarizing Performance and Its Simulation Error

For each scenario, report performance measures that match the design’s purpose: correct or acceptable dose selection, unsafe selection, stopping without a recommendation, allocation to overly toxic doses, and decision timing. Compare the reference scenario with the challenged scenarios in a table or plot, while retaining the scenario definitions so that each difference has an interpretable cause. One-factor-at-a-time tests are easy to interpret; combined challenges are useful for assessing whether several departures compound one another. Neither replaces the other.

Simulation estimates themselves vary by chance. If $R$ independent hypothetical trials are run and $I_r=1$ when replicate $r$ meets a stated criterion and $I_r=0$ otherwise, then $\hat p=R^{-1}\sum_{r=1}^{R}I_r$ estimates the probability $p$ of meeting that criterion under the specified scenario. Each indicator has variance $p(1-p)$, so the sample mean has variance $p(1-p)/R$ and standard error $\sqrt{p(1-p)/R}$. Since $p$ is unknown, the usual plug-in estimate is $\sqrt{\hat p(1-\hat p)/R}$.

Justification: The indicators are independent across simulated replicates and each has variance $p(1-p)$. Therefore, $$ \operatorname{Var}(\hat p)=\operatorname{Var}\left(\frac{1}{R}\sum_{r=1}^{R}I_r\right) =\frac{1}{R^2}\sum_{r=1}^{R}\operatorname{Var}(I_r) =\frac{p(1-p)}{R}. $$ Taking the square root gives the standard error; replacing the unknown $p$ by $\hat p$ gives its plug-in estimate.

For example, if 640 of 1,000 simulated trials meet a criterion, then $\hat p=0.640$ and the estimated Monte Carlo standard error is $\sqrt{0.640(1-0.640)/1000}\approx0.01518$. An approximate 95% Monte Carlo interval is $0.640\pm1.96(0.01518)$, or about 0.610 to 0.670. This interval reflects simulation error in estimating performance under the specified scenario. It does not represent uncertainty about the true clinical outcome probabilities.

Failure Modes and Practical Comparisons

A stress test is informative only if its scope and interpretation are disciplined. Watch for these warning signs:

Only one challenge is tested. A design that performs well under a plateau may still be vulnerable to delayed toxicity or a changed subgroup mix. Use a reasoned set of scenarios, including selected combinations.
Marginal rates are mistaken for a joint distribution. Two scenarios can have identical efficacy and toxicity rates but materially different joint outcomes. Specify compatible joint probabilities when dependence matters.
Pending outcomes are coded as absent outcomes. This changes the meaning of the data and can distort interim decisions. Simulate observation times and apply the prespecified rule for pending outcomes.
Performance is judged only by the most common recommendation. A high selection rate can coexist with an unacceptable unsafe-selection rate. Report multiple criteria and the allocation or stopping pattern.
Small simulation differences are overinterpreted. Compare a difference with its Monte Carlo uncertainty and increase the number of replicates if precision is inadequate.

Stress testing complements, rather than replaces, calibration. Calibration addresses whether chosen operating targets are met in specified scenarios; stress testing examines how performance changes beyond the scenarios that motivated those choices. Deterministic sensitivity calculations can quickly show how a decision rule responds to altered probabilities, while full trial simulation captures adaptive allocation, stopping, delayed observation, and random variation together. Stress testing cannot establish that the scenario set contains the true clinical process, and it cannot guarantee performance under every unconsidered condition.

Key takeaway: Keep the design fixed, vary one clinically meaningful assumption at a time, and then test selected combinations. Define joint outcome probabilities and observation timing explicitly, and judge robustness using several decision-relevant measures with their simulation uncertainty.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If $P(E=1)=0.50$ and $P(T=1)=0.30$, what is the largest possible value of $P(E=1,T=1)$?
A simulation estimates correct selection in 450 of 900 replicates. Calculate the estimated rate and plug-in Monte Carlo standard error.
Why is $3/12$ not automatically an interim estimate of the eventual probability of efficacy without toxicity when only 6 of 12 patients have both outcomes observed and 3 are favorable?
Name two design performance measures, besides correct dose selection, that could be compared under prompt and delayed outcomes.
Why can a change in subgroup enrollment proportions change the dose selected under a population-average benefit-risk rule?

Solutions. (1) The joint probability cannot exceed either marginal probability, so its maximum is $\min(0.50,0.30)=0.30$; this value is feasible. (2) The estimate is $450/900=0.500$, and the plug-in standard error is $\sqrt{0.500(1-0.500)/900}\approx0.01667$. (3) Six patients have incomplete joint follow-up; using 3/12 implicitly treats them as not having the eventual favorable outcome, which is not justified without a specified rule. (4) Examples include unsafe-dose selection, stopping without a recommendation, safety-rule triggers, and time to recommendation. (5) A population-average outcome probability is a weighted average of subgroup-specific probabilities; changing the weights can alter dose eligibility and the ranking of eligible doses.