Sensitivity Analysis for Misspecified Delay Models

Why Delay-Model Sensitivity Matters

When outcomes take time to resolve, an allocation rule may use predictions for participants whose responses are still pending. Those predictions depend on assumptions about who resolves quickly. If response-positive and response-negative participants have different delay distributions, then the set of outcomes observed so far need not represent the set still pending. A delay model that misses this relationship can therefore change estimated arm response rates and the next assignment probabilities.

Sensitivity analysis asks a focused question: how much would the allocation change if a plausible alternative model described the response delays? It does not identify the correct model by itself, and it does not establish that any model is adequate. Its value is to make the dependence of a decision on delay assumptions explicit.

Predicting a Pending Response Under a Delay Model

Let $Y=1$ denote a response and $Y=0$ no response. Let $T$ be time from assignment until the outcome resolves. For treatment arm $a$, write $\theta_a=P(Y=1\mid A=a)$, and define the response-specific delay survival function $S_{ay}(u)=P(T>u\mid A=a,Y=y)$. Thus, $S_{ay}(u)$ is the probability that an outcome of type $y$ in arm $a$ remains unresolved after $u$ time units.

Definition: Under a specified response-and-delay model, the predicted response probability for an unresolved participant in arm $a$, whose outcome has remained unresolved through time $u$, is $$ q_a(u)=P(Y=1\mid T>u,A=a) =\frac{\theta_a S_{a1}(u)} {\theta_a S_{a1}(u)+(1-\theta_a)S_{a0}(u)}. $$ The prediction uses both the response probability and the relative chance of remaining unresolved for responders and nonresponders.

This expression follows from Bayes' rule. The numerator is the joint probability of a response and continued nonresolution; the denominator is the probability of continued nonresolution under either response status. If $S_{a1}(u)=S_{a0}(u)$, delay is noninformative about response at time $u$, and $q_a(u)=\theta_a$. If responders resolve faster, then $S_{a1}(u)<S_{a0}(u)$, so the pending group is less response-prone than the arm as a whole, all else equal.

The calculation assumes that the model's $\theta_a$ and survival functions refer to the relevant arm and follow-up time, and that the pending status really means the outcome has not yet resolved, rather than an unrecorded outcome or dropout. For participants with different elapsed times $u_i$, predictions should use their own $q_a(u_i)$. In the examples below, elapsed times are held fixed for simplicity.

Worked Example: A Pending Participant With Response-Dependent Delay

Consider a hypothetical trial. In arm A, a working response model gives $\theta_A=0.60$. At the current follow-up time, suppose the delay model gives $S_{A1}(u)=0.30$ and $S_{A0}(u)=0.60$. Calculate the predicted response probability for a participant whose outcome is still pending.

Substitution into the conditional probability formula gives $$ q_A(u)=\frac{0.60(0.30)} {0.60(0.30)+(1-0.60)(0.60)} =\frac{0.18}{0.18+0.24} =\frac{0.18}{0.42} =0.4286. $$

The model therefore assigns a 42.86% predicted response probability to this pending participant, below the arm-level response probability of 60%. The reason is that responders are assumed less likely than nonresponders to remain unresolved: the respective survival probabilities are 0.30 and 0.60. This is a model-based prediction, not a statement that the participant's response is known.

From Pending Predictions to Allocation Probabilities

To compare delay assumptions, hold the response model, observed data, and allocation rule fixed; change only the delay component. For a transparent illustration, define the current predicted arm response score as the average of observed outcomes and predicted outcomes among all assigned participants:

\tilde{\theta}_a = \frac{\sum_{i:A_i=a,\ \text{resolved}}Y_i+ \sum_{i:A_i=a,\ \text{pending}}q_a(u_i)} {n_a}.

Here, $n_a$ is the number assigned to arm $a$, resolved outcomes contribute their observed values, and pending outcomes contribute their model-based response probabilities. This score is used here as a specified allocation input; other procedures may estimate response rates differently.

For the examples, use a success-proportional allocation rule: the next participant is assigned to arm A with probability $$ p_A=\frac{\tilde{\theta}_A}{\tilde{\theta}_A+\tilde{\theta}_B}, \qquad p_B=1-p_A. $$ This is a deliberately simple rule for isolating the impact of delay assumptions. It requires nonnegative scores whose sum is positive. In an actual protocol, the allocation function and any restrictions on its probabilities must be specified in advance.

Worked Example: Comparing Three Delay Assumptions

Suppose a hypothetical trial has 10 participants in each arm. In arm A, 6 outcomes have resolved, with 4 responses; in arm B, 6 have resolved, with 3 responses. Four outcomes in each arm are pending at the same elapsed time. Keep the response-model estimates fixed at $\theta_A=0.60$ and $\theta_B=0.55$. Compare three delay specifications: (I) response-independent delays, giving pending predictions $q_A=0.60$, $q_B=0.55$; (II) a model with $q_A=0.40$, $q_B=0.70$; and (III) an alternative with $q_A=0.60$, $q_B=0.45$. These are hypothetical model outputs, not trial findings.

Under specification I, the pending predictions equal the arm-level response probabilities. The scores are $$ \tilde{\theta}_A=\frac{4+4(0.60)}{10}=0.6400, \qquad \tilde{\theta}_B=\frac{3+4(0.55)}{10}=0.5200. $$ Thus $p_A=0.6400/(0.6400+0.5200)=0.5517$. The next participant has a 55.17% assignment probability to A under this rule.

Under specification II, $$ \tilde{\theta}_A=\frac{4+4(0.40)}{10}=0.5600, \qquad \tilde{\theta}_B=\frac{3+4(0.70)}{10}=0.5800, $$ so $p_A=0.5600/(0.5600+0.5800)=0.4912$. The allocation now slightly favors B.

Under specification III, $$ \tilde{\theta}_A=\frac{4+4(0.60)}{10}=0.6400, \qquad \tilde{\theta}_B=\frac{3+4(0.45)}{10}=0.4800, $$ so $p_A=0.6400/(0.6400+0.4800)=0.5714$. Across these specifications, $p_A$ ranges from 0.4912 to 0.5714, and the preferred arm changes under specification II. That change is a substantive sensitivity finding: the assignment depends on the assumed relationship between response and delay.

Why Alternative Models Can Reverse a Decision

The conditional prediction depends on the relative survival $S_{a1}(u)/S_{a0}(u)$, not just on the overall proportion of unresolved outcomes. For fixed $\theta_a$, a smaller relative survival for responders lowers $q_a(u)$; a larger relative survival raises it. A useful derivation is to compare the odds among pending participants with the model's arm-level response odds:

\frac{q_a(u)}{1-q_a(u)} = \frac{\theta_a S_{a1}(u)} {(1-\theta_a)S_{a0}(u)} = \frac{\theta_a}{1-\theta_a} \frac{S_{a1}(u)}{S_{a0}(u)}.

Thus, pending response odds equal the arm-level response odds multiplied by the relative probability of remaining unresolved for responders versus nonresponders. If that ratio is below one, the pending group has lower response odds than the arm-level model; if above one, it has higher odds. This result explains why a model that treats delays as noninformative can misrepresent pending outcomes even when its response-rate component is unchanged.

Worked Example: Quantifying the Delay Contrast Needed to Change Allocation

Use the data and allocation rule from the previous example, and keep $q_B=0.55$. The B score is then $\tilde{\theta}_B=(3+4(0.55))/10=0.5200$. What value of $q_A$ makes the allocation exactly balanced, $p_A=0.50$?

Under the stated rule, $p_A=0.50$ exactly when the two scores are equal. Set $(4+4q_A)/10=0.5200$. Solving gives $4+4q_A=5.2$, so $q_A=0.30$. If the pending prediction for A is below 0.30 while the B prediction remains 0.55, the score for A is below B's and $p_A<0.50$; if it is above 0.30, $p_A>0.50$.

For example, at $q_A=0.25$, $\tilde{\theta}_A=(4+4(0.25))/10=0.5000$, and $p_A=0.5000/(0.5000+0.5200)=0.4902$. The threshold makes the decision's sensitivity interpretable: it identifies how low the pending-response prediction for A must be, given the other inputs, to change which arm receives the larger assignment probability.

Conducting and Interpreting a Sensitivity Analysis

A useful analysis varies the assumptions that are uncertain while keeping other ingredients constant. Candidate delay models might include a response-independent model, an arm-specific model in which response status predicts resolution time, and plausible alternatives with weaker or stronger response-delay association. The alternatives should be defensible from subject-matter knowledge or available delay information; arbitrary extremes can show mathematical possibility but may not be informative.

Fix the decision rule and response component.
Record the observed data, response-model estimates, elapsed follow-up times, and allocation function that will be held constant.

Specify alternative delay relationships.
For each candidate, state how resolution depends on arm, response status, and elapsed time, and verify that its survival probabilities are valid probabilities between zero and one.

Recalculate pending predictions and scores.
Use the conditional prediction under each model, then recompute each arm score using the same resolved outcomes and denominator.

Compare allocation consequences.
Report the resulting assignment probabilities and whether the ranking or practical size of the allocation changes. Identify which model component drives any difference.

In the numerical comparison, changing the delay model moves $p_A$ from 0.4912 to 0.5714, a range of 0.0802, or 8.02 percentage points. It also changes whether A or B receives the larger probability. Such a result warrants reporting the model dependence and considering whether the protocol's chosen assumptions are adequately supported. The range is not a confidence interval: it summarizes selected scenarios, not repeated-sampling uncertainty.

Diagnostics, Comparisons, and Limitations

Before interpreting the sensitivity results, check that each model's predicted survival probabilities are coherent across time: $S_{ay}(u)$ should lie between zero and one and should not increase as $u$ increases. Check that a pending prediction lies between zero and one, that resolved outcomes remain their observed values, and that the same participants, response component, and allocation rule were used in every scenario. A large change in allocation from a small change in the delay assumptions is evidence of decision fragility, not proof that a particular alternative is correct.

The main contrast is with simply waiting for more outcomes or using a single assumed delay model. Waiting avoids model-based predictions for currently pending outcomes but delays the adaptive update. A single model yields one operational allocation probability but conceals its dependence on model choice. Sensitivity analysis retains timely predictions while showing how alternatives affect them; it does not remove the trade-off or determine which policy is best.

The analysis is limited by its candidate set. If every alternative shares a wrong feature, similar results may provide false reassurance. Estimates of the delay relationship can also be uncertain, especially when few outcomes have resolved. This tutorial holds the response component fixed to isolate delay-model sensitivity; in practice, uncertainty or misspecification in the response model can also affect allocation. Sensitivity results are conditional on the specified data, candidate models, and allocation rule, and should be presented as such.

Common Pitfall

Common mistake: “The observed response rate among resolved participants is the right prediction for everyone still pending, because all participants were randomized.” Randomization assigns treatment; it does not guarantee that resolution time is independent of response within an arm. If responders and nonresponders resolve at different rates, pending status changes the conditional response probability. Use the response-independent prediction only when that assumption is justified or as an explicit sensitivity scenario, and compare it with plausible alternatives.

A second interpretive error is to call the largest allocation probability “the treatment most likely to work.” Here, the probability is produced by a particular allocation rule applied to model-based scores; it is neither a probability that an arm is truly superior nor a direct measure of uncertainty about treatment effects.

Check Your Understanding

Answer each question using the definitions and allocation rule above. Solutions follow.

For $\theta_a=0.50$, $S_{a1}(u)=0.40$, and $S_{a0}(u)=0.80$, calculate the predicted response probability for a pending participant.
In one arm, 5 of 7 resolved outcomes are responses and 3 participants remain pending. Under a model predicting $q=0.40$ for each pending participant, calculate the predicted response score for all 10 assigned participants.
If both arm scores are 0.60 under the success-proportional rule, what is $p_A$?
Why is a range of allocation probabilities across candidate delay models not a confidence interval?

Solutions. (1) $q=0.50(0.40)/[0.50(0.40)+0.50(0.80)]=0.20/0.60=0.3333$. (2) The score is $[5+3(0.40)]/10=6.2/10=0.6200$. (3) $p_A=0.60/(0.60+0.60)=0.5000$. (4) The range reports results under selected assumptions; it does not quantify repeated-sampling coverage for an unknown parameter.