Two Ways to Handle Pending Responses
In a response-adaptive trial, an allocation decision can arrive before some earlier participants’ outcomes are known. One option is to pause enrollment until those outcomes resolve, then update the allocation rule using the observed responses. Another is to assign the next participant now by averaging the allocation rule over the possible pending outcomes. These decisions differ in timing and in the information available when the assignment is made.
The comparison depends on more than the number of pending responses. It depends on the model for their outcomes, the rule that converts updated response information into assignment probabilities, and whether the delay process itself is informative. The calculations below use a fully specified binary-response model and a deliberately simple allocation rule. That rule is an example for comparing the two approaches, not a universal prescription.
Model and Allocation Rule for the Calculations
Suppose each participant has a binary response, coded as success or failure. Within each treatment arm, responses are conditionally independent Bernoulli observations with a fixed success probability \(p_j\) for arm \(j\). Before observing outcomes, use independent \(\operatorname{Beta}(1,1)\) priors for the two arms. Given \(s_j\) resolved successes and \(f_j\) resolved failures, the posterior for \(p_j\) is \(\operatorname{Beta}(1+s_j,1+f_j)\). This model assumes that, conditional on the response probability, the pending outcomes have the same response distribution as resolved outcomes. It also assumes that the response probabilities are stable over the period considered.
For illustration, let \(m_A\) and \(m_B\) denote the posterior mean response probabilities for arms A and B. The trial’s specified allocation rule assigns the next participant to A with probability
This rule uses posterior mean response probabilities as allocation scores and normalizes them to sum to one. It is not the only possible response-adaptive rule; the comparison is meaningful only when both approaches use the same rule. For a prediction-based decision, the relevant quantity is generally the expected value of the allocation probability after the pending outcomes are observed, not simply the allocation probability computed from an averaged response estimate.
Worked Example: Two Pending Responses on Arm A
Worked Example: Two Pending Responses on Arm A
Hypothetical trial state. In arm A, two resolved participants responded and two did not, while two additional outcomes are pending. With the stated \(\operatorname{Beta}(1,1)\) prior, the posterior based on resolved outcomes is \(\operatorname{Beta}(3,3)\). In arm B, suppose the resolved data consist of zero successes and one failure, giving posterior \(\operatorname{Beta}(1,2)\). There are no pending B outcomes. Thus \(m_B=1/3\).
Let \(K\) be the number of successes among the two pending A outcomes. Given the current A posterior, its beta-binomial predictive probability is
Substitution gives \(P(K=0)=B(3,5)/B(3,3)=2/7\), \(P(K=1)=2B(4,4)/B(3,3)=3/7\), and \(P(K=2)=B(5,3)/B(3,3)=2/7\). These probabilities sum to one. In particular, the two pending responses do not have to be treated as independent draws with a fixed success probability of \(1/2\): integrating over uncertainty in \(p_A\) gives the beta-binomial distribution.
If the pending outcomes resolve with \(K=k\), the updated A posterior is \(\operatorname{Beta}(3+k,5-k)\), whose mean is \((3+k)/8\). Applying the specified allocation rule with \(m_B=1/3\) gives:
- For \(K=0\), \(m_A=3/8\), so \(q_A=(3/8)/(3/8+1/3)=9/17=0.5294\).
- For \(K=1\), \(m_A=4/8=1/2\), so \(q_A=(1/2)/(1/2+1/3)=3/5=0.6000\).
- For \(K=2\), \(m_A=5/8\), so \(q_A=(5/8)/(5/8+1/3)=15/23=0.6522\).
A pause-until-resolution policy makes no assignment at the current decision time. After resolution, its probability of assigning the next participant to A will be the applicable value above, depending on the observed \(K\). A posterior-prediction policy assigns now using the average of those probabilities:
Thus, under this specified rule and model, the predictive policy assigns the next participant to A with probability 0.5947 now. If the trial instead waits, the eventual probability will be between 0.5294 and 0.6522 according to the pending responses. The average of those future probabilities under the stated predictive model is also 0.5947. That equality concerns an average over possible future outcomes; it does not make the timing or the realized decisions identical.
Why the Prediction Averages Allocation Probabilities
The key justification is the law of total probability, applied to the allocation event and partitioned by the possible values of \(K\). Let \(A_{\text{next}}\) denote the event that the next participant is assigned to A. If the trial waits and then uses \(q_A(k)\) after observing \(K=k\), the chance of assignment to A averaged over the currently unresolved outcomes is
The second equality follows because the allocation rule assigns to A with probability \(q_A(k)\) after \(K=k\) is known. The posterior-prediction rule uses this sum as its current randomization probability. This is a complete justification of the averaging step, provided the predictive probabilities and the allocation rule are correctly specified.
In general, averaging the response probability first and then applying the allocation rule is different. In the example, the posterior predictive mean of A’s response probability after the two pending outcomes remains \(1/2\). Plugging that mean and \(m_B=1/3\) directly into the rule gives \(0.6000\), not \(0.5947\). The difference occurs because \(m_A/(m_A+m_B)\) is nonlinear in \(m_A\). Averaging a nonlinear function does not usually equal applying that function to an average.
Worked Example: One Pending Response
Worked Example: One Pending Response
Use the same hypothetical resolved data and posteriors as above, but suppose only one A outcome is pending. The current A posterior is still \(\operatorname{Beta}(3,3)\), so the pending response is a success with predictive probability \(3/(3+3)=1/2\), and a failure with probability \(1/2\).
If it is a success, the A posterior becomes \(\operatorname{Beta}(4,3)\), with mean \(4/7\). The resulting allocation probability is
If it is a failure, the posterior becomes \(\operatorname{Beta}(3,4)\), with mean \(3/7\), and
The posterior-prediction probability is therefore
Using the current posterior mean \(m_A=1/2\) directly would yield \(0.6000\). The calculation confirms that the predicted allocation probability averages the possible post-update decisions. The pause policy still waits; if it resumes after a success, its probability is 0.6316, and if it resumes after a failure, its probability is 0.5625.
Worked Example: A Different Posterior Changes the Comparison
Worked Example: A Different Posterior Changes the Comparison
Consider a second hypothetical trial state with A’s resolved data yielding posterior \(\operatorname{Beta}(2,4)\), two A outcomes pending, and B’s posterior still \(\operatorname{Beta}(1,2)\). This example retains the same prior family, outcome model, and allocation rule, but the resolved A data now imply a lower posterior mean and a different predictive distribution.
For \(K\), the number of successes among the two pending responses, the beta-binomial probabilities are
Evaluating gives \(P(K=0)=10/21\), \(P(K=1)=8/21\), and \(P(K=2)=3/21\). These probabilities sum to one. The updated A posterior means for \(K=0,1,2\) are respectively \(2/8=1/4\), \(3/8\), and \(4/8=1/2\). With \(m_B=1/3\), the corresponding allocation probabilities are \(3/7=0.4286\), \(9/17=0.5294\), and \(3/5=0.6000\).
The posterior-prediction probability is
A pause policy would wait for the two responses and then use the relevant conditional probability. A predictive policy would assign now to A with probability 0.4915. The calculation shows why the number pending alone does not determine the assignment: both the posterior predictive distribution and the post-outcome allocation probabilities matter.
Assumptions, Diagnostics, and Practical Trade-offs
The calculations are valid under explicit conditions. The binary response model must be appropriate; the Beta posterior must represent the resolved information; and the pending outcomes must be exchangeable with the outcomes used to form that posterior, conditional on the modeled information. The response probability must not have shifted materially with calendar time or participant characteristics omitted from the model. The delay process must also be non-informative about response after conditioning on the information included in the prediction. If, for example, success and failure outcomes resolve at different rates, knowing that an outcome is still pending can itself change its predictive distribution.
The allocation rule is another assumption, not a consequence of the Beta model. The tutorial’s normalized-posterior-mean rule must be the actual prespecified rule for the numerical calculations to apply. A rule based on posterior probability of superiority, a target allocation ratio, or a minimum allocation floor would produce different conditional probabilities and therefore a different predictive average. The same rule must be used after waiting and under prediction to make the comparison interpretable.
Useful diagnostics include tracking the number and age of unresolved outcomes by arm, comparing resolution rates by known response predictors, and checking whether the pending-case mix differs from the resolved-case mix. A growing backlog is important because predictions then influence decisions for longer and may be based on a less representative observed subset. If allocation probabilities move sharply across plausible pending-outcome scenarios, the prediction is particularly sensitive to model assumptions; reporting the scenario-specific values, as in the first example, makes that dependence visible.
| Approach | Immediate decision | Main advantage | Main limitation |
|---|---|---|---|
| Pause until resolution | No assignment until the specified outcomes are observed | Uses observed outcomes rather than predictions for that update | Can slow enrollment and leave capacity or participants waiting |
| Posterior prediction | Assigns now using the average post-update allocation probability | Maintains enrollment while incorporating uncertainty about pending responses | Depends on a credible outcome and delay model |
| Single imputation | Substitutes one predicted value for each pending response | Simple to calculate | Can hide uncertainty and need not equal the correct probability average |
Single imputation is not the same as posterior prediction. Replacing each pending response by its most likely value and applying the allocation rule once discards the other possible outcomes. Nor is waiting automatically free of modeling concerns: waiting changes the timing of enrollment and can alter which calendar-time conditions apply when assignments resume. The comparison here isolates the pending-response calculation and assumes no such change during the wait.
Common Mistake
The pause and prediction approaches answer different operational questions. Pausing asks what decision to make after direct outcome information arrives; prediction asks how to act now while representing uncertainty about that information. Neither is universally preferable. The choice should reflect the cost of delayed enrollment, the consequences of allocation errors, and the credibility of the outcome and delay assumptions. Prediction does not eliminate uncertainty; it makes that uncertainty part of the randomization probability.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- With a \(\operatorname{Beta}(2,2)\) posterior and one pending binary response, what is the predictive probability of success?
- In the two-pending-response example, why is the predictive allocation probability an average of \(q_A(k)\), rather than just \(q_A\) evaluated at the mean response probability?
- Suppose pending failures resolve more slowly than pending successes, and resolution timing is not included in the model. Which assumption is threatened?
- For the \(\operatorname{Beta}(2,4)\) example, what is the probability of assigning the next participant to A after waiting if both pending responses are failures?
Solutions. (1) The predictive success probability is \(2/(2+2)=1/2\). (2) The rule is nonlinear, so the expected post-update allocation probability is \(\sum_k q_A(k)P(K=k\mid\text{resolved data})\); in general this is not the rule evaluated at an averaged response estimate. (3) The assumption that pending outcomes are exchangeable with resolved outcomes, conditional on modeled information, is threatened; pending status carries information about response. (4) Two failures mean \(K=0\), for which the updated A posterior is \(\operatorname{Beta}(2,6)\) with mean \(1/4\); therefore \(q_A=(1/4)/(1/4+1/3)=3/7=0.4286\).