Allocation Updates With Delayed Outcomes

Why Delayed Outcomes Change an Allocation Update

In an adaptive trial, an allocation decision can be due before every participant’s response is available. A participant may have started treatment but not yet reached the response assessment, or the assessment may have occurred but not yet been recorded. At that decision time, the design must specify exactly which information is allowed to affect the next assignment.

This tutorial considers an allocation rule that uses responses observed by the decision time. A response that has not yet been observed does not count as either a response or a nonresponse. This distinction matters: the allocation probability is calculated from the information currently available under the protocol, not from what may later become known.

Let \(D_t^{\mathrm{obs}}\) denote the observed response history just before the next assignment. If there are \(J\) treatment arms, let \(q_{t+1,j}\) be the probability of assigning the next participant to arm \(j\). A completed-only allocation rule has the form

The function \(g\) is specified by the design. It may use posterior probabilities, posterior expected utilities, or another prespecified allocation calculation. The defining restriction here is that \(g\) uses only the permitted observed history at that decision. Randomization then uses the resulting probabilities; it does not make future or unrecorded outcomes available to the rule.

A Binary-Response Model for Completed Data

For illustration, suppose participant \(i\), if assigned to arm \(j\), has a binary response \(Y_i\), with \(Y_i=1\) for response and \(Y_i=0\) for no response at a fixed assessment time. Let \(p_j\) be the response probability on arm \(j\). A simple model assumes that responses within arm \(j\) are conditionally independent Bernoulli observations with common probability \(p_j\). Assign independent Beta priors, \(p_j\sim\operatorname{Beta}(a_j,b_j)\), where \(a_j,b_j>0\).

At time \(t\), let \(n_j(t)\) be the number of completed responses observed on arm \(j\), and let \(y_j(t)\) be the number of those responses equal to 1. Under this model, the posterior based on completed observations is

The contribution of each completed response follows directly from the Bernoulli likelihood. The likelihood for the \(n_j(t)\) observed responses is proportional to \(p_j^{y_j(t)}(1-p_j)^{n_j(t)-y_j(t)}\). Multiplying by the Beta prior density, proportional to \(p_j^{a_j-1}(1-p_j)^{b_j-1}\), gives a density proportional to

which is the kernel of the stated Beta posterior. A participant with no observed response contributes no Bernoulli factor to this completed-data likelihood at this decision. In particular, a pending response is not entered as \(Y_i=0\).

The assumptions supporting this calculation include a correctly defined binary response and assessment time, a stable arm-specific response probability over the period being modelled, and conditional independence of responses given the arm probabilities. For completed observations to represent the modelled response process, the timing of observation must not select participants according to their unobserved response in a way that the model ignores. Administrative staggering—such as later enrollees simply not having reached a fixed assessment window yet—can be compatible with the model when enrollment timing is not driven by the unobserved response. If observation or recording depends on response, or response probabilities change over calendar time, the completed-only posterior may not describe the intended response probabilities without additional modelling.

Worked Allocation Calculations With Staggered Follow-Up

The examples use a transparent illustration: allocate in proportion to the posterior mean response probability. This is one possible rule, not a general recommendation. Its purpose is to show precisely how the observed history enters a probability calculation. For two arms, if the current posterior means are \(m_A\) and \(m_B\), the rule is \(q_A=m_A/(m_A+m_B)\) and \(q_B=m_B/(m_A+m_B)\). The probabilities are positive and sum to one when both means are positive.

Protocol, Checks, and Alternative Update Choices

A reproducible delayed-outcome rule requires more than a posterior formula. The protocol should state the response definition and assessment window, the exact cutoff for deciding whether a response is observed, the scheduled update times, the statistical rule \(g\), and how the random assignment is generated from its probabilities. These details determine \(D_t^{\mathrm{obs}}\) and therefore determine the allocation.

Useful operational checks include comparing the participant-level assignment and response timestamps with the update cutoff; checking that pending outcomes have not entered completed-response counts; confirming that the arm-specific counts satisfy \(0\leq y_j(t)\leq n_j(t)\); and verifying that calculated probabilities are between zero and one and sum to one. Monitoring how many outcomes are completed by each update can reveal whether the actual information available is much thinner or more imbalanced than anticipated.

Two common design choices are to update using completed outcomes at scheduled decision times or to delay an update until a prespecified set of assessments is complete. The first can allow allocation to respond sooner to observed data, but different decisions may use different amounts of information and follow-up. The second can provide a more synchronized data snapshot, but may lengthen the time between allocation decisions. Either choice needs a prespecified schedule and consistent application. A completed-only rule should not silently switch into treating pending outcomes as failures or into using unobserved future responses.

The simple model and allocation rule also have limits. With few completed observations, posterior means can be sensitive to the prior, and allocation probabilities can change substantially when a single response matures. If response probability varies over calendar time, a posterior that assumes one constant \(p_j\) can mix different response processes. If follow-up or outcome recording depends on the unobserved response, the completed cases may be systematically unrepresentative. In these situations, the completed-only calculation should not be treated as automatically valid merely because it uses recorded data. The model and the observation process need to match the trial.