Calculating Allocation Probabilities With Delayed Data

From Delayed Outcomes to an Assignment Probability

An adaptive target is not yet a randomization probability until the trial’s allocation rule says how to turn that target into an assignment for the next patient or cohort. With delayed outcomes, this calculation must also respect what is known at the decision time. Some assigned patients may not yet have a recorded response, and their outcomes must not be silently treated as either successes or failures.

The relevant information can include resolved responses, pending-outcome status, and predictions permitted by the protocol. Earlier material on staleness bias explains why delayed feedback can matter for response estimates; here the task is narrower and operational: given the information and target rule, calculate the probability of each assignment.

Definition: At decision time \(t\), let \(K\) be the number of treatment arms and let \(q_{a,t}\) be the target allocation share for arm \(a\), where \(a=1,\ldots,K\). A valid target-share vector satisfies \(q_{a,t}\geq 0\) for every arm and \(\sum_{a=1}^{K}q_{a,t}=1\). The allocation probability \(\pi_{a,t}\) is the probability that the randomization procedure assigns the next patient to arm \(a\), conditional on the information and randomization constraints in force at time \(t\).

The target and the probability coincide when the protocol assigns each new patient independently using the target shares. They need not coincide when the protocol adds an exploration floor, imposes allocation caps, or requires a cohort to meet fixed counts. The protocol’s rule determines which calculation applies.

Specify the Information Before Calculating

For each arm, distinguish the number of resolved outcomes from the number of pending outcomes. In a binary-response example, write \(n_a\) for resolved outcomes and \(s_a\) for resolved successes. If \(m_a\) outcomes are pending, an allowed predictive procedure might supply expected success probabilities \(r_{aj}\) for pending patient \(j\) on arm \(a\). The corresponding expected number of successes is \(\sum_{j=1}^{m_a}r_{aj}\). It is a predictive quantity, not a count of observed successes.

One possible input to an allocation rule is the predictive response estimate

\tilde p_{a,t} = \frac{s_a+\sum_{j=1}^{m_a}r_{aj}}{n_a+m_a}.

This calculation assumes that resolved and pending patients belong to the same defined response cohort for this estimate, that the pending predictions are available at the decision time, and that the protocol permits those predictions to be used. If no prediction is permitted, the formula is not a license to invent one: use the protocol’s specified rule for pending outcomes. A pending patient is already assigned and must not be randomized again.

Once the permitted information is fixed, apply the target rule exactly as specified. There is no universal rule mapping response estimates to target shares. For example, one protocol might allocate in proportion to estimated response rates, while another might use a square-root rule or a model-based target. The calculations below use explicit hypothetical rules so that the arithmetic is reproducible.

Normalize Weights and Apply an Exploration Floor

A common calculation starts with nonnegative arm-specific weights \(w_{a,t}\), obtained from the protocol’s adaptive rule, and normalizes them:

r_{a,t}=\frac{w_{a,t}}{\sum_{b=1}^{K}w_{b,t}}.

This requires a positive total weight. The normalized values sum to one, so they form a probability vector. If the protocol requires every arm to retain at least a minimum probability \(\epsilon\), one standard floor rule is

\pi_{a,t}=\epsilon+(1-K\epsilon)r_{a,t}, \qquad 0\leq \epsilon\leq \frac{1}{K}.

This rule reserves a total probability of \(K\epsilon\) for the minimum allocations, then distributes the remaining probability according to the normalized weights. It guarantees \(\pi_{a,t}\geq\epsilon\), and

\sum_{a=1}^{K}\pi_{a,t} =K\epsilon+(1-K\epsilon)\sum_{a=1}^{K}r_{a,t} =K\epsilon+(1-K\epsilon)=1.

The floor is applied to the normalized shares as shown; simply replacing any share below \(\epsilon\) with \(\epsilon\) generally makes the probabilities sum to more than one. Other protocols may use caps or a different constrained transformation, in which case the stated rule must be followed instead.

Worked Example: Two Arms With Predicted Pending Responses

Suppose a hypothetical trial has two arms and the protocol allocates in proportion to estimated response rates, with a minimum allocation probability of \(\epsilon=0.10\) per arm. Arm A has 12 successes among 20 resolved patients and 4 pending patients, each with a permitted predicted response probability of 0.75. Arm B has 8 successes among 20 resolved patients and 5 pending patients, each with a permitted predicted response probability of 0.40. Assume the predictions are the inputs specified by the protocol and that the response estimates use the resolved and pending patients described.

For arm A, the predicted contribution from pending patients is \(4(0.75)=3\) expected successes, so

\tilde p_{A,t}=\frac{12+3}{20+4}=\frac{15}{24}=0.625.

For arm B, the predicted contribution is \(5(0.40)=2\), giving

\tilde p_{B,t}=\frac{8+2}{20+5}=\frac{10}{25}=0.400.

Under this example’s proportional-to-response rule, the normalized shares are

r_{A,t}=\frac{0.625}{0.625+0.400}=0.6098, \qquad r_{B,t}=\frac{0.400}{0.625+0.400}=0.3902.

There are two arms, so \(1-K\epsilon=1-2(0.10)=0.80\). Applying the floor gives

\pi_{A,t}=0.10+0.80(0.6098)=0.5878, \qquad \pi_{B,t}=0.10+0.80(0.3902)=0.4122.

The probabilities sum to 1.0000. For the next patient, the assignment probabilities are therefore approximately 0.588 to A and 0.412 to B. The pending responses entered through the stated predictions; they were not recorded as observed successes.

Other Target Rules and Cohort Constraints

Some allocation rules transform scores rather than using them directly. For example, if a protocol specifies weights \(w_{a,t}=\sqrt{\tilde p_{a,t}}\), take the square root of each permitted input first, then normalize the weights. The exponent or transformation is a design choice, not a correction to be selected after seeing the results.

Worked Example: Three Arms Under a Square-Root Rule

Suppose a hypothetical three-arm trial’s delayed-data update supplies response scores of 0.50, 0.40, and 0.25 for arms A, B, and C. Assume the protocol specifies weights equal to the square roots of these scores, followed by the exploration-floor rule with \(\epsilon=0.05\). Treat the supplied scores as inputs; no further pending-outcome model is needed for this calculation.

The weights are \(\sqrt{0.50}=0.7071\), \(\sqrt{0.40}=0.6325\), and \(\sqrt{0.25}=0.5000\), with total \(1.8396\). The normalized shares are approximately

(r_{A,t},r_{B,t},r_{C,t}) = \left( \frac{0.7071}{1.8396}, \frac{0.6325}{1.8396}, \frac{0.5000}{1.8396} \right) = (0.3844,0.3438,0.2718).

Here \(1-K\epsilon=1-3(0.05)=0.85\). Thus

\begin{aligned} \pi_{A,t}&=0.05+0.85(0.3844)=0.3767,\\ \pi_{B,t}&=0.05+0.85(0.3438)=0.3422,\\ \pi_{C,t}&=0.05+0.85(0.2718)=0.2810. \end{aligned}

The rounded probabilities sum to 0.9999 because of rounding; before rounding, the formula gives a sum of 1. The probabilities each exceed 0.05, and arm A has the largest share because its protocol weight is the largest.

A different issue arises when a cohort must meet exact allocation counts. In that design, the target shares may be converted into integer quotas for the cohort, and assignments are randomized subject to those remaining quotas. The next patient’s probability then depends on which quota slots remain, not just on the original target share.

Worked Example: Sequential Probabilities Within a Fixed Cohort

Suppose a hypothetical update, based on the information permitted at the time, sets cohort target shares to 0.60 for A and 0.40 for B. The protocol requires exactly five patients in the next cohort and assigns quotas of three to A and two to B. Assume the protocol randomizes the order of those five quota slots uniformly. This is a constrained cohort rule, not independent randomization with probability 0.60 at every position.

At the first position, three of the five slots are for A, so the probability of assigning A is \(3/5=0.60\). If the first patient is assigned to A, two A slots remain among four positions, giving a second-position probability of \(2/4=0.50\). If the first patient is assigned to B, all three A slots remain among four positions, giving \(3/4=0.75\) at the second position.

In both branches, the probability changes because the available quota slots change. Patients assigned in earlier cohorts whose outcomes remain pending are not extra slots in this new five-patient cohort. Their outcomes may affect the updated target only through the protocol’s permitted delayed-data procedure.

Calculation Checks and Failure Modes

Before using a calculated probability, check both the inputs and the randomization mechanism. A correct arithmetic result can still implement the wrong design if it uses information unavailable at the decision time or ignores a cohort constraint.

Check the information set. Use only resolved outcomes and pending-outcome predictions permitted at that decision time. Do not treat pending patients as observed failures, successes, or new patients to assign.
Check the denominator. For normalized weights, verify that the total weight is positive. If every weight is zero, the stated normalization is undefined; use the protocol’s prespecified fallback rather than dividing by zero.
Check constraints. Verify that the floor is feasible, that each final probability meets it, and that the probabilities sum to one. Apply any protocol caps or quota rules before randomization.
Keep precision until the end. Rounding intermediate shares can produce a vector that does not sum exactly to one or can alter a small probability. Round for reporting after the unrounded values have been used in the randomization calculation.
Check the unit of randomization. Independent patient-level randomization, fixed cohort quotas, and other constrained mechanisms can have different next-patient probabilities even when they use the same target shares.

The main alternatives differ in how they handle delayed feedback. A waiting rule postpones an update until specified outcomes resolve; it avoids using predictions in that update but can delay adaptation. A predictive or imputation-based rule can update sooner, but the target then depends on the prescribed predictive inputs. A complete-resolved-outcomes calculation can also be specified, but it discards pending patients from the estimate and may reflect a different information set. These are design choices; none changes the need to calculate and document the final randomization probability correctly.

Allocation arithmetic alone does not establish that a predictive model is well calibrated, that the resulting estimator has a particular bias, or that the target is optimal. Those are separate questions. The practical limitation is especially important when delays are informative or differ across arms: the allocation calculation can be executed exactly while the inputs remain sensitive to the assumptions that produced them.

Key takeaway: State the permitted delayed-data inputs, apply the protocol’s target transformation, and then apply its randomization constraints. Report the resulting probability for the next assignment—not merely the target share—and verify its range and total.

Check Your Understanding

Work each problem, then compare your calculations with the solutions below.

A two-arm rule uses weights 0.6 and 0.4, with independent assignment and no floor. What are the assignment probabilities?
A two-arm rule uses normalized shares 0.70 and 0.30 and an exploration floor \(\epsilon=0.10\). Calculate the final probabilities using the stated floor rule.
In a five-patient cohort, the fixed quotas are three for A and two for B. If the first two assignments were A then B, what is the probability of A at the third position under a uniformly randomized order of quota slots?
Why is a pending response prediction not counted as an observed success in an allocation calculation?

Solutions. (1) The weights sum to 1, so the probabilities are \(0.6/1.0=0.60\) for A and \(0.4/1.0=0.40\) for B. (2) For two arms, \(1-K\epsilon=1-2(0.10)=0.80\). Thus \(\pi_A=0.10+0.80(0.70)=0.66\) and \(\pi_B=0.10+0.80(0.30)=0.34\); they sum to 1. (3) After A then B, two A quota slots remain among three positions, so the probability is \(2/3=0.6667\). (4) The response has not been observed. A prediction may enter only as the protocol specifies, and must remain distinguishable from a realized outcome.