Diagnosing a Backlog of Unresolved Outcomes

Why Measure the Backlog?

Response-adaptive randomization uses accumulated outcome information to inform later assignments. When outcomes are unresolved, the trial may have enrolled many participants but have much less current response information available than its enrollment totals suggest. A backlog diagnostic makes that gap visible. It does not, by itself, say whether unresolved outcomes are unusual, whether treatment is effective, or why resolution is delayed. Its purpose is operational: describe what is pending, where it is pending, and whether the available feedback may be insufficient for the allocation procedure being used.

A useful diagnostic starts with a clearly defined snapshot time and a complete accounting of participants enrolled by that time. Raw numbers alone are rarely enough. If one arm has twice as many enrolled participants as another, it can have more pending outcomes even when its resolution process is faster. Comparisons therefore need denominators, comparable time at risk for resolution, and an understanding of how the allocation algorithm uses the information that has arrived.

Definitions at a Snapshot

Let $E_i$ be participant $i$'s enrollment time, $A_i$ the assigned treatment arm, and $R_i$ the time at which that participant's outcome becomes resolved and available for the adaptive procedure. At calendar time $t$, consider participants enrolled by $t$. Define $I(\cdot)$ to be 1 when its condition holds and 0 otherwise.

Definition: The pending-outcome count in arm $a$ at time $t$ is $$ B_a(t)=\sum_{i:A_i=a,\ E_i\le t} I(R_i>t). $$ The enrolled count is $N_a(t)=\sum_i I(A_i=a,\ E_i\le t)$, and the resolved count is $C_a(t)=N_a(t)-B_a(t)$. When $N_a(t)>0$, the pending fraction is $B_a(t)/N_a(t)$, and the resolved fraction is $C_a(t)/N_a(t)$.

These definitions require consistent operational rules. The snapshot time must be specified; “resolved” must have a consistent meaning across arms; and the enrolled roster and arm assignments must be correctly recorded. If an outcome has several stages, the protocol should state which stage counts as resolution for adaptive feedback. For example, a preliminary response that is not yet usable by the allocation rule should not be counted as available feedback merely because it has been recorded.

A snapshot pending fraction is descriptive, not a delay-adjusted comparison. A participant enrolled yesterday and one enrolled a month ago both contribute one unit to $B_a(t)$ if unresolved, even though they have had very different opportunities to resolve. To compare resolution performance, use cohorts with comparable follow-up time or a model that accounts for enrollment time and other relevant information. The earlier material on informative and treatment-specific outcome delays is relevant when resolution patterns depend on participant or arm characteristics.

Worked Example: A Snapshot with Unequal Enrollment

At a hypothetical trial snapshot, arm A has 80 enrolled participants, of whom 60 have resolved outcomes. Arm B has 40 enrolled participants, of whom 34 have resolved outcomes. Calculate the pending counts and fractions.

For arm A, $B_A=80-60=20$. Its pending fraction is $20/80=0.25$, or 25%, and its resolved fraction is $60/80=0.75$, or 75%.

For arm B, $B_B=40-34=6$. Its pending fraction is $6/40=0.15$, or 15%, and its resolved fraction is $34/40=0.85$, or 85%.

Arm A has more than three times as many pending outcomes as arm B (20 versus 6), but it also has twice as many enrolled participants. The pending fractions show that a larger share of A's enrolled participants is pending at this snapshot. This is a signal to investigate, not proof that outcomes resolve more slowly in A: the arms may differ in how long their participants have been enrolled.

Resolution Rates Need a Follow-Up Window

For an operational resolution-rate comparison, define a cohort using a fixed enrollment interval and a common follow-up horizon $u$. Among participants in arm $a$ enrolled in that cohort, the empirical resolution proportion by $u$ time units after enrollment is the number resolved within that window divided by the number with that full window of observation. This common horizon prevents recently enrolled participants from being treated as if they had the same opportunity to resolve as earlier enrollees.

For example, if a report uses a 10-day window, the denominator should consist of participants whose enrollment was at least 10 days before the data cutoff, unless a correctly specified method accounts for incomplete follow-up. State the time unit and the cohort definition. A simple proportion summarizes resolution by that horizon; it is not a rate per unit time and does not describe what happens after the horizon.

Worked Example: Comparing Resolution at a Common Horizon

In a hypothetical cohort with at least 10 days of follow-up for every participant, 40 of 50 participants in arm A and 45 of 50 in arm B have outcomes resolved within 10 days of enrollment. Calculate and interpret the arm-specific proportions.

For A, the 10-day resolution proportion is $40/50=0.80$, or 80%. For B, it is $45/50=0.90$, or 90%. The descriptive difference, A minus B, is $0.80-0.90=-0.10$, or 10 percentage points lower in A.

Because the cohort members have the same follow-up horizon, this comparison is not mechanically driven by one arm having more recently enrolled participants. It remains a descriptive comparison; uncertainty and possible differences in participant characteristics or the resolution process would matter for broader conclusions. The five-versus-ten pending counts in this cohort are less informative than the proportions because both denominators happen to be 50.

Expected Backlog Under Adaptive Assignment

A delay model can provide a reference for how many outcomes might be pending, but the conditioning must match the design. Write $q_i(t)$ for the conditional probability that participant $i$'s outcome is still pending at $t$, under the information being conditioned on. That information may include enrollment times, assignments, and relevant observed history. It must not be assumed that an arm-specific delay distribution automatically gives the correct probability after conditioning on an adaptive assignment sequence: later assignments can reveal information about earlier resolution status.

Definition: For any specified conditioning information $\mathcal{F}$ that includes the roster and assignments relevant to the calculation, but does not already fix the pending indicators being predicted, define $$ q_i(t)=P(R_i>t\mid \mathcal{F}). $$ Then the conditional expected backlog in arm $a$ is $$ \mathbb{E}\{B_a(t)\mid \mathcal{F}\} = \sum_{i:A_i=a,\ E_i\le t} P(R_i>t\mid \mathcal{F}) = \sum_{i:A_i=a,\ E_i\le t} q_i(t). $$

This identity follows from linearity of conditional expectation; it does not require the participants' resolution indicators to be independent. To see this, write $B_a(t)$ as the sum of its pending indicators. The conditional expectation of each indicator $I(R_i>t)$ is $P(R_i>t\mid\mathcal{F})$, so taking the conditional expectation of the sum gives the displayed expression.

If resolution delays in arm $a$ have distribution function $G_a$, then $1-G_a(t-E_i)$ may represent a participant's pending probability under an appropriate delay model and conditioning scheme. Substituting it for $q_i(t)$ requires the additional assumption that it equals $P(R_i>t\mid\mathcal{F})$ for the chosen $\mathcal{F}$. That equality does not follow solely from having an arm-specific marginal resolution-time distribution. In particular, adaptive assignments can depend on earlier outcomes or their resolution status, making the assignment sequence informative about pending status.

Worked Example: Conditional Expected Backlog

Suppose three hypothetical participants are assigned to arm A and enrolled by the snapshot time. A delay model, calibrated for the conditioning information used in this calculation, gives their conditional probabilities of remaining pending as 0.20, 0.60, and 0.40. Find the expected backlog and compare it with a possible observed backlog.

Using the conditional-probability formula, $$ \mathbb{E}\{B_A(t)\mid\mathcal{F}\}=0.20+0.60+0.40=1.20. $$

An expected count of 1.20 is appropriate even though an observed count must be an integer. For instance, if the first and third participants are actually pending and the second is resolved, the observed count is $1+0+1=2$. An observed count of 2 does not contradict an expectation of 1.20; individual realizations vary. The calculation is valid only if the three probabilities are conditional probabilities for the stated information. It would be unjustified to insert marginal arm-specific probabilities if adaptive assignments make those probabilities inappropriate after conditioning on the assignments.

Reading Backlog as a Feedback Diagnostic

The backlog matters to response-adaptive randomization because the allocation rule can use only the outcome information it has received, unless it explicitly uses predictions for unresolved outcomes. A large pending fraction can mean that the current allocation is based on a small resolved subset of the enrolled sample. It can also mean that the algorithm's effective feedback is old: recent participants may contribute to enrollment totals but not yet to response summaries. Whether this materially impairs the procedure depends on its update schedule, the number of resolved outcomes it needs, and whether it waits, predicts, or combines both approaches.

No single backlog threshold is universally appropriate. A practical monitoring report can include the following indicators at regular snapshots:

Enrolled, resolved, and pending counts by arm, with the pending and resolved fractions.
Resolution proportions by arm at common follow-up horizons and within clearly defined enrollment cohorts.
The age of pending outcomes, summarized by elapsed time since enrollment, so that newly pending and long-outstanding outcomes are not treated as equivalent.
Observed counts compared with model-based expected counts, with the model and conditioning information stated.
The number of resolved outcomes currently available to the allocation rule, alongside the allocation probabilities or update status.

These indicators answer different questions. Counts describe workload; fractions account for arm size; common-horizon proportions compare resolution over equal follow-up; and model-based expectations assess whether observed pending counts are unusual relative to a specified model. None alone establishes that feedback is impaired. For that operational judgment, connect the diagnostics to the prespecified update rule: does the number and recency of resolved outcomes meet the rule's conditions for an update, or is the rule repeatedly acting on sparse or stale information?

Common Pitfalls and Alternatives

Diagnostic choice	What it can show	Main limitation
Raw pending count by arm	Current number of unresolved outcomes	Confounded descriptively by arm size and enrollment timing
Pending fraction by arm	Share of enrolled participants still pending	Still combines participants with different follow-up time
Common-horizon resolution proportion	Resolution experience over equal follow-up windows	Depends on cohort definition and does not summarize later resolution
Model-based expected backlog	Reference count under a specified delay model and conditioning set	Can mislead if the model or conditioning assumptions are wrong

A common mistake is to say, “Arm A has more pending outcomes, so its outcomes resolve more slowly.” In the first worked example, A had 20 pending outcomes and B had 6, but A also had twice as many enrolled participants. Even after comparing fractions, different enrollment ages could explain some or all of the difference. The correction is to report denominators and, when comparing resolution performance, use comparable follow-up or an appropriate model.

Another failure mode is to treat an expected backlog formula as automatic once an arm-specific delay distribution has been specified. Under adaptive assignment, assignments can depend on earlier feedback and therefore carry information about resolution status. The correction is to state the conditioning set and use conditional pending probabilities that are valid for it. If the model is misspecified, expected-versus-observed comparisons can flag a discrepancy but cannot identify its cause.

Finally, backlog indicators have limits. They summarize availability of outcomes, not outcome values or treatment effects. A modest pending fraction can still be problematic if the resolved outcomes are concentrated in an unrepresentative subset or if the algorithm is especially sensitive to recent information. Conversely, a large count may be operationally manageable if updates are scheduled only after a planned follow-up period. Interpretation must remain tied to the trial's stated feedback and update procedure.

Key takeaway: Diagnose unresolved outcomes with a snapshot count, a denominator-based fraction, and a resolution comparison using a common follow-up window. For model-based expectations, use conditional pending probabilities appropriate to the adaptive assignment information. Relate the results to the allocation rule before concluding that feedback is impaired.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

An arm has 72 enrolled participants and 54 resolved outcomes at a snapshot. What are its pending count, pending fraction, and resolved fraction?
Why is it potentially misleading to compare raw pending counts between arms with different enrollment totals?
In a common 14-day follow-up cohort, 36 of 45 participants in arm A and 40 of 50 in arm B are resolved. Compare the resolution proportions.
Two participants in an arm have conditional pending probabilities 0.25 and 0.70 under the specified conditioning information. What is the expected pending count, and does it require independence?
Why may an arm-specific marginal delay distribution fail to give the correct pending probability after conditioning on the full adaptive assignment sequence?

Solutions. (1) The pending count is $72-54=18$; the pending fraction is $18/72=0.25$, or 25%; the resolved fraction is $54/72=0.75$, or 75%. (2) A larger arm can have more pending outcomes simply because more participants were enrolled, and arms may also differ in enrollment age. Report denominators and use comparable follow-up for resolution comparisons. (3) A is $36/45=0.80$, or 80%; B is $40/50=0.80$, also 80%. Their descriptive resolution proportions are equal at 14 days. (4) The expected count is $0.25+0.70=0.95$. Independence is not required because expectation of a sum equals the sum of expectations. (5) Adaptive assignments can depend on earlier resolution information, so conditioning on assignments can change the probability that earlier outcomes remain pending. Use probabilities conditional on the specified information, or justify the assumptions that permit a simpler model.