Informative Outcome Delays

Why a Resolved Outcome May Not Be Representative

In response-adaptive randomization, an allocation update may use outcomes that have become available by a decision time while other participants’ outcomes are still pending. That approach is reliable as feedback about the enrolled group only when the resolved outcomes are sufficiently representative, or when the analysis accounts for how resolution was selected. If the eventual response itself affects how quickly an outcome is resolved, the set of available outcomes can overrepresent one response category.

For example, a response might be recorded sooner when a participant improves than when improvement is absent. At an interim update, the observed response rate could then be higher than the response rate among all participants enrolled so far. This is a feature of the observation process, not a change in the eventual outcomes. Random allocation at enrollment does not, by itself, make the subset with resolved outcomes representative.

The definition concerns outcome availability, not necessarily the biological time at which a response occurs. A protocol may schedule assessments at fixed times, or a response may be recorded as soon as it occurs; either arrangement can produce different resolution patterns for different eventual outcomes. The key question is whether the probability of being observed by the update time differs with the eventual response.

How Response-Dependent Delays Select Outcomes

Consider a binary endpoint, with \(Y=1\) for response and \(Y=0\) for no response. Let \(p=P(Y=1)\) be the response probability in the population of interest at a specified enrollment point. For a common follow-up duration \(u\), write \(q_1(u)=P(D\leq u\mid Y=1)\) and \(q_0(u)=P(D\leq u\mid Y=0)\). These are the probabilities that a response or a nonresponse, respectively, has resolved by \(u\).

Assume that all participants under consideration have the same follow-up duration, that their eventual binary outcomes are defined, and that resolved outcomes are recorded correctly. Then the response fraction among resolved outcomes is the conditional probability \(P(Y=1\mid R(u)=1)\). The definition of conditional probability gives:

The denominator is the overall probability of resolution by \(u\), obtained by adding the probabilities of being a resolved responder and a resolved nonresponder. Subtracting the target response probability \(p\) yields:

When \(0<p<1\) and the denominator is positive, the sign of the difference is the sign of \(q_1(u)-q_0(u)\). If responses are more likely to have resolved by \(u\), the resolved-outcome response fraction is too high for \(p\); if nonresponses are more likely to have resolved, it is too low. If \(q_1(u)=q_0(u)\), resolution is independent of this binary outcome at that duration, and the resolved fraction equals \(p\). This is a statement about the expected fraction, not a guarantee that a finite sample fraction will equal \(p\).

Conditions That Can Prevent or Address Selection

The derivation points to two broad strategies: design the observation process so availability does not depend on the outcome, or model the observation process using adequate information and assumptions. These strategies answer different practical needs. Waiting can reduce the number of pending outcomes but delays feedback; adjustment can provide earlier feedback but depends on the adequacy of the model and available data.

Sometimes resolution is associated with the outcome only because both are related to measured baseline information. For instance, a measured clinical category might predict both response and the speed of assessment. If, within each category, resolution by \(u\) is independent of the eventual response, stratifying or weighting on that category can address the resulting selection. This requires that the covariate information be available for resolved and unresolved participants, that the conditional independence assumption be credible, and that every relevant covariate group have a positive chance of resolution.

Diagnostics, Alternatives, and Limitations

A useful diagnostic begins with the follow-up schedule: participants enrolled at different times may have had different opportunities for resolution. Compare like with like by considering a common elapsed follow-up duration, and examine resolution rates across known baseline predictors and, where eventual outcomes later become available, across outcome categories. A cohort followed to completion can help reveal whether early resolved cases differed from the eventual full cohort. Such checks can expose a problem, but unresolved outcomes do not reveal their own response values, so an interim data set alone cannot prove that delays are outcome-independent.

A large number of resolved outcomes is not sufficient evidence against selection. In the worked binary example, half the outcomes were resolved, yet the response fraction among them differed substantially from the cohort fraction. Nor does a small observed difference establish independence: different response-specific resolution probabilities can happen to produce a modest difference in a particular sample. Assess the mechanism and assumptions, not only the size of the pending-outcome backlog.

In an adaptive trial, a distorted interim response fraction can affect subsequent decisions because the update treats the observed outcomes as feedback about the enrolled population. That does not mean every delayed-outcome analysis is biased: the concern depends on the resolution mechanism, the target of the update, and the assumptions used. Conversely, randomization at enrollment is not a remedy for selection that occurs after assignment, when inclusion in the available-outcome set depends on post-enrollment resolution.

Common Mistake