Deriving Staleness Bias in Adaptive Response Estimates

Why a Resolved Estimate Can Be Stale

In a response-adaptive trial, an estimate based only on resolved outcomes is based on participants who enrolled earlier. If the response distribution changes over calendar time, or if the mix of treatments assigned changes, that estimate can describe an earlier distribution rather than the one faced by participants enrolling now. This difference is staleness bias: a systematic gap between the target at the current time and the average response represented by the resolved data.

Staleness is not inevitable. If responses within a treatment arm have a stable distribution over enrollment time, resolution is non-informative, and the target is that arm’s stable response mean, an estimate from resolved participants can be unbiased for that mean. It may still be based on older information and have substantial sampling error. A changing allocation mix can also make a pooled estimate stale even when each treatment’s response distribution is stable. The target therefore needs to be stated before describing an estimate as biased.

Deriving the Staleness Difference

The key calculation conditions on which participants were assigned treatment \(a\) and whose responses have resolved. Suppose the conditional mean model \(\mu_a(t)\) is appropriate and, given treatment and enrollment time, resolution does not select participants according to their response. This is a non-informative-resolution assumption for the response mean. These assumptions do not by themselves imply that each included response has mean \(\mu_a(T_i)\) conditional on the complete assignment vector, because later assignments may depend on earlier outcomes. For the conditional identity below, we additionally assume that, for each participant with \(A_i=a\) and \(R_i(\tau)=1\), \(E[Y_i\mid A,T,R]=\mu_a(T_i)\). This stronger condition may fail when later assignments depend on earlier outcomes.

Write \(S_a(\tau)=\{i:A_i=a,\ R_i(\tau)=1\}\) for the resolved participants assigned treatment \(a\), and let \(m_a\) be their number. By linearity of expectation, the conditional expected value of their sample mean is the average of their enrollment-time-specific means:

Subtracting the current target gives the conditional staleness difference:

This is an exact conditional identity under the additional conditional-mean assumption stated above. It shows that the difference is a weighted average of past-versus-current response-mean differences. If response means are constant, every term on the right is zero. If the mean rises over time, older resolved participants tend to pull the estimate below the current target; if it falls, they tend to pull it above. For a realized dataset, the observed estimate also includes sampling variation around this conditional expected value.

Linear Drift and the Average Enrollment Lag

A simple way to quantify the effect is to assume a linear response-mean trend over the relevant enrollment period. Let \(\beta_a\) be the change in the mean response per unit of calendar time, so that \(\mu_a(t)=\mu_a(\tau)+\beta_a(t-\tau)\). Define the average enrollment lag among resolved participants on treatment \(a\) as \(\bar{L}_a(\tau)=m_a^{-1}\sum_{i\in S_a(\tau)}(\tau-T_i)\), measured in units of time. Substitution into the conditional identity yields:

Thus the size of the difference depends on both the rate of drift and how old the resolved sample is on average. The units agree: response units per month multiplied by months gives response units. The expression is exact when the linear mean model holds across the represented enrollment times. For nonlinear trends, the weighted-average formula remains valid, but one cannot generally summarize it by a slope times the average lag alone.

Allocation Changes Can Make a Pooled Estimate Stale

The treatment-specific derivation does not by itself justify pooling across treatments. Suppose treatment-specific response means are stable, but the proportions assigned to each treatment change during the trial. A pooled resolved-sample mean then represents the historical treatment mix among resolved participants. If the target is the mean response under the current allocation mix, those two mixtures need not agree.

For two treatments, let \(q_a^{\mathrm{res}}\) be the proportion of resolved participants assigned treatment \(a\), and let \(q_a^{\mathrm{cur}}\) be the current allocation proportion, with the proportions for the two treatments summing to one. If arm-specific response means are stable and resolution is non-informative, the difference between the resolved pooled mean and the current allocation-mixture mean is:

This is a composition difference, not drift in either arm’s response distribution. The target matters: a pooled estimate may be appropriate for describing the historical resolved sample, yet inappropriate for describing the current allocation mixture. Arm-specific estimates and allocation-mixture estimates answer different questions.

Resolution Delays Determine Which Enrollment Times Are Represented

For outcomes that resolve after a delay, the resolved sample’s enrollment-time distribution depends on the delay distribution as well as enrollment and allocation. Let \(D_i\) denote time from enrollment to resolution. Under non-informative resolution within treatment and enrollment time, a participant enrolled at time \(t\) has probability \(F_a(\tau-t)\) of being resolved by \(\tau\), where \(F_a\) is the treatment-specific delay distribution function. Thus more recent participants are often less likely to contribute, although the exact pattern depends on the delay model and trial history.

With many participants, an expected resolved-sample mean can be approximated by weighting enrollment times according to enrollment intensity, treatment allocation, and the probability of resolution by the current time. This large-sample weighted average is a useful descriptive approximation; it is not generally an exact identity for the expectation of a random ratio with a random denominator. The exact conditional identity above applies once the resolved set is fixed.

Assumptions, Diagnostics, and Alternatives

A staleness calculation is only as useful as its target and assumptions. In particular, check the following before interpreting a resolved-only estimate:

• Specify the target. An arm-specific current mean, a historical resolved-sample mean, and a mean under the current allocation mix are distinct quantities.
• Check resolution selection. If, conditional on treatment and enrollment time, resolution is associated with the response, resolved outcomes may not represent the response mean even for that enrollment cohort. A trend-only staleness calculation is then incomplete.
• Check composition. Changes in allocation proportions or in participant characteristics can alter the resolved sample’s mix. If response depends on those characteristics, a change in mix can matter even when calendar-time means appear stable.
• Inspect age and cohort patterns. Record enrollment time, resolution status, treatment, and response where observed. Summarize the average lag and compare resolved counts across enrollment cohorts. A large backlog of recent participants signals that the resolved set may emphasize older cohorts.
• Separate systematic difference from sampling error. A realized estimate can lie above or below its conditional expected value. An apparent gap from the current target does not, by itself, establish a temporal trend.

Using only matured outcomes is transparent and avoids assigning modeled responses to unresolved participants, but it can provide old information and reduce the number of observations available. Waiting for more outcomes can reduce the lag at the cost of slower updates. Predictive updating or imputation can use information from pending outcomes sooner, but depends on a correctly specified response-and-resolution model. The earlier material on informative outcome delays and joint response–resolution modeling is relevant when resolution may depend on response. These approaches address different trade-offs; none makes the target definition or assumptions unnecessary.

Practical Interpretation

The central result is a decomposition, not a universal correction formula. For an arm-specific mean under non-informative resolution, the resolved estimate averages past means, and its conditional difference from the current mean is determined by the enrollment times represented. Under linear drift, that difference reduces to slope times average lag with a negative sign. For a pooled estimate, changing allocation proportions can create a further mixture difference even without response drift.

This perspective makes staleness measurable and helps identify when a more detailed model is needed. It also clarifies what a resolved-only estimate can support: without stable response means or an explicit model for time and resolution, it describes the observed resolved sample more directly than it describes the current response distribution.