Conditional Unbiasedness Under Delayed Observation

Why Delayed Observation Changes the Unbiasedness Question

At a decision time, an adaptive trial may have assigned many participants but received only some of their outcomes. The observed outcomes are not necessarily a simple random subset: assignment probabilities can vary with the trial’s past, and response or resolution times can vary by treatment and participant. Consequently, an average of the outcomes currently available need not estimate the mean outcome in the intended cohort.

The relevant question is not merely whether the assignment mechanism is randomized. It is whether each participant’s contribution, after accounting for both assignment and observation, has the right expectation. The conditioning must also be chosen carefully. A participant’s contribution can have a valid expectation conditional on that participant’s own pre-assignment history, while the sum of contributions need not have the same expectation conditional on the entire sequence of adaptive histories.

Target, Assignment, and Observation

Fix a cohort of $N$ participants and a decision time $\tau$. For participant $i$, let $X_i$ denote baseline information and $H_i$ the information available immediately before that participant is assigned. Let $A_i$ be the assigned treatment, and let $Y_i(a)$ be the potential outcome under treatment $a$. The arm-specific finite-cohort target is

\mu_a=\frac{1}{N}\sum_{i=1}^{N}Y_i(a).

Define $D_i(a;\tau)=1$ if the outcome under treatment $a$ would be available by time $\tau$, and $D_i(a;\tau)=0$ otherwise. This indicator describes observation by the cutoff; a pending outcome is not automatically a missing outcome forever. By consistency, the observed outcome is $Y_i(A_i)$ when $D_i(A_i;\tau)=1$.

Definition: Let $\pi_i(a\mid H_i,X_i)=P(A_i=a\mid H_i,X_i)$ be the design assignment probability. Let $q_i(a\mid H_i,X_i,Y_i(a))$ be the probability that the outcome is available by $\tau$, conditional on treatment $a$, the stated information, and the potential outcome. The joint inclusion probability for an observed outcome in arm $a$ is $$ g_i(a)=\pi_i(a\mid H_i,X_i)\,q_i(a\mid H_i,X_i,Y_i(a)), $$ when the observation probability is conditional on the stated variables after assignment. The inverse-probability contribution is $$ Z_i(a)=\frac{\mathbf{1}\{A_i=a\}D_i(a;\tau)Y_i(a)}{g_i(a)}. $$

The product form follows from the sequential factorization: assignment to $a$ occurs with probability $\pi_i$, and, given that assignment and the variables in the observation model, resolution by $\tau$ occurs with probability $q_i$. More generally, use the correctly specified joint probability $g_i(a)=P(A_i=a,D_i(a;\tau)=1\mid\cdots)$ if the factorization does not apply. The estimator of the cohort mean is the Horvitz–Thompson form

\widehat{\mu}^{\,HT}_a(\tau)=\frac{1}{N}\sum_{i=1}^{N}Z_i(a).

For this estimator to be defined and unbiased by this argument, every relevant participant must have positive probability of assignment to arm $a$ and observation by $\tau$: $g_i(a)>0$. The assignment probabilities must be the probabilities actually used by the randomization procedure. The observation probabilities must also correctly represent the response and delay process. Potential outcomes are treated as fixed for the design-based expectation; the expectation is over assignment and observation randomness.

Deriving the Unbiasedness Condition

First consider a setting where $q_i$ does not depend on the unobserved outcome once $H_i$, $X_i$, and treatment are given. Conditional on participant $i$’s own pre-assignment information and fixed potential outcomes, the probability that the weighted contribution is nonzero is $\pi_iq_i$. Its nonzero value is $Y_i(a)/(\pi_iq_i)$. Therefore,

E\!\left[Z_i(a)\mid H_i,X_i,\{Y_j(b)\}\right] =(\pi_iq_i)\frac{Y_i(a)}{\pi_iq_i} =Y_i(a).

Taking expectations over each participant’s pre-assignment history and then summing gives the finite-cohort result:

E\!\left[\widehat{\mu}^{\,HT}_a(\tau)\mid \{Y_j(b)\}\right] =\frac{1}{N}\sum_{i=1}^{N}E[Z_i(a)\mid\{Y_j(b)\}] =\frac{1}{N}\sum_{i=1}^{N}Y_i(a) =\mu_a.

This is unconditional design unbiasedness given fixed potential outcomes. The individual conditional expectation is valid given $H_i$ and $X_i$, but it does not justify conditioning the entire estimator on every participant’s pre-assignment history. A later history can contain earlier assignments, thereby revealing earlier terms in the estimator.

If observation depends on the potential outcome, the same algebra works in theory when the correct $q_i(a\mid H_i,X_i,Y_i(a))$ is used and is positive:

E[Z_i(a)\mid H_i,X_i,Y_i(a)] =\pi_iq_i(a\mid H_i,X_i,Y_i(a)) \frac{Y_i(a)}{\pi_iq_i(a\mid H_i,X_i,Y_i(a))} =Y_i(a).

However, outcome-dependent observation probabilities may be difficult to know or estimate: for a pending participant, the outcome needed to select the appropriate probability is not yet observed. A practical sufficient condition is that, conditional on measured information available for all participants, the chance of resolution by $\tau$ does not further depend on the unobserved outcome, and that this conditional probability is known or correctly specified. This is a condition on the observation process, not a consequence of randomization alone.

Worked Example: Weighting for Both Assignment and Resolution

Suppose a hypothetical cohort has three participants, with fixed potential outcomes under treatment $a$ equal to 4, 6, and 8 points. Given each participant’s own pre-assignment history, the assignment probabilities for $a$ are 0.50, 0.60, and 0.40. The corresponding probabilities of having the outcome by $\tau$, conditional on assignment to $a$ and measured history, are 0.80, 0.50, and 1.00. Assume these probabilities are correct, positive, and do not depend further on the unobserved outcome.

For participant 1, the probability of an observed outcome in arm $a$ is $0.50(0.80)=0.40$. The weighted contribution is $4/0.40=10$ if observed and zero otherwise, so its conditional expectation is $0.40(10)=4$.

For participant 2, the joint probability is $0.60(0.50)=0.30$, and the observed contribution is $6/0.30=20$. Its expected contribution is $0.30(20)=6$. For participant 3, the joint probability is $0.40(1.00)=0.40$, and its expected contribution is $0.40(8/0.40)=8$.

Thus the expected weighted total is $4+6+8=18$, and $E[\widehat{\mu}^{\,HT}_a(\tau)\mid\{Y_i(a)\}]=18/3=6$ points, the target mean. A realized estimate need not equal 6: weighting establishes the expectation, not exact recovery in each trial.

Why the Full-History Conditional Claim Fails

Worked Example: Adaptive Histories Reveal Earlier Assignments

Consider two participants with $Y_1(a)=Y_2(a)=1$, no delayed outcomes, and target mean $\mu_a=1$. Participant 1 has probability 0.50 of assignment to $a$. Participant 2’s history $H_2$ includes participant 1’s assignment; suppose participant 2 has probability 0.80 of assignment to $a$ if participant 1 received $a$, and probability 0.20 otherwise. All assignment probabilities are positive and correctly recorded.

The Horvitz–Thompson estimate is $\widehat{\mu}^{\,HT}_a=\tfrac12\{\mathbf{1}(A_1=a)/0.50+\mathbf{1}(A_2=a)/\pi_2(a\mid H_2)\}$. For each participant, the expected weighted contribution given that participant’s own pre-assignment history is 1. Therefore its unconditional expectation is $(1+1)/2=1$.

But $H_2$ reveals $A_1$. If $A_1=a$, the first contribution is known to be $1/0.50=2$; participant 2’s conditional expected weighted contribution is still 1. Hence $E[\widehat{\mu}^{\,HT}_a\mid H_2,A_1=a]=(2+1)/2=1.5$. If $A_1\ne a$, that first contribution is zero and the conditional expectation is $(0+1)/2=0.5$. Neither equals 1. Unconditional unbiasedness is intact; conditioning on the later history changes the expectation because that history discloses an earlier assignment.

Selection Among Outcomes Available by the Cutoff

Worked Example: A Complete-Case Mean Under Outcome-Dependent Delay

In a hypothetical cohort of 100 participants assigned to one arm, suppose 81 observed outcomes by $\tau$ equal 0 points, one equals 10 points, and nine equal 5 points. The other nine outcomes are still pending at $\tau$; suppose their eventual values are 10 points. The complete-case mean is

$(81(0)+1(10)+9(5))/91=55/91=0.6044$ points (to four significant figures). The target cohort mean, including the pending outcomes, is

$(81(0)+1(10)+9(5)+9(10))/100=145/100=1.45$ points.

The observed-outcome average is therefore not the target mean in this hypothetical cohort. Here, some outcomes equal to 10 remain pending, so observation status is related to outcome. Merely having randomized treatment does not make the observed cases representative of all 100 participants. Moreover, the pending 10-point outcomes have zero probability of observation by $\tau$ in this illustration, so positivity fails for that stratum; no finite inverse-probability weight can recover their contribution from observations by this cutoff.

Assumptions, Diagnostics, and Alternatives

The weighting result is useful only when its conditions are credible. In an adaptive trial, inspect the assignment record to verify that each $\pi_i$ is the probability used immediately before assignment, not a later or averaged probability. For delayed outcomes, define precisely what counts as available by $\tau$, and distinguish an unresolved outcome from permanent nonresponse or dropout. Check resolution rates by treatment and by measured predictors of delay, such as follow-up schedule or enrollment cohort. Such checks can reveal discrepancies, though they cannot prove that delay is independent of an unobserved outcome.

Small values of $\pi_iq_i$ create large weights. This can make an unbiased estimator highly variable and sensitive to errors in the probabilities. Report the range of joint probabilities and weights, and inspect how much each observed participant contributes. If some relevant $g_i(a)$ is zero, the target mean is not identified by this weighting argument at that cutoff without additional assumptions or information. Weight truncation may reduce variability, but generally changes the estimator and removes the exact unbiasedness guarantee.

Approach	What it uses	Main qualification
Complete-case mean	Outcomes available by $\tau$	Can be biased when availability selects participants with different outcomes.
Inverse-probability weighting	Observed outcomes and joint assignment-observation probabilities	Requires correct probabilities and positivity; small probabilities can make estimates unstable.
Imputation or predictive updating	Observed information and a model for pending outcomes	Depends on the predictive model; an imputed value is not an observed outcome.

Weighting and imputation answer related but distinct questions. Weighting corrects observed contributions under a specified inclusion mechanism; prediction fills in or uses information about pending outcomes under a model. Neither approach makes informative delays harmless without assumptions. In particular, a model for assignment probabilities alone does not correct selective observation.

Common Mistakes

Conditioning too broadly. The termwise calculation conditions on $H_i$, not on the complete sequence of histories. Later histories may reveal earlier assignments, so do not infer that the whole estimator is unbiased conditional on all histories.
Using only assignment weights. If outcomes are not all observed by $\tau$, dividing by $\pi_i$ alone leaves the observation process unaccounted for. The relevant probability is the joint assignment-and-observation probability.
Treating pending as permanently missing. A pending outcome is unavailable at the cutoff, but may resolve later. State the cutoff and the observation event explicitly.
Assuming randomization prevents selection bias. Randomization governs treatment assignment; it does not ensure that outcome availability is unrelated to outcome or participant characteristics.
Ignoring positivity. If an outcome stratum has no chance to be observed by the decision time, an inverse weight cannot reconstruct its contribution from the available outcomes.

Key takeaway: With correct positive joint probabilities for assignment and observation, each participant’s weighted contribution has the correct expectation given that participant’s own pre-assignment information. Summing yields an estimator unbiased over the assignment and observation process, conditional on fixed potential outcomes. This does not generally imply unbiasedness conditional on the full adaptive history.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

If assignment probability is 0.5 and observation-by-cutoff probability is 0.6, what joint probability belongs in the weight when the stated sequential factorization holds?
Why does a correct termwise conditional expectation not establish unbiasedness given the entire sequence of adaptive histories?
What happens to the inverse-probability argument if a relevant participant has zero probability of observation by the decision time?
When can an outcome-dependent observation probability still give a theoretical unbiasedness result, and what practical difficulty can arise?

Solutions. (1) The joint probability is $0.5(0.6)=0.30$, so the observed contribution is weighted by its reciprocal, $1/0.30$. (2) A later history can reveal earlier assignments, making earlier weighted contributions known after conditioning; the unconditional sum argument does not survive that conditioning. (3) Positivity fails, so the outcome’s contribution cannot be recovered from observations by this cutoff using a finite inverse weight. (4) The correct positive probability conditional on the potential outcome gives the theoretical termwise identity, but for a pending participant the outcome needed to determine that probability is not yet known; a usable analysis needs a justified observation model based on available information or other assumptions.

Approach	What it uses	Main qualification
Complete-case mean	Outcomes available by \(\tau\)	Can be biased when availability selects participants with different outcomes.
Inverse-probability weighting	Observed outcomes and joint assignment-observation probabilities	Requires correct probabilities and positivity; small probabilities can make estimates unstable.
Imputation or predictive updating	Observed information and a model for pending outcomes	Depends on the predictive model; an imputed value is not an observed outcome.