Outcome-Dependent Sampling And Estimator Bias

Why the Allocation Rule Can Affect an Estimate

In a response-adaptive trial, an observed outcome can influence the probability of assigning a later participant to each treatment. This can change more than the allocation ratio: it can change which earlier responses are represented in each arm mean and how many later observations enter each mean. The ordinary difference in arm means may therefore be biased for the treatment effect, even when the allocation probabilities are known and every treatment remains possible.

The issue is not that randomization has stopped. Rather, the estimator gives equal weight to the observed participants in each arm, while the probability of including a participant in an arm may depend on the preceding outcomes. To see the distinction precisely, first specify the target and the sampling rule.

Write \(H_{i-1}\) for the assignment and outcome history available before participant \(i\) is randomized. The allocation probability for A is \(\pi_i=P(A_i=1\mid H_{i-1})\); the probability for B is \(1-\pi_i\). A typical response-adaptive rule makes \(\pi_i\) depend on outcomes in \(H_{i-1}\). For the probability-weighted result below, assume the potential outcomes are fixed for the finite-sample argument, the randomization probabilities are correctly recorded, and \(0<\pi_i<1\) at every assignment. These conditions ensure each participant has a positive chance of assignment to either treatment. They do not, by themselves, make the naive difference unbiased.

A Finite-Sample Calculation

Consider a hypothetical trial with three participants. The first two are randomized so that one receives A and one receives B; a fair coin determines which of the two gets A. Outcomes are independent Bernoulli variables, with success probabilities \(p_A=0.6\) and \(p_B=0.4\), and those probabilities are stable across participants. After seeing the first A outcome, the trial assigns participant 3 to A with probability \(q_1=0.8\) if that outcome was a success, and with probability \(q_0=0.2\) if it was a failure. The assignment is still randomized in either case.

The calculation is finite-sample: it averages over the first A response, the adaptive assignment, and the third outcome. It does not claim that every realized difference will be below \(0.200\). The source of the bias is visible in the conditional calculations: the final arm sizes depend on \(Y_A\), so the final means do not combine outcomes in a way that preserves the target contrast on average.

Why the Allocation Probabilities Matter

For a fixed set of potential outcomes, a probability-weighted estimate of each arm’s finite-sample mean can use the chance that each participant was assigned to that arm. Define the weighted estimates of the finite-sample means by \(\hat{\mu}_A^{HT}=n^{-1}\sum_i A_iY_i/\pi_i\) and \(\hat{\mu}_B^{HT}=n^{-1}\sum_i (1-A_i)Y_i/(1-\pi_i)\). Their difference is a probability-weighted estimate of \(\tau\). The weights are larger for outcomes observed under a treatment that had a smaller assignment probability.

This justification does not say that the weighted estimate will be close to the target in every trial. It establishes unbiasedness across repetitions of the randomization under the stated conditions. In particular, when an allocation probability is very small, an observed outcome can receive a large weight, making a single estimate highly variable.

In an adaptive trial, the same argument uses the probability conditional on the history at the time of each assignment. The probability must be attached to the participant and treatment at that decision, not replaced by the final overall allocation proportion. An overall proportion, such as “40% were assigned to A,” does not generally capture the different probabilities participants faced as the outcome history changed.

Conditions, Diagnostics, and Alternatives

The probability-weighted justification depends on more than recording numbers in a table. The randomization probability must be the probability actually used, conditional on the information available at assignment. Each relevant probability must be positive; if a treatment is impossible for a participant, the missing potential outcome for that participant cannot be recovered by weighting. The outcome must be recorded consistently, and the target population or finite sample must be specified. If outcomes, eligibility, or follow-up are missing in a way not accounted for by the assignment probabilities, assignment weighting alone does not correct that additional selection.

Useful diagnostics focus on how allocation and the observed history are connected:

• Tabulate or plot the recorded \(\pi_i\) values over enrollment order and alongside the response history used by the rule. Probabilities that do not change when the rule says they should, or change before the relevant outcome is available, can indicate implementation or recording errors.
• Check the smallest and largest probabilities and the resulting weights \(1/\pi_i\) and \(1/(1-\pi_i)\). Probabilities close to zero or one mean that some weighted contributions can dominate.
• Compare the naive arm means with probability-weighted estimates, while recognizing that a difference between them is a warning to investigate, not proof that either estimate is correct.
• Inspect arm sizes and the timing of observed responses. A final arm size that is associated with earlier outcomes is a direct sign that ordinary equal weighting may not reflect the target estimand.

The main alternative is fixed randomization with probabilities that do not depend on observed outcomes. Under suitable randomization and a clearly defined estimand, that design avoids this particular outcome-driven sampling mechanism. Response-adaptive randomization can change the allocation probabilities as information accumulates, but it requires attention to those probabilities when estimating treatment means. Probability weighting addresses unequal chances of assignment under its assumptions; it is not a remedy for every design or analysis problem.

Common Mistake and Limitations

Common mistake: “Randomization was used, so the final difference in arm means must be unbiased.” Randomization at each step does not imply that the ratio of observed outcomes to the random arm size is unbiased when that arm size depends on earlier responses. The correction is to examine the sequential allocation probabilities and the estimator together. In the finite-sample example, all assignments remained randomized, but the expected naive difference was \(0.128\), not the target \(0.200\).

A second mistake is to treat a weighted estimate as automatically superior because it has an unbiasedness argument. Inverse-probability weights can be unstable when probabilities are extreme, and a weighted estimate can be far from its target in a particular trial. If the allocation probabilities are misspecified, omitted, or recorded only as a final allocation ratio, the justification does not apply. These limitations concern the validity and precision of the estimate; weighting does not retroactively change the design.