Potential Outcomes and Bandit Feedback

Potential Outcomes in an Adaptive Trial

A patient can receive only one treatment in a trial, but the scientific question often concerns how that patient would have responded to each available treatment. Potential-outcomes notation makes this distinction explicit. In a response-adaptive trial, the assignment may depend on information accumulated from earlier patients; nevertheless, each patient’s outcome is observed only under the treatment actually assigned. This is the bandit-feedback structure.

Consider a trial with \(K\) treatments, indexed by \(k=1,\ldots,K\), and patients indexed by \(i=1,\ldots,n\). Let \(A_i\) denote the treatment assigned to patient \(i\). For each treatment \(k\), define \(Y_i(k)\) as the outcome that patient \(i\) would have under treatment \(k\). The realized observed outcome is \(Y_i\). These quantities are related by consistency: \(Y_i=Y_i(A_i)\). For a favorable binary endpoint, for example, \(Y_i(k)\) might be 1 if patient \(i\) would meet the response criterion under treatment \(k\), and 0 otherwise.

In a bandit formulation, the assigned treatment is an action and its observed patient outcome is the reward. The defining feedback constraint is patient-level: after assigning treatment \(A_i\), the trial learns \(Y_i(A_i)\), not the same patient’s outcomes under the other treatments. The algorithm may use observed outcomes to update subsequent assignment probabilities, but that updating does not reveal the earlier patient’s counterfactual outcomes.

Assumptions That Give the Notation Meaning

Potential outcomes are useful only when treatment and outcome are defined precisely. The following conditions are central to interpreting them in an adaptive clinical trial.

• Well-defined treatments and consistency. Each treatment label must correspond to a sufficiently specified intervention, and the recorded outcome under the treatment actually received must equal that treatment’s potential outcome. If “Treatment A” includes materially different versions, the notation needs to distinguish those versions or explain why they can be treated as equivalent.
• No interference between patients. Patient \(i\)’s potential outcome under a treatment is not changed by which treatment other patients receive. If one patient’s assignment affects another patient’s outcome, the potential outcomes must instead represent relevant treatment configurations or exposure conditions.
• Randomized assignment conditional on available history. Let \(H_i\) denote the information available just before assigning patient \(i\). The allocation rule specifies \(p_i(k)=P(A_i=k\mid H_i)\). Randomization conditional on this history means the assignment is generated by the specified rule, rather than chosen using unrecorded knowledge of that patient’s unobserved potential outcomes.
• Positivity for comparisons. To learn about treatment \(k\) for patients at a given history, that treatment must have positive probability of assignment there: \(p_i(k)>0\). A small probability is technically positive but can still make estimates highly variable.

For a finite set of trial patients, potential outcomes can be treated as fixed quantities and the randomization as the source of uncertainty. In a broader population model, potential outcomes may themselves be random across patients. Either way, the observed-data identity \(Y_i=Y_i(A_i)\) remains the same. The key point is not that the unassigned outcomes are zero or absent from reality; it is that the trial does not observe them for that patient.

Adaptive Assignment and the Observed Record

In response-adaptive randomization, assignment probabilities can change as outcomes from earlier patients accumulate. Write \(p_i(k)=P(A_i=k\mid H_i)\) for the probability of assigning patient \(i\) to treatment \(k\), given the recorded history immediately before that assignment. The history might include earlier assignments and outcomes, as well as prespecified patient characteristics. The probability applies to the current assignment; it is not the probability that the patient will respond.

A clear trial record therefore distinguishes the assignment, the probability used to generate it, and the outcome eventually observed. Recording \(p_i(k)\) for every patient and treatment makes it possible to audit the allocation mechanism and to identify when a treatment had little opportunity to be observed. If the outcome is recorded only after a follow-up period, the record should also distinguish “not yet observed” from “not observed because the patient received another treatment.”

Why Positive Assignment Probabilities Matter

Although counterfactual outcomes are not individually observed, random assignment with positive probabilities can support comparisons of treatment outcomes across patients. Adaptive probabilities need not be constant across the trial. They do need to be known or recoverable from the allocation record, and each treatment under comparison must retain positive probability at the relevant assignments.

A simple justification uses an inverse-probability contribution for one treatment \(k\). Let \(I(A_i=k)\) equal 1 when patient \(i\) is assigned to \(k\) and 0 otherwise. Conditional on the history \(H_i\) and on the patient’s potential outcomes, the allocation rule gives \(P(A_i=k\mid H_i)=p_i(k)\). When \(p_i(k)>0\), the expected weighted contribution is:

The first equality uses consistency: if \(A_i=k\), then \(Y_i=Y_i(k)\); if \(A_i\ne k\), the indicator is zero. Thus, across the random assignments allowed by the rule, the weighted contribution has conditional expectation equal to the potential outcome under \(k\). This is a justification for an estimator, not a claim that the unobserved outcome has been recovered for an individual patient.

For a finite cohort of \(n\) patients, the average of these contributions estimates the cohort’s mean potential outcome under \(k\). Its value in one realized trial can differ substantially from that mean, particularly when assignment probabilities are small. The calculation also depends on using the probability that applied to that patient, conditional on the history at assignment—not an overall probability averaged across the trial.

Diagnostics, Alternatives, and Limits

The principal diagnostics concern the assignment record and the definition of the outcomes. Check that the recorded probabilities sum to 1 across treatments for each assignment, agree with the allocation algorithm and available history, and are strictly positive for treatments needed in a comparison. Inspect their distribution: very small probabilities create large weights and unstable estimates. Also verify that the response window and outcome coding are consistent across arms. A delayed outcome is not yet known; a genuinely unobserved follow-up outcome is a separate missing-data problem; and an unassigned treatment outcome is a counterfactual that the trial design did not observe.

Potential outcomes apply to fixed-ratio randomized trials as well as adaptive trials. The distinctive feature of adaptation is that the conditional assignment probabilities may depend on earlier trial information, rather than being the same for every patient. In observational data, treatment may instead be selected by clinical judgment or other factors; a causal comparison then needs assumptions about confounding that are not supplied merely by writing potential outcomes. Bandit feedback describes what the trial observes after each assignment; it does not by itself guarantee unbiased estimation, adequate precision, or clinical validity.

No-interference assumptions may be inappropriate when treatment assignments affect shared resources or transmit effects between patients. Positivity also fails if the algorithm permanently removes an arm or assigns it probability zero. In that case, outcomes under the excluded treatment cannot be learned for histories where it is never assigned by relying on this trial’s randomization. Very low but nonzero probabilities are not a cure for practical lack of information: the resulting weighted estimates may be dominated by a few patients.