Bias in the Difference of Adaptive Arm Means

Why a difference can be biased even when each arm is randomized

A difference between treatment-arm means looks like a familiar estimator of a treatment effect. Under adaptive assignment, however, the number assigned to each arm can depend on earlier outcomes. Those outcomes also appear in the arm means. Consequently, the final denominators can be related to the observations being averaged, and the expected difference need not equal the treatment-effect estimand in a finite trial.

The point is not that every adaptive difference is biased. Its bias depends on the allocation rule, the outcome distributions, and the estimand. In particular, a design that treats the arms symmetrically can have zero expected contrast under a suitably symmetric null, while the same design can have nonzero bias under an alternative.

For clarity, examples below use independent participants whose potential outcomes have the same distribution within each arm, and an allocation probability at each time that may depend on observed history but is set before the current participant’s outcome is observed. Both arms are kept nonempty. These conditions define the examples; they do not imply that the arm means are unbiased. If later assignments respond to earlier outcomes, the final \(N_a\) can depend indirectly on outcomes already included in \(\bar{Y}_a\).

Where the finite-sample bias enters

Write \(\varepsilon_t(a)=Y_t(a)-\mu_a\). Subtracting the estimand from the observed contrast gives the exact identity

The identity separates the two arm-specific contributions, but it does not make either expectation zero. Although \(A_t\) is decided before \(Y_t(A_t)\) is observed, \(N_a\) includes assignments made later. Those later assignments may depend on \(Y_t(A_t)\). Thus the ratio involving the final denominator can be associated with the current centered outcome.

With fixed arm counts, or with assignments independent of all outcomes, the denominator does not select observations according to their outcomes. Under the independent-outcome model above, each arm mean then has expectation \(\mu_a\), so their difference has expectation \(\Delta\). Adaptation changes this argument: it can make inclusion in a particular arm’s final average depend on previous responses. Even when that changes both arm means, their biases may cancel in the contrast—or may not.

A small Bernoulli design that exposes the bias

Consider a hypothetical three-participant trial. The first participant is assigned to arm 1 and the second to arm 0. Their outcomes are \(X\sim\mathrm{Bernoulli}(p_1)\) and \(Z\sim\mathrm{Bernoulli}(p_0)\), independently. The third participant is assigned to arm 1 with probability \(q(X)\), where \(q(1)=q_1\) and \(q(0)=q_0\). Otherwise the third participant is assigned to arm 0. Assume \(0<q_0,q_1<1\), and that the third outcome is an independent Bernoulli outcome from its assigned arm. Both arm means are defined for every assignment because the first two allocations put one participant in each arm.

If the third participant goes to arm 0, the expected contrast conditional on \(X,Z\) is \(X-p_0\): the control mean averages \(Z\) and a fresh outcome with mean \(p_0\). If instead the third participant goes to arm 1, the conditional expected contrast is \((X+p_1)/2-Z\). The difference between these two conditional expectations is \((p_1-X)/2+(p_0-Z)/2\). Since assignment probability depends on \(X\), but not on \(Z\), averaging over \(Z\) removes its centered contribution. Therefore

The final equality follows by splitting over \(X=1\) and \(X=0\): \(E[q(X)(p_1-X)]=p_1q_1(p_1-1)+(1-p_1)q_0p_1 =p_1(1-p_1)(q_0-q_1)\). This formula is exact for this three-participant design, not an approximation.

What null symmetry does—and does not—guarantee

A null hypothesis of equal means, \(H_0:\mu_1=\mu_0\), is not by itself enough to ensure that an adaptive difference has expectation zero. The first worked example has equal Bernoulli means, and therefore identical outcome distributions, but the allocation rule favors arm 1 after a success in that arm and control after a failure. The rule is not unchanged by swapping the arm labels, so equal distributions do not force a zero expected contrast.

A stronger design symmetry can give a useful null result. Suppose the two arms have identical outcome distributions, both remain represented, and the allocation rule and estimator are invariant to swapping arm labels. Swapping the labels changes the sign of \(\bar{Y}_1-\bar{Y}_0\) but leaves its distribution unchanged. Hence its expectation equals its own negative and must be zero. This is a symmetry argument about the complete design, not a consequence of equal means alone. In particular, if the outcome distributions differ despite having equal means, identical-distribution symmetry is unavailable.

Under an alternative, even label-symmetric allocation does not generally protect the difference from bias. Treatment and control outcomes now come from different distributions, so swapping labels is not a symmetry of the data-generating process. A small symmetric-rule calculation makes the distinction concrete.

Comparison, diagnostics, and limitations

The ordinary difference in adaptive arm means is transparent and targets a familiar contrast, but its finite-sample expectation depends on the interaction between outcomes and future allocation. A fixed-count randomized design avoids that particular random-denominator mechanism under the independent-outcome model: the count in each arm is set without using responses, and the arm means are unbiased for their respective means. An outcome-independent randomized allocation can also leave counts random while retaining unbiased arm means, provided assignment and counts are independent of outcomes and both means are defined. These are benchmarks, not claims that fixed allocation is preferable for every clinical objective.

Inverse-probability weighting is a different estimator family intended to address outcome-dependent assignment probabilities; it is not the same quantity as the unweighted difference studied here. Its construction and properties require separate conditions. Likewise, a small bias in this contrast does not establish good uncertainty quantification or valid testing; those are distinct operating characteristics.

In practice, the source code or written allocation rule should be inspected for whether assignment probabilities depend on earlier outcomes, whether the rule treats arm labels symmetrically, and whether both arm counts are guaranteed positive. Repeated simulations under specified null and alternative outcome models can estimate the design-specific bias by averaging \(\bar{Y}_1-\bar{Y}_0-\Delta\) across trials. Such a simulation should preserve the full sequential assignment mechanism rather than condition on a final allocation pattern.

A finite-sample bias calculation is specific to the model and policy. The Bernoulli formula above cannot simply be carried over to continuous outcomes, different histories, or other sample sizes. It also says nothing by itself about the variance of the estimator or whether an inferential procedure accounts correctly for adaptation. The key conclusion is narrower: the unweighted arm-mean difference is not automatically unbiased merely because each participant’s current assignment is randomized.