Random Sample Sizes and Finite-Sample Bias

Why a Random Arm Size Can Matter

In a response-adaptive trial, the allocation probability for a new participant can depend on outcomes already observed. Consequently, the final number assigned to an arm is random and may be related to that arm’s observed outcomes. The ordinary arm mean divides the arm’s outcome total by this random number. That random denominator can matter: an outcome may influence how likely it is that later observations are added to the same arm, changing the weight that outcome receives in the final mean.

The key distinction is not simply fixed versus random sample size. A random arm size that is independent of that arm’s outcomes need not bias its mean. The concern here is a finite-sample dependence: outcomes help determine future allocations, and therefore help determine the denominator used to average them. The derivation below isolates that mechanism in a small hypothetical trial.

For a clear calculation, suppose outcomes within arm A are independent Bernoulli random variables with success probability \(p\), so \(\mu_A=p\). A balanced initialization gives arm A one observation, \(Y_1\). A later participant is assigned to A with probability \(q(Y_1)\), where \(q(0)\) and \(q(1)\) are both strictly between 0 and 1. Thus the later assignment is randomized, and its probability uses past data. If the participant is assigned to A, their outcome \(Y_2\) is an independent Bernoulli random variable with success probability \(p\). Otherwise, arm A’s only observed outcome is \(Y_1\).

This simple setup assumes a fixed two-stage horizon, no missing outcomes, and the same Bernoulli response distribution for every participant assigned to A. The initial observation ensures that \(N_A\) is never zero. The second-stage assignment is randomized with positive probability for either arm. These assumptions let us focus on the effect of the outcome-dependent sample size, rather than on missingness or changes in the response distribution.

A Two-Stage Derivation

Write \(I=1\) if the second participant is assigned to A and \(I=0\) otherwise. Then \(N_A=1+I\). If \(I=1\), the ordinary mean is \((Y_1+Y_2)/2\); if \(I=0\), it is \(Y_1\). Conditional on the first outcome \(Y_1=y\), the probability of adding a second A observation is \(q(y)\), and that second outcome has mean \(p\). Therefore:

Taking expectations over \(Y_1\), whose mean is \(p\), gives the bias:

The last equality follows because \(E(Y_1)=p\): expanding the covariance gives \(E\{q(Y_1)Y_1\}-E\{q(Y_1)\}p\), the negative of the expression inside the expectation. In this two-stage model, the sign of the bias is therefore the opposite of the covariance between the first outcome and the chance of a later A assignment. If a success makes another A assignment more likely, that covariance is positive, and the ordinary mean is biased downward. A favorable first result can attract another observation; that observation regresses toward the population mean and dilutes the first result in the average.

This is a result for the stated two-stage setup, not a universal formula for every adaptive design. With more stages, multiple arms, or other response distributions, the magnitude and direction depend on the full allocation process. The general issue is still the relationship between observed outcomes and the random weights they receive in the final average.

Random Size Is Not Enough

It is important not to conclude that every random denominator creates bias. Suppose instead that arm A receives either one or two observations, with probability \(0.5\) for each sample size, and that this choice is made independently of all A outcomes. Each A outcome is still an independent Bernoulli variable with probability \(0.4\). Conditional on receiving one observation, the mean has expectation \(0.4\). Conditional on receiving two, its expectation is also \(0.4\). Averaging over the two possible sample sizes therefore gives \(E(\bar{Y}_A)=0.4\).

More generally, if \(N_A\) is independent of the A outcomes and is positive, and the outcomes are identically distributed with mean \(\mu_A\), then conditional on \(N_A=n\), the mean of \(n\) such outcomes has expectation \(\mu_A\). Averaging over the possible values of \(N_A\) preserves that expectation. The essential problem in the response-adaptive example is that \(N_A\) is not independent of the outcomes being averaged.

The Direction Depends on the Allocation Rule

An adaptive rule need not favor an arm after a favorable outcome. In the two-stage model, if the chance of another A assignment is lower after a success than after a failure, the covariance in the bias expression can be negative, making the bias positive. The following calculation reverses the allocation probabilities from the first example while keeping the response model unchanged.

What to Check and What the Result Does Not Say

For an actual design, begin by writing down the allocation rule at each stage and identifying which earlier outcomes it uses. Then trace how each observed outcome can affect the probability of later assignments to its own arm. This makes the possible dependence between outcomes and final sample sizes concrete. If the arm can receive no participants, also decide how the estimator is defined in that event; an ordinary arm mean is undefined when \(N_a=0\), and silently omitting such trials changes the quantity being averaged.

Repeated-trial simulation can help diagnose the mechanism. Simulate outcomes and assignments under the stated design, calculate the ordinary arm mean in every trial where it is defined, and compare its average across replications with the known \(\mu_a\). Also record the arm size and the allocation probabilities after different outcome histories. A pattern in which favorable outcomes are followed by higher same-arm allocation can be consistent with the downward mechanism in the two-stage derivation, though a simulation pattern alone does not establish a general result for a more complex design.

The main alternatives answer different questions. A fixed allocation chosen before outcomes avoids this particular feedback between earlier responses and later arm size, though other design or analysis issues may remain. An outcome-independent random sample size can also preserve the mean under the independent, identically distributed outcome assumptions stated above. A response-adaptive design instead uses past outcomes to guide later assignments, so its random sample sizes must be assessed under the actual allocation rule. Analyses using known assignment probabilities are another possible approach, but their estimands, assumptions, and finite-sample properties require separate treatment; the ordinary mean derivation here does not establish that any particular alternative is preferable.

The result is deliberately limited. It does not say that every response-adaptive arm mean is biased, nor that the bias must be large, nor that the same sign holds for every policy. It concerns the expectation of a finite-sample ordinary arm mean under repeated operation of a specified design. It also does not by itself determine bias in a treatment contrast, the validity of a confidence interval, or the operating characteristics of a test. Those questions require their own analyses.

Common Mistake

Wrong reasoning: “Every patient is randomized, so the ordinary mean within each arm must be unbiased.” Randomization of the next assignment protects the assignment from being chosen with certainty based on past outcomes, but it does not make the final arm size independent of those outcomes. In the example, the second assignment is randomized with probability \(0.2\) or \(0.8\), yet the expected A mean is \(0.425\), not \(0.5\).

Correction: Check whether the design makes the weight of an observed outcome in the final mean depend on that outcome. With a random denominator, the weight assigned to an observation is affected by the eventual arm size. Randomization of each decision does not, on its own, remove that dependence.