Sensitivity to Response-Rate Estimation Noise

Why Estimation Noise Matters

A response-adaptive rule does not usually know each treatment arm’s response rate. It estimates rates from observed outcomes and uses those estimates to choose later randomization probabilities. Even when the estimation procedure is unbiased or nearly unbiased, its estimates vary from one trial to another. That variation is passed into the allocation rule: an arm can receive more or fewer later assignments simply because its early estimate happened to be high or low.

This tutorial isolates that mechanism with a simple batch design. The calculations are not universal operating characteristics for every adaptive algorithm. They show how to trace estimation noise through a specified rule, and what must be checked before applying the same reasoning to a different design.

Definition: Suppose outcomes used for allocation are binary, with response probabilities $p_A$ and $p_B$ on arms A and B. From independent pilot samples of sizes $n_A$ and $n_B$, let $\hat p_A$ and $\hat p_B$ be the observed response proportions. The plug-in allocation rule considered here assigns each person in the next block to A with probability $$ \hat q_A=\frac{\hat p_A}{\hat p_A+\hat p_B}, $$ and to B with probability $1-\hat q_A$. If both observed proportions are zero, define $\hat q_A=1/2$. This is one illustrative rule, not a general requirement of response-adaptive randomization.

The rule allocates in proportion to estimated response rates: if one estimated rate is twice the other, its estimated allocation probability is twice as large. The pilot outcomes must be independent Bernoulli observations within arms, with fixed probabilities $p_A,p_B$, for the binomial variance calculations below. The later block is randomized independently conditional on the pilot data, using the resulting fixed $\hat q_A$. This batch setup makes the two sources of variation distinguishable: uncertainty in the pilot-based probability and random assignment within the later block.

From Rate Noise to Probability Noise

Even if the true rates are fixed, $\hat q_A$ varies because it is calculated from random pilot outcomes. When $p_A+p_B>0$ and both pilot samples are sufficiently informative, a first-order Taylor approximation gives its approximate variance. For $f(a,b)=a/(a+b)$, the partial derivatives are $b/(a+b)^2$ and $-a/(a+b)^2$. Independent binomial samples have $\operatorname{Var}(\hat p_A)=p_A(1-p_A)/n_A$ and $\operatorname{Var}(\hat p_B)=p_B(1-p_B)/n_B$. Substituting these into the delta method yields

\operatorname{Var}(\hat q_A)\ \approx\ \left(\frac{p_B}{(p_A+p_B)^2}\right)^2 \frac{p_A(1-p_A)}{n_A} + \left(\frac{p_A}{(p_A+p_B)^2}\right)^2 \frac{p_B(1-p_B)}{n_B}.

This approximation is local: it works best when the pilot estimates are concentrated near positive rates and the denominator is not often near zero. It can be poor with very small samples or rare responses. The ratio is nonlinear, so a small variance does not establish that the ratio is exactly unbiased.

Worked Example: A Noisy Pilot Changes the Allocation Probability

Suppose the hypothetical true response probabilities are $p_A=0.40$ and $p_B=0.60$. Each arm has 20 pilot observations. In one hypothetical trial, arm A has 6 responses and arm B has 14.

The observed proportions are $\hat p_A=6/20=0.30$ and $\hat p_B=14/20=0.70$. The plug-in rule therefore gives $$ \hat q_A=\frac{0.30}{0.30+0.70}=0.30. $$

If the next block has 100 people, the expected number assigned to A conditional on this pilot is $100(0.30)=30$; the corresponding expected number for B is 70. If the rule used the true rates instead, its probability for A would be $0.40/(0.40+0.60)=0.40$, corresponding to expected counts of 40 and 60. The ten-person difference in these expected counts is not an allocation error within the next block: it is the consequence of this pilot’s noisy rate estimates changing the probability used for that block.

How Probability Noise Accumulates in Arm Counts

Let $M$ be the size of the later block and let $K$ be the number assigned to A. Conditional on the pilot-based probability, $K\sim\operatorname{Binomial}(M,\hat q_A)$. Thus the conditional expected count is $M\hat q_A$ and its conditional variance is $M\hat q_A(1-\hat q_A)$. Across repeated trials, however, $\hat q_A$ also varies. The law of total variance separates these contributions:

\begin{aligned} \operatorname{Var}(K) &= \operatorname{E}\{\operatorname{Var}(K\mid\hat q_A)\} + \operatorname{Var}\{\operatorname{E}(K\mid\hat q_A)\}\\ &=M\operatorname{E}\{\hat q_A(1-\hat q_A)\} +M^2\operatorname{Var}(\hat q_A). \end{aligned}

The first term is the random variation from assigning people within a block at a given probability. The second is the additional variation because that probability itself changes across pilots. Its $M^2$ multiplier matters: when one noisy estimate sets the probability for a large block, the same probability error affects many assignments. This formula is exact under the stated batch-randomization model; approximating its terms requires estimating the distribution of $\hat q_A$.

Worked Example: Approximate Allocation-Count Variability

Use the hypothetical rates $p_A=0.40$, $p_B=0.60$, with 20 independent pilot observations per arm, and a later block of $M=100$. The true-rate allocation probability is $q_A=0.40$. At this point, assume the delta-method approximation is adequate.

The estimated proportions each have variance $0.40(0.60)/20=0.012$. The derivatives of $f$ at $(0.40,0.60)$ are $0.60$ and $-0.40$. Therefore, $$ \operatorname{Var}(\hat q_A)\approx (0.60)^2(0.012)+(-0.40)^2(0.012) =0.00432+0.00192=0.00624. $$ The approximate standard deviation of the allocation probability is $\sqrt{0.00624}=0.0790$.

For a moment approximation, take $\operatorname{E}(\hat q_A)\approx0.40$ and $\operatorname{Var}(\hat q_A)\approx0.00624$. Then $$ \operatorname{E}\{\hat q_A(1-\hat q_A)\} \approx 0.40(1-0.40)-0.00624=0.23376. $$ Consequently, $$ \operatorname{Var}(K)\approx100(0.23376)+100^2(0.00624) =23.376+62.4=85.776. $$ The approximate standard deviation of $K$ is $\sqrt{85.776}=9.26$ people. If the probability were fixed at the known value $0.40$, the count variance would instead be $100(0.40)(0.60)=24$, with standard deviation $4.90$. The difference illustrates how pilot estimation noise can add substantial count variability. These are approximations, not an exact finite-sample distribution for the ratio estimator.

The all-zero denominator has a defined fallback in this rule. Under the stated true rates and independent pilot outcomes, its probability is $$ (1-0.40)^{20}(1-0.60)^{20} =(0.60^{20})(0.40^{20})=(0.24)^{20} \approx4.02\times10^{-13}. $$ It is negligible for this numerical example, but the fallback remains necessary to define the rule for every possible pilot dataset.

Consequences for Treatment-Effect Precision

Allocation variability is not itself a treatment-effect variance, but it changes the sample sizes that determine precision. To isolate this link, suppose the response is also the binary endpoint of interest, pilot data are used only to set allocation probabilities, and the treatment contrast is estimated from the later block. Conditional on its arm counts, the variance of the difference in sample proportions is $$ \operatorname{Var}(\hat p_A-\hat p_B\mid n_A,n_B) =\frac{p_A(1-p_A)}{n_A}+\frac{p_B(1-p_B)}{n_B}. $$ This calculation assumes independent outcomes in the later block and fixed arm-specific response probabilities. If the response being used to guide allocation differs from the primary endpoint, or if outcome variances differ by arm, the corresponding endpoint variances must replace these terms. The earlier material on variance estimation for adaptive treatment contrasts addresses inference under adaptive assignments; the calculation here emphasizes how arm counts affect conditional precision.

Worked Example: Comparing Precision at Two Allocation Splits

Continue with hypothetical response probabilities $p_A=0.40$ and $p_B=0.60$, so each arm’s Bernoulli outcome variance is $0.24$. For a later block of 100 people, compare a realized 30/70 split, consistent with the noisy pilot in the first example, with a 40/60 split, consistent with the true-rate probability.

At 30 and 70 observations, the conditional variance of the difference in sample proportions is $$ \frac{0.24}{30}+\frac{0.24}{70} =0.008000+0.003429=0.01143. $$ The conditional standard error is $\sqrt{0.01143}=0.1069$, in response-proportion units.

At 40 and 60 observations, the variance is $$ \frac{0.24}{40}+\frac{0.24}{60} =0.006000+0.004000=0.01000, $$ and the standard error is $0.1000$. For these particular realized counts and equal arm variances, the 30/70 allocation has $0.01143/0.01000-1=14.3\%$ greater variance and about $6.9\%$ greater standard error. This comparison is conditional on the displayed counts; it does not by itself calculate the average precision over the random counts generated by the adaptive design.

Assumptions, Diagnostics, and Alternatives

The calculations rely on a deliberately transparent setup. Pilot responses must be representative of the response probabilities relevant to the later block; outcomes within each arm are modeled as independent Bernoulli trials; and the allocation probability is held fixed within that block. If rates change over calendar time, eligibility changes, or outcomes arrive selectively, the pilot estimates may not represent later responses. If probabilities are updated after every outcome instead of once per block, later assignments are dependent through the evolving estimates, and the simple binomial count formula does not describe the whole trial.

Useful diagnostics include examining the pilot sample sizes and response counts, plotting the implied allocation probability over plausible pilot outcomes, and repeating the calculation across a range of response probabilities and pilot sizes. For a design evaluation, simulation can reproduce the exact estimation and allocation steps, including any fallback rule, and record the distribution of arm counts and treatment-contrast standard errors. A single expected allocation probability is not enough: two procedures with the same mean probability can have different variability across trials.

The main comparison is between three allocation references. A fixed allocation probability has no pilot-estimation component, although random assignment still causes count variation. An oracle rule using known response rates has the same limitation in practice—it is a benchmark, not an implementable procedure. A plug-in response-adaptive rule uses estimated rates and can change expected patient allocation, but brings estimation noise into the randomization process. Which design is preferable depends on the objective and assumptions; reducing count imbalance alone does not establish better patient outcomes or more precise treatment-effect estimates.

Common pitfall: “The estimated allocation probability is close to the target on average, so estimation noise cannot matter much.” This ignores the $M^2\operatorname{Var}(\hat q_A)$ contribution to later count variance and the reciprocal dependence of contrast variance on arm sizes. Instead, assess both the distribution of the estimated probability and the resulting arm-count and precision distributions.

The delta approximation also has limits when estimated rates are near zero, when the denominator can be small with meaningful probability, or when pilot samples are sparse. In those settings, report the exact rule and examine its behavior by simulation rather than treating a first-order variance as reliable. This tutorial does not specify how to choose a prior or modify the response-rate estimator: those choices change the distribution of the probability that is passed into the allocation rule.

Check Your Understanding

Answer each question, then compare with the solutions below.

If the estimated rates are 0.20 on A and 0.80 on B, what allocation probability does the stated rule assign to A?
In the batch model, which term in $\operatorname{Var}(K)$ represents uncertainty in the pilot-based probability, and why does it grow quadratically with block size?
For a 100-person later block, calculate the conditional variance and standard error of a difference in proportions with 50 observations per arm when both response probabilities are 0.50.
Name one condition under which the delta-method approximation for $\operatorname{Var}(\hat q_A)$ may be unreliable.

Solutions. (1) $\hat q_A=0.20/(0.20+0.80)=0.20$, so the rule assigns 20% to A. (2) The term is $M^2\operatorname{Var}(\hat q_A)$. The random probability changes the expected count by $M$ times its change, so the variance of that contribution scales as $M^2$. (3) Each Bernoulli variance is $0.50(0.50)=0.25$, so the contrast variance is $0.25/50+0.25/50=0.0100$; its standard error is $0.100$ in response-proportion units. (4) It may be unreliable with sparse pilot samples or when the denominator $\hat p_A+\hat p_B$ can be near zero with appreciable probability, because the ratio is nonlinear and the local approximation may not describe its distribution well.