Stabilizing Response Estimates with Sparse Data

Why Stabilize a Response Estimate?

A DBCD target map may use estimated response probabilities to determine a desired allocation vector. When an arm has only a few observed outcomes, its raw response rate can move sharply after one additional response. Because that estimate is an input to the target map, the resulting target can also move sharply. A prespecified smoothed estimate offers one way to reduce this sensitivity, but it does so by introducing information beyond the arm’s observed outcomes.

The issue is not that a raw rate is inherently wrong. For binary responses, it is the natural sample estimate of the arm’s response probability. Rather, a raw rate based on a small number of observations is imprecise. A design may therefore use a regularized estimate as a more stable input to a target map. This is a design choice: it does not make the underlying response probability different, and it does not by itself guarantee better statistical inference.

Definition: For treatment arm $i$, let $n_i$ be the number of observed binary responses and let $S_i$ be the number of responses recorded as successes. When $n_i>0$, the raw response-rate estimate is $\hat{p}_i=S_i/n_i$. For a prespecified baseline $m_i\in[0,1]$ and pseudo-count strength $\lambda_i\geq 0$, define the fixed-baseline shrinkage estimate by $$ \tilde{p}_i=\frac{S_i+\lambda_i m_i}{n_i+\lambda_i}, \qquad n_i+\lambda_i>0. $$ When $n_i>0$, this can also be written as $$ \tilde{p}_i= \frac{n_i}{n_i+\lambda_i}\hat{p}_i+ \frac{\lambda_i}{n_i+\lambda_i}m_i. $$ If $n_i=0$, the raw rate and this weighted-average representation using $\hat{p}_i$ are undefined. The first formula is defined when $\lambda_i>0$, in which case $\tilde{p}_i=m_i$. The case $n_i=0,\lambda_i=0$ is excluded because its denominator is zero.

The baseline $m_i$ is a reference response rate chosen before using the current arm’s outcomes; it might come from external evidence or a prespecified design value. The strength $\lambda_i$ is measured in pseudo-observations: larger values give the baseline more weight. If $n_i>0$, the estimate is a weighted average of $\hat{p}_i$ and $m_i$, with weights $n_i/(n_i+\lambda_i)$ and $\lambda_i/(n_i+\lambda_i)$. If $\lambda_i=0$, the smoothed estimate equals the raw rate. If $m_i\in[0,1]$, the estimate remains in $[0,1]$.

From Response Estimate to DBCD Target

Write the prespecified target map as $T$, and let $\boldsymbol{\rho}=T(\mathbf{p})$ denote the target allocation vector at response-probability vector $\mathbf{p}$. In practice, the design may plug either the raw response estimates or the smoothed estimates into the map:

\hat{\boldsymbol{\rho}}_{\mathrm{raw}}=T(\hat{\mathbf{p}}), \qquad \hat{\boldsymbol{\rho}}_{\mathrm{smooth}}=T(\tilde{\mathbf{p}}).

These are candidate target vectors, not the same thing as the probabilities used to assign the next participant. The DBCD rule uses its target and current allocation state to determine assignment probabilities. Smoothing changes the response estimates supplied to the target map; it does not, by itself, alter the rule that converts a target into assignment probabilities.

The difference between the two plug-in targets depends on both the smoothing and the map. Even a modest change in one response estimate can have a substantial target effect if the map is locally sensitive there. Conversely, a map may damp the effect. Thus, comparing response estimates is necessary but not always sufficient: assess the target values that the design actually uses.

Worked Example: Comparing Raw and Smoothed Inputs

Suppose two arms have binary responses. Arm 1 has $S_1=1$ success among $n_1=2$ observed outcomes, and arm 2 has $S_2=6$ among $n_2=10$. Use $m_1=m_2=0.50$ and $\lambda_1=\lambda_2=4$. For illustration, suppose the prespecified target map is $\rho_1=p_1/(p_1+p_2)$ and $\rho_2=1-\rho_1$, with positive input rates.

The raw estimates are $$ \hat{p}_1=\frac{1}{2}=0.5000, \qquad \hat{p}_2=\frac{6}{10}=0.6000. $$ The smoothed estimates are $$ \tilde{p}_1=\frac{1+4(0.50)}{2+4}=\frac{3}{6}=0.5000, \qquad \tilde{p}_2=\frac{6+4(0.50)}{10+4}=\frac{8}{14}=0.5714. $$

With raw inputs, the target share for arm 1 is $$ \hat{\rho}_{1,\mathrm{raw}}=\frac{0.5000}{0.5000+0.6000} =\frac{0.5000}{1.1000}=0.4545. $$ With smoothed inputs, it is $$ \hat{\rho}_{1,\mathrm{smooth}}= \frac{0.5000}{0.5000+0.5714} =\frac{0.5000}{1.0714}=0.4667. $$ The corresponding arm 2 target shares are $0.5455$ and $0.5333$, respectively. Here smoothing moves the target share for arm 1 by about $0.0121$, or 1.21 percentage points. These are hypothetical calculations for the stated map, not a claim that this map is optimal for every trial objective.

What the Smoothing Does—and Does Not—Guarantee

A simple fixed-sample model makes the trade-off explicit. Suppose, for this calculation, that arm $i$ has a fixed sample size $n_i>0$ and independent Bernoulli outcomes with a common success probability $p_i$. Then $S_i$ has a binomial distribution with size $n_i$ and probability $p_i$. The baseline $m_i$ and strength $\lambda_i$ are fixed rather than estimated from these same outcomes. Under these assumptions, the expectation and variance of the smoothed estimate follow directly from the mean and variance of $S_i$.

E(\tilde{p}_i)= \frac{n_i p_i+\lambda_i m_i}{n_i+\lambda_i}, \qquad \operatorname{Bias}(\tilde{p}_i)= E(\tilde{p}_i)-p_i =\frac{\lambda_i(m_i-p_i)}{n_i+\lambda_i}.

The variance is

\operatorname{Var}(\tilde{p}_i) =\frac{\operatorname{Var}(S_i)}{(n_i+\lambda_i)^2} =\frac{n_i p_i(1-p_i)}{(n_i+\lambda_i)^2}.

For comparison, the raw rate has expectation $p_i$ and variance $p_i(1-p_i)/n_i$ under this fixed-sample model. Since the mean squared error is variance plus squared bias, the smoothed estimate has

\operatorname{MSE}(\tilde{p}_i) =\frac{n_i p_i(1-p_i)}{(n_i+\lambda_i)^2} +\left(\frac{\lambda_i(m_i-p_i)}{n_i+\lambda_i}\right)^2.

The variance term decreases as $\lambda_i$ increases, but the squared-bias term can increase if $m_i$ is far from $p_i$. Smoothing therefore does not always reduce mean squared error. Its advantage is most plausible when the baseline is reasonably aligned with the arm’s response probability and the raw rate is particularly noisy.

Worked Example: Quantifying the Bias–Variance Trade-Off

For a hypothetical fixed-sample illustration, suppose an arm has $n_i=4$ independent Bernoulli outcomes with true response probability $p_i=0.70$. Use baseline $m_i=0.50$ and strength $\lambda_i=4$. The true probability is specified here to evaluate the estimator’s repeated-sampling properties; it would not be known when operating the design.

The raw estimate has variance $$ \operatorname{Var}(\hat{p}_i)=\frac{0.70(1-0.70)}{4} =\frac{0.21}{4}=0.0525 $$ and zero bias under the stated model.

For the smoothed estimate, the bias is $$ \operatorname{Bias}(\tilde{p}_i) =\frac{4(0.50-0.70)}{4+4} =\frac{-0.80}{8}=-0.1000. $$ Its variance is $$ \operatorname{Var}(\tilde{p}_i) =\frac{4(0.70)(0.30)}{8^2} =\frac{0.84}{64}=0.01313. $$ Consequently, its mean squared error is $$ \operatorname{MSE}(\tilde{p}_i)=0.01313+(-0.1000)^2 =0.02313. $$ In this particular hypothetical calculation, the smoothed estimate has lower mean squared error than the raw estimate, $0.02313<0.0525$, despite its bias. That comparison depends on the assumed true rate, baseline, strength, and fixed sample size; it is not a universal guarantee.

Worked Choices for Sparse Arms

The choice of baseline and strength matters most when $n_i$ is small, because the baseline’s weight is $\lambda_i/(n_i+\lambda_i)$. For a fixed $\lambda_i$, that weight shrinks as observations accumulate. This produces a transparent form of regularization: early estimates are pulled more toward the prespecified baseline, while later estimates are increasingly determined by the arm’s own data.

Worked Example: How the Baseline Changes an Early Estimate

Suppose an arm has $S_i=2$ successes among $n_i=2$ observed outcomes. The raw response rate is $$ \hat{p}_i=\frac{2}{2}=1.000. $$ Use $\lambda_i=4$. With baseline $m_i=0.50$, $$ \tilde{p}_i=\frac{2+4(0.50)}{2+4} =\frac{4}{6}=0.6667. $$ If instead the prespecified baseline is $m_i=0.80$, then $$ \tilde{p}_i=\frac{2+4(0.80)}{2+4} =\frac{5.2}{6}=0.8667. $$ Both calculations use the same observations and strength; only the baseline changes. A target map receiving these inputs may therefore produce meaningfully different targets. The example illustrates why a baseline should be justified on substantive grounds and fixed according to the design plan, not selected after seeing which value produces a preferred allocation.

Assumptions, Diagnostics, and Alternatives

The fixed-sample derivation above assumes a stable binary outcome probability within each arm, independent outcomes, and a sample size treated as fixed for the calculation. In a response-adaptive trial, assignment depends on earlier information and the arm-specific sample size is random. The algebraic definition of $\tilde{p}_i$ remains available, but the fixed-sample binomial variance and mean squared error calculations should not automatically be read as exact repeated-sampling properties of the adaptive procedure. Those properties may require analysis that accounts for the allocation process and stopping rule.

The practical conditions for using fixed-baseline smoothing as a DBCD input are narrower and explicit:

The response is coded consistently as binary, and $S_i$ counts successes among the $n_i$ observed outcomes.
The baseline $m_i$ lies in $[0,1]$ and has a prespecified rationale. If arms plausibly have different response rates, do not impose a common baseline without justification.
The strength $\lambda_i$ is specified on the pseudo-count scale. For an observed arm, the estimate is defined for any $\lambda_i\geq0$; with no observations, it requires $\lambda_i>0$.
The target map is defined for the resulting estimates, and the design documentation makes clear whether it receives raw or smoothed inputs.

Useful diagnostics include plotting or tabulating raw and smoothed estimates side by side by arm and enrollment stage, recording the effective baseline weight $\lambda_i/(n_i+\lambda_i)$, and checking the resulting target vectors under plausible choices of $m_i$ and $\lambda_i$. Large differences across defensible choices indicate that allocation targets are sensitive to regularization. Also check for delayed or missing outcomes: $n_i$ must count outcomes actually observed, not merely participants assigned to the arm. If response probabilities may change over calendar time or differ by important covariates, a constant arm-specific probability and a fixed baseline may be a poor representation.

The main alternatives have different costs. The raw rate uses only the arm’s observed outcomes and is unbiased in the fixed-sample Bernoulli model, but can be volatile with sparse data. Fixed-baseline shrinkage is transparent and easy to compute, but can be biased toward a poorly chosen baseline. A model-based approach, such as a beta-binomial model, can express prior uncertainty and produce a posterior mean; its result depends on the prior and model, and it is not automatically interchangeable with a fixed pseudo-count rule in every inferential or adaptive setting. Pooling information across arms can stabilize estimates, but is inappropriate if it assumes similar response probabilities when meaningful treatment differences are expected. These methods should be compared using the design’s purpose, assumptions, and target-map sensitivity—not by smoothness alone.

Regularizing an input also does not repair a misspecified target map, guarantee desirable treatment allocation, or establish that the response estimate is suitable for final treatment-effect inference. It is a choice about the signal supplied to the allocation target. The earlier material on target-map sensitivity is relevant here: an estimate change that appears numerically small can still matter if the map reacts strongly in that region. Keep the estimation rule prespecified and distinguish its role in allocation from any later analysis of treatment effects.

Common pitfall: “The smoothed estimate is closer to the baseline, so it must be more accurate.” That conclusion does not follow. Shrinkage lowers variance under the fixed-sample model, but it adds bias whenever $m_i\ne p_i$. Compare plausible baselines and strengths, and assess the resulting target sensitivity; do not treat a stable-looking input as proof of better estimation.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

An arm has $S_i=3$, $n_i=5$, $m_i=0.40$, and $\lambda_i=5$. Calculate the raw and smoothed estimates.
For $n_i=0$, $m_i=0.30$, and $\lambda_i=2$, calculate $\tilde{p}_i$. What happens if $\lambda_i=0$ instead?
In the fixed-sample model, explain in one sentence why smoothing can lower variance yet fail to lower mean squared error.
Suppose the baseline is changed after examining the current arm’s outcomes to make the target more favorable. Identify the problem with this practice.

Solutions. (1) The raw estimate is $\hat{p}_i=3/5=0.6000$. The smoothed estimate is $\tilde{p}_i=[3+5(0.40)]/(5+5)=5/10=0.5000$. (2) Since $\lambda_i>0$, $\tilde{p}_i=[0+2(0.30)]/(0+2)=0.3000=m_i$. If $\lambda_i=0$, the denominator is zero, so the estimate is undefined; neither the raw rate nor its weighted-average representation is available. (3) Smoothing reduces the variance term but can add enough squared bias toward a baseline different from $p_i$ that the total mean squared error increases. (4) Choosing the baseline after seeing outcomes makes the regularization data-dependent and outcome-directed, so the stated prespecified shrinkage rule no longer describes the procedure; it can also distort the target in a selectively favorable direction.