Deriving Bias in a Two-Stage Selected Trial

Why Derive the Bias Under a Selection Rule?

When a trial selects a treatment arm using interim results, the data used to make that choice are also part of the final estimate. The selected arm’s stage-one estimate is therefore not an ordinary unbiased estimate conditional on selection: values that helped an arm win tend to be unusually high. A useful way to quantify this effect is to calculate the expectation of the final estimate directly under a specified selection rule.

This tutorial studies a simple two-stage normal model. The selection rule is fixed in advance: select arm 1 if its stage-one estimate exceeds arm 2’s by more than a threshold $c$; otherwise select arm 2. The calculation concerns the bias of the final estimate for the effect of the arm that was actually selected. It is not the bias for a fixed arm regardless of the selection decision.

Model and assumptions: For arms $i=1,2$, let $\theta_i$ be the true treatment effect on a common scale. The stage-one estimates satisfy $\hat{\theta}_{i1}\sim N(\theta_i,v_1)$, independently across arms. The estimates from stage 2 satisfy $\hat{\theta}_{i2}\sim N(\theta_i,v_2)$, independently of stage 1 and of one another. The variances $v_1$ and $v_2$ are known and equal across arms. The trial selects arm 1 if $D=\hat{\theta}_{11}-\hat{\theta}_{21}>c$, and arm 2 otherwise. The final estimate for the selected arm is $\hat{\theta}_J=w\hat{\theta}_{J1}+(1-w)\hat{\theta}_{J2}$, where $J$ denotes the selected arm and $w$ is a fixed stage-one weight.

For example, with equal per-observation variance and stage sample sizes $n_1$ and $n_2$, the inverse-variance weighted combined estimate has $w=n_1/(n_1+n_2)$. The derivation below requires the stated fixed weight and independence assumptions; it does not require that particular sample-size interpretation. Write $\delta=\theta_1-\theta_2$, $s=\sqrt{2v_1}$, $a=(c-\delta)/s$, and let $\phi$ and $\Phi$ denote the standard normal density and cumulative distribution functions.

Deriving the Truncated-Normal Expectation

The contrast $D$ is normal with mean $\delta$ and variance $2v_1$, so $D\sim N(\delta,s^2)$. The selection probabilities are $P(J=1)=1-\Phi(a)$ and $P(J=2)=\Phi(a)$. To find the bias, define the stage-one errors $e_1=\hat{\theta}_{11}-\theta_1$ and $e_2=\hat{\theta}_{21}-\theta_2$. The selection event depends on their difference, $D=\delta+e_1-e_2$, which makes these errors associated with the selection event.

Because the errors and $D$ are jointly normal, their conditional expectations given $D$ are linear. Their covariances with $D$ are $\operatorname{Cov}(e_1,D)=v_1$ and $\operatorname{Cov}(e_2,D)=-v_1$. Consequently,

E(e_1\mid D)=\frac{v_1}{s^2}(D-\delta),\qquad E(e_2\mid D)=-\frac{v_1}{s^2}(D-\delta).

Integrating over the relevant truncated regions gives the error contributions for each selection outcome. For the upper tail, substitute $z=(d-\delta)/s$; for the lower tail, use the same change of variable and the symmetry of the normal density:

\begin{aligned} E\{e_1 I(D>c)\} &=\frac{v_1}{s^2}\int_c^\infty(d-\delta)f_D(d)\,dd =\frac{v_1}{s}\int_a^\infty z\phi(z)\,dz =\frac{v_1}{s}\phi(a),\\ E\{e_2 I(D\le c)\} &=-\frac{v_1}{s^2}\int_{-\infty}^c(d-\delta)f_D(d)\,dd =-\frac{v_1}{s}\int_{-\infty}^a z\phi(z)\,dz =\frac{v_1}{s}\phi(a). \end{aligned}

Here $I(\cdot)$ is an indicator, equal to 1 when its condition holds and 0 otherwise. The integral identities follow from $\phi'(z)=-z\phi(z)$: integrating $z\phi(z)$ from $a$ to infinity gives $\phi(a)$, while integrating from negative infinity to $a$ gives $-\phi(a)$. Thus each branch contributes a positive amount to the expected selected-arm stage-one error.

Result: Under the model and threshold rule above, the unconditional bias relative to the selected arm’s true effect is $$ E(\hat{\theta}_J-\theta_J)=w\,\frac{2v_1}{s}\phi(a),\qquad s=\sqrt{2v_1},\quad a=\frac{c-(\theta_1-\theta_2)}{s}. $$ The stage-two contribution to this expectation is zero because its estimation errors are independent of the selection decision and have mean zero. The result is an expectation over repeated trials, including the selection decision.

Conditional bias is a different quantity. Given that arm 1 is selected, the stage-one error has expectation $(v_1/s)\phi(a)/[1-\Phi(a)]$. Given that arm 2 is selected, its stage-one error has expectation $(v_1/s)\phi(a)/\Phi(a)$. Each is positive, but their magnitudes need not match: the probabilities of the two selection events can differ. Multiplication by $w$ gives the corresponding conditional bias of the final estimate, since the stage-two estimate remains conditionally unbiased.

Worked Example: Rare Selection of Arm 1

Suppose the two arms have equal true effects, $\theta_1=\theta_2=0$ effect units, and each stage-one estimate has variance $v_1=0.25$ effect-units squared. Select arm 1 if its stage-one estimate exceeds arm 2’s by more than $c=1$ effect unit. Let $w=0.50$. The normal model assumes independent stage-one estimates with the stated common variance, as required.

Here $\delta=0$, $s=\sqrt{2(0.25)}=0.7071$ effect units, and $a=(1-0)/0.7071=1.4142$. Thus $P(J=1)=1-\Phi(1.4142)\approx0.07865$ and $\phi(a)\approx0.1468$.

Conditional on the relatively rare selection of arm 1, the stage-one error is \[ \frac{v_1}{s}\frac{\phi(a)}{1-\Phi(a)} =\frac{0.25}{0.7071}\frac{0.1468}{0.07865} \approx0.6597 \] effect units. The final estimate’s conditional bias is $w$ times this value, or $0.50(0.6597)\approx0.3299$ effect units. The calculation shows why a rare selection event can produce a large conditional bias.

For the unconditional bias, add the two branch contributions before multiplying by $w$: \[ w\frac{2v_1}{s}\phi(a) =0.50\frac{2(0.25)}{0.7071}(0.1468) \approx0.0519 \] effect units. This is smaller than the conditional bias for arm 1 because that branch occurs only about 7.865% of the time; it is the bias averaged over both possible selections.

How the Threshold and True Effects Shape Bias

The formula depends on $a$, the threshold measured in standard deviations from the mean stage-one contrast. When $a$ is large and positive, selecting arm 1 is a rare upper-tail event, and its conditional bias can be substantial. The unconditional bias contains $\phi(a)$, so it becomes small when the threshold is far into either tail: in that case one branch is nearly certain and the decision is close to fixed. The largest unconditional bias occurs when $a=0$, when the threshold is at the mean contrast.

Worked Example: Unequal True Effects and a Zero Threshold

Suppose $\theta_1=0.40$ and $\theta_2=0$ effect units, $v_1=0.25$ effect-units squared, $c=0$, and $w=0.50$. As in the preceding example, independent normal stage-one errors with the stated variance are assumed; stage-two errors are independent and mean zero.

Now $\delta=0.40$, $s=0.7071$, and $a=(0-0.40)/0.7071=-0.5657$. From the standard normal distribution, $\phi(a)\approx0.3400$ and $P(J=1)=1-\Phi(-0.5657)=\Phi(0.5657)\approx0.7142$.

The unconditional bias of the final selected-arm estimate is \[ 0.50\frac{2(0.25)}{0.7071}(0.3400)\approx0.1202 \] effect units. To check the target being used, the selected arm’s true effect is $\theta_J$, which is $0.40$ when arm 1 is selected and 0 otherwise. Its expectation is $0.40(0.7142)+0(1-0.7142)\approx0.2857$ effect units. Therefore the expected final estimate is approximately $0.2857+0.1202=0.4059$ effect units. The bias is measured relative to the random selected-arm effect, not relative to $\theta_1$ alone.

Worked Example: Changing the Stage-One Weight

Suppose $\theta_1=\theta_2=0$, $v_1=0.04$ effect-units squared, and the rule selects arm 1 when $D>0.20$ effect units. Let the final estimate give stage one weight $w=0.40$. Assume independent normal stage-one errors, independent mean-zero stage-two errors, and a fixed weight.

The contrast standard deviation is $s=\sqrt{2(0.04)}=0.2828$ effect units, so $a=0.20/0.2828=0.7071$, and $\phi(a)\approx0.3107$. The probability of selecting arm 1 is about $1-\Phi(0.7071)=0.2398$, confirming that this is an upper-tail selection.

The bias is \[ 0.40\frac{2(0.04)}{0.2828}(0.3107)\approx0.03515 \] effect units. If the stage-one weight were 0.80 instead, with the same selection rule and stage-one precision, the bias would double to approximately $0.07030$ effect units. This comparison isolates the direct role of reusing selection-stage data: more weight on those data means more of their selection-related expectation carries into the final estimate.

Assumptions, Diagnostics, and Alternatives

The derivation is exact for the specified normal model, fixed threshold, common stage-one variance, independent arm estimates, fixed weight, and independent stage-two data. It also treats the two-arm selection rule as prespecified. If variances differ by arm, the contrast and its covariances change; if selection uses multiple interim statistics or additional adaptation criteria, the truncated region is no longer the single half-line used here. The displayed formula should not be applied unchanged in those settings.

Diagnostics should focus on whether the design and data support the model’s ingredients. Check that the threshold really was fixed as stated, that stage-two observations do not contribute to the selection decision, and that the final estimator uses the specified weight. For continuous outcomes, inspect the stagewise estimate distributions and investigate outliers or strong departures from the assumed normal approximation. If stage-one estimates have unequal precision or are dependent, recalculate the contrast variance and covariances rather than inserting a pooled variance mechanically.

A fixed-arm estimate that does not use a selection decision is the main benchmark: under this model it has zero bias for its own arm effect. That is not the same target as the selected-arm effect. A conditional calculation reports bias among trials selecting a specified arm; the unconditional calculation averages over both arms and their selection probabilities. Neither should be confused with a calculation that conditions on a different adaptation event. Simulation under the fully specified design can serve as a numerical check, but it does not replace identifying the target, conditioning event, and model assumptions. Outside this model, the integral may need to be evaluated for the actual selection region or the expectation assessed by simulation.

Common Pitfall

A common error is to divide the arm-1 upper-tail contribution by its selection probability, then report that conditional bias as the overall bias. That division gives the stage-one error expectation given arm 1 was selected. Unconditional bias instead adds the contributions from both selection branches, each weighted by the probability of that branch through its truncated expectation; the formula is $w\,2v_1\phi(a)/s$. Another error is to compare the estimate with a fixed arm’s effect even though the stated target is $\theta_J$. State whether the target is fixed or selected before interpreting any bias.

Key takeaway: In this two-stage normal threshold design, selection creates a positive expected stage-one error for whichever arm is chosen. The unconditional bias of the final estimate equals $w\,2v_1\phi(a)/\sqrt{2v_1}$, while conditional biases additionally divide by the probability of the specified selection event.

Check Your Understanding

Answer each question, then compare with the brief solutions below. Use the model and definitions given in this tutorial.

If $a=0$, which selection probability is $1/2$, and what feature of the threshold does this represent?
With $v_1=0.50$, $a=0$, and $w=0.25$, calculate the unconditional bias in effect units.
Why is the conditional arm-1 bias not generally equal to the unconditional bias?
If stage-two data are used to decide which arm is selected, which part of the derivation no longer follows from the stated assumptions?

Solutions. (1) Both probabilities are $1/2$, because $P(J=1)=1-\Phi(0)=1/2$ and $P(J=2)=\Phi(0)=1/2$; the threshold equals the mean contrast. (2) Here $s=\sqrt{2(0.50)}=1$, $\phi(0)=0.3989$, and the bias is $0.25[2(0.50)/1](0.3989)\approx0.09974$ effect units. (3) Conditional bias divides a branch’s truncated expectation by that branch’s selection probability, whereas unconditional bias averages the contributions from both branches. (4) Stage-two errors would no longer be independent of selection, so their expected contribution could not be set to zero by the independence argument used here.