Why the Conditioning Set Matters
When a multi-arm trial selects a treatment arm using interim estimates, the estimate associated with the selected arm is not being viewed in the same way as an estimate for a prespecified arm. The selected estimate has been singled out because of the observed data. Its expected error among trials selecting that arm can therefore differ from its expected error across all trials.
The distinction is between two questions. Conditional bias asks how the selected estimate behaves among repetitions in which a particular arm is selected. Unconditional bias asks for its average error across repetitions, allowing the selected arm to vary. Neither quantity is universally preferable: each answers a different estimation question.
The last expression is a weighted average, with weights equal to the probabilities that the arms are selected. It is not generally the ordinary average of the \(b_i\). The definition also makes the target explicit: the unconditional error compares the estimate with \(\theta_J\), the true effect for the selected arm. Since \(J\) can vary across repetitions, this target is itself random across repetitions.
For comparison, before selection, each fixed-arm estimator may be unbiased: \(E(\hat{\theta}_i-\theta_i)=0\). That fact does not imply \(b_i=0\). Conditioning on \(J=i\) changes the distribution of the data used to estimate arm \(i\). Nor does the fixed-arm statement by itself give the unconditional bias \(B\) of the selected estimate, because \(B\) combines estimates for different arms against their respective effects.
Assumptions and a Basic Decomposition
The weighted-average identity requires only that the possible selection events \(J=i\) are mutually exclusive and exhaustive and that the estimation errors have finite expectations. It follows from the law of total expectation. It does not require normal estimates, equal standard errors, or independent treatment-effect estimates. To interpret the terms as selection bias relative to ordinary estimation, it is also useful to distinguish the prespecified-arm errors \(\hat{\theta}_i-\theta_i\) from the selection rule that determines \(J\).
Here \(I(J=i)\) is 1 when arm \(i\) is selected and 0 otherwise. The second line partitions the selected-estimate error by the identity of the selected arm. The third line uses the definition of conditional expectation. If a design can select no arm, that possibility must be included as another event, with its own explicitly defined output and target; it cannot simply be omitted from the average.
A common model for examining this distinction is \(\hat{\theta}_i=\theta_i+\varepsilon_i\), where each error has mean zero before selection. If the rule selects the largest estimate, then \(J=\arg\max_i\hat{\theta}_i\), with a tie rule specified if ties can occur. The selection event depends on the estimates, so the selected error need not have mean zero. The following normal calculation illustrates the mechanism; it is not needed for the general decomposition.
Worked Example: Two Equal Arms With Normal Estimates
Suppose two hypothetical treatment arms have the same true effect, \(\theta_1=\theta_2=\theta\), and independent estimates \(\hat{\theta}_i=\theta+\varepsilon_i\), with \(\varepsilon_i\sim N(0,\sigma^2)\). Select the arm with the larger estimate. Find the conditional bias for either selected arm and the unconditional bias of the selected estimate. Assume \(\sigma=2\) effect units.
Arm 1 is selected precisely when \(\varepsilon_1>\varepsilon_2\). Define \(D=\varepsilon_1-\varepsilon_2\). Then \(D\sim N(0,2\sigma^2)\), and the joint normal model gives $$ E(\varepsilon_1\mid D)=\frac{\operatorname{Cov}(\varepsilon_1,D)}{\operatorname{Var}(D)}D =\frac{\sigma^2}{2\sigma^2}D=\frac{D}{2}. $$ For a zero-mean normal variable with standard deviation \(\sqrt{2}\sigma\), its mean conditional on being positive is \(2\sigma/\sqrt{\pi}\). Therefore $$ b_1=E(\varepsilon_1\mid D>0)=\frac{1}{2}\frac{2\sigma}{\sqrt{\pi}}=\frac{\sigma}{\sqrt{\pi}}. $$ By symmetry, \(b_2=\sigma/\sqrt{\pi}\) as well. Each arm is selected with probability \(1/2\), so the decomposition gives \(B=(1/2)b_1+(1/2)b_2=\sigma/\sqrt{\pi}\).
Substituting \(\sigma=2\) effect units gives \(2/\sqrt{\pi}=1.128\) effect units (to four significant figures). Thus, among trials selecting either particular arm, its estimate averages 1.128 units above its true effect; averaged over all trials, the estimate of the selected arm’s effect has the same upward bias in this symmetric example. This is the winner’s-curse effect: selection favors estimates with positive errors.
Unequal Selection Probabilities Change the Average
Even when each arm’s conditional bias is well defined, the unconditional bias depends on how frequently each arm is selected. This dependence matters when true effects differ, standard errors differ, or estimates are correlated: those features can change both the conditional distributions and the selection probabilities. The conditional bias for arm \(i\) is not a weight-free summary of the unconditional performance of the selected estimate.
Worked Example: Weighting Conditional Biases Correctly
Consider a hypothetical two-arm selection rule that chooses the larger observed estimate. Let the true effects be \(\theta_1=0\) and \(\theta_2=1\) effect unit. For a transparent calculation, suppose the independent estimation errors for each arm are equally likely to be \(-1\) or \(+1\) effect unit. The four error pairs are equally likely. Calculate the conditional biases and their unconditional average.
The possible observations and selection outcomes are:
| \((\varepsilon_1,\varepsilon_2)\) | \((\hat{\theta}_1,\hat{\theta}_2)\) | Selected arm | Selected error |
|---|---|---|---|
| \((-1,-1)\) | \((-1,0)\) | 2 | \(-1\) |
| \((-1,+1)\) | \((-1,2)\) | 2 | \(+1\) |
| \((+1,-1)\) | \((1,0)\) | 1 | \(+1\) |
| \((+1,+1)\) | \((1,2)\) | 2 | \(+1\) |
Arm 1 is selected in one of four equally likely cases, and its error there is \(+1\), so \(P(J=1)=1/4\) and \(b_1=1\) effect unit. Arm 2 is selected in three cases, with errors \(-1,+1,+1\), so $$ P(J=2)=\frac{3}{4},\qquad b_2=\frac{-1+1+1}{3}=\frac{1}{3}\text{ effect unit}. $$ The unconditional bias is therefore $$ B=\frac{1}{4}(1)+\frac{3}{4}\left(\frac{1}{3}\right)=\frac{1}{2}\text{ effect unit}. $$
The unweighted average of the two conditional biases would be \((1+1/3)/2=2/3\) effect unit, which is not the unconditional bias: the arms are not selected equally often. The table also confirms the weighted calculation directly: the four selected errors sum to \(-1+1+1+1=2\), so their average is \(2/4=0.5\) effect unit. This discrete example is chosen for exact arithmetic; the decomposition itself is not specific to this error distribution.
Which Unconditional Quantity Is Being Asked For?
“Unconditional bias” can refer to different estimands if the target is not stated. The quantity \(E(\hat{\theta}_J-\theta_J)\) averages the error of the estimate for the arm selected in each trial. In contrast, \(E(\hat{\theta}_i-\theta_i)\) concerns one fixed, prespecified arm. These are different expectations, and a numerical answer to one does not answer the other.
In particular, an estimate can be unbiased for every fixed arm before selection while being biased after the arm is chosen using those same estimates. The source is not a failure of the fixed-arm estimator’s original expectation; it is the dependence between its error and the event that it is selected. If selection is based on data independent of the data used for estimation, that dependence can be removed under suitable assumptions.
Worked Example: Selecting With One Data Set, Estimating With Another
Use the two-arm setting in the previous example to select an arm from interim estimates, so \(P(J=1)=1/4\) and \(P(J=2)=3/4\). Now suppose the reported estimate comes from independent validation data: \(V_i=\theta_i+\eta_i\), where \(E(\eta_i)=0\) and the vector of validation errors is independent of the interim data used to choose \(J\). Find the conditional and unconditional biases of \(V_J\).
Because selection depends only on the interim data, conditioning on \(J=i\) does not change the mean of the independent validation error. Thus $$ E(V_i-\theta_i\mid J=i)=E(\eta_i\mid J=i)=E(\eta_i)=0 $$ for each arm with positive selection probability. The unconditional bias is $$ E(V_J-\theta_J)=\frac{1}{4}(0)+\frac{3}{4}(0)=0\text{ effect units}. $$
Under these assumptions, the validation estimate is unbiased both conditional on the selected arm and across all trials. This does not make the selection irrelevant: the reported target remains the effect of the arm chosen at interim, and the validation estimate may be less precise or require additional data. The example isolates why reusing the selection data for estimation can create a different bias calculation.
Diagnostics, Comparisons, and Limitations
A useful check is to write down the random quantity whose expectation is being called “bias” before doing any calculation. Then identify the conditioning event and the target parameter. For a single selected arm, the basic audit is to verify that the selection probabilities sum to 1 and that the unconditional result equals the probability-weighted conditional results. In a simulation, estimate each \(P(J=i)\), each conditional mean error among trials selecting \(i\), and the overall mean selected error; the weighted and direct overall estimates should agree up to Monte Carlo error.
Conditional and unconditional summaries answer complementary questions. Conditioning on \(J=i\) describes the repeated-trial performance when arm \(i\) is the winner. Averaging across all possible winners describes average performance over the trial-selection process. An analysis conditional on a particular selection outcome can be important when interpreting that outcome, while an unconditional summary is natural for evaluating average behavior of the design. Neither can substitute for the other without specifying the intended use.
| Quantity | Conditioning | Target | Question answered |
|---|---|---|---|
| Fixed-arm bias | None | \(\theta_i\) for a prespecified arm | Is the arm’s estimator unbiased before selection? |
| Conditional selected-arm bias | \(J=i\) | \(\theta_i\) among trials selecting arm \(i\) | How does the estimate behave when this arm wins? |
| Unconditional selected-estimate bias | None; average over \(J\) | \(\theta_J\), the selected arm’s true effect | What is the average error across all trial outcomes? |
A common mistake is to calculate conditional biases for each arm and take their simple average. That is correct only when the arms have equal selection probabilities. The correction is to use \(B=\sum_i P(J=i)b_i\). Another mistake is to infer that the selected estimate is unbiased because every fixed-arm estimate is unbiased before selection. The correction is to recognize that selecting on the same noisy estimates changes their error distributions. Conversely, a positive conditional bias for one arm does not alone determine the unconditional bias: the selection probabilities and the conditional biases for the other possible outcomes also matter.
These expectations describe bias, not the whole quality of an estimator. They do not by themselves quantify variability, confidence-interval coverage, or the probability of selecting the truly best arm. They can also conceal important variation: a small unconditional bias could result from cancellation among conditional biases rather than accurate estimation in every selection scenario. Report the conditioning event, target, and selection probabilities whenever the distinction matters.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If \(P(J=1)=0.2\), \(P(J=2)=0.8\), and the conditional biases are \(b_1=2\) and \(b_2=-0.5\) effect units, what is the unconditional bias of the selected estimate?
- Why does \(E(\hat{\theta}_i-\theta_i)=0\) not establish that \(E(\hat{\theta}_i-\theta_i\mid J=i)=0\)?
- In the equal-arm normal example, what feature of the selection rule makes the conditional bias positive?
- If a trial can select no arm, why is \(\sum_iP(J=i)b_i\) incomplete unless the no-selection outcome is handled?
Solutions. 1. The weighted bias is \(0.2(2)+0.8(-0.5)=0.4-0.4=0\) effect units. 2. The selection event depends on the estimates, so conditioning on that event can change the mean error even when its unconditional mean is zero. 3. Selecting the larger estimate favors positive estimation errors; conditional on arm 1 winning, for example, its error exceeds the other arm’s error. 4. The arm-selection events then do not exhaust all trial outcomes, so their probabilities sum to less than 1. The no-selection event and its reported estimate and target must be defined to calculate an unconditional expectation over all trials.