How True Effect Patterns Shape Winner’s-Curse Bias

Why the Pattern of True Effects Matters

In a multi-arm, multi-stage design, an arm may be carried forward because its interim estimate is largest. Even if each arm’s estimate is unbiased before selection, the estimate attached to the winner is generally not unbiased after selection. The size of this winner’s-curse bias depends not just on the amount of noise, but also on the pattern of the true effects: are all arms ineffective, is one arm clearly better, or are several arms similarly effective?

The distinction between choosing the arm with the largest estimate and choosing the arm with the largest true effect is important. When arms have similar effects, the selected arm can often be the wrong arm, even when its estimate is inflated. When one arm is far better than the others relative to the estimation noise, it is more likely to be selected, and the inflation tends to be smaller.

Definition: Let $\hat\theta_i$ be the estimate of the treatment effect for arm $i$, whose true effect is $\theta_i$. If the selected arm is $J=\operatorname{argmax}_{1\leq i\leq K}\hat\theta_i$, the expected winner’s-curse bias is $$ E[\hat\theta_J-\theta_J]. $$ Here $\theta_J$ is the true effect of the arm selected by the data, so $E[\theta_J]$ can itself depend on which arm wins. The selected estimate’s expectation decomposes as $E[\hat\theta_J]=E[\theta_J]+E[\hat\theta_J-\theta_J]$.

For the calculations below, assume that each arm’s estimate is normally distributed around its true effect, with a common standard error $\tau$, and that the estimates are independent: $\hat\theta_i=\theta_i+\varepsilon_i$, where $\varepsilon_i\sim N(0,\tau^2)$. Assume larger effects are better and that the selection rule chooses the largest estimate. Treat $\tau$ as known and use the same units for $\theta_i$, $\hat\theta_i$, and $\tau$. These assumptions make the examples transparent; shared-control contrasts in MAMS designs may be correlated, in which case the independent-estimate calculations do not directly apply.

What a Two-Arm Comparison Shows

For two independent estimates with the same standard error $\tau$, write $d=\theta_1-\theta_2$ and $s=\sqrt{2}\tau$. The difference $\hat\theta_1-\hat\theta_2$ is normal with mean $d$ and standard deviation $s$. Consequently, arm 1 is selected with probability $\Phi(d/s)$, where $\Phi$ is the standard normal cumulative distribution function. Let $\phi$ denote the standard normal density.

The expected maximum of these two estimates can be written as the expected true effect of the selected arm plus the expected selected noise:

P(J=1)=\Phi\!\left(\frac{d}{s}\right), \qquad E[\hat\theta_J] =\theta_1\Phi\!\left(\frac{d}{s}\right) +\theta_2\left[1-\Phi\!\left(\frac{d}{s}\right)\right] +s\phi\!\left(\frac{d}{s}\right).

Thus, the expected true effect of the selected arm is the weighted average of $\theta_1$ and $\theta_2$, using their selection probabilities. The final term, $s\phi(d/s)$, is the winner’s-curse bias. It is positive for a finite gap $d$, and it is largest when the arms have equal true effects. As the gap becomes large relative to $s$, the best arm is selected almost surely and this bias approaches zero.

Interpretation: Winner’s-curse bias is the expected excess of the selected estimate over the true effect of that same selected arm. It is not the difference between the selected estimate and the best true effect, and it is not the probability that the selected arm is truly best.

Worked Example: All Arms Are Ineffective

Suppose three hypothetical treatment arms all have true effect $0$ outcome units. Their independent estimates have common standard error $\tau=0.5$ outcome units, and the arm with the largest estimate is selected. Find the expected selected estimate and its winner’s-curse bias.

Because all three true effects are zero, the selected arm’s true effect is always zero. If $Z_1,Z_2,Z_3$ are independent standard normal variables, then the selected estimate has the same distribution as $0.5\max(Z_1,Z_2,Z_3)$. The density of the maximum is $3\phi(z)\Phi(z)^2$, so its mean is $$ \int_{-\infty}^{\infty} 3z\phi(z)\Phi(z)^2\,dz =\frac{3}{2\sqrt{\pi}} \approx 0.8463. $$

Therefore, $$ E[\hat\theta_J] =0+0.5\left(\frac{3}{2\sqrt{\pi}}\right) \approx 0.4231\text{ outcome units}. $$ Since $E[\theta_J]=0$, the expected bias is also $0.4231$ outcome units. No treatment has a true effect in this hypothetical example, yet selection makes the winning estimate positive on average. The result is not evidence that any arm is effective; it is a consequence of choosing the largest of three noisy estimates.

When One Arm Is Clearly Better

A genuinely superior arm changes the interpretation of the selected estimate. It can make the winner more likely to be the best arm, but it does not automatically eliminate the winner’s curse. The relevant comparison is the true-effect gap measured against the variability of the contrast, not the gap alone.

Worked Example: One Superior Arm and One Ineffective Arm

Consider two hypothetical arms with true effects $\theta_1=1.0$ and $\theta_2=0$ outcome units. Suppose the estimates are independent and each has standard error $\tau=0.5$ outcome units. Calculate the probability that arm 1 is selected, the expected true effect of the selected arm, and the expected selected estimate.

Here $d=1.0$ outcome unit and $s=\sqrt{2}(0.5)=0.7071$ outcome units. Thus $d/s=1.4142$, and $$ P(J=1)=\Phi(1.4142)\approx 0.9214. $$ Arm 1 is truly superior and is selected about 92.14% of the time under this model. The expected true effect of the selected arm is $$ E[\theta_J] =1.0(0.9214)+0(1-0.9214) \approx 0.9214\text{ outcome units}. $$

The expected selected noise, and hence the winner’s-curse bias, is $$ s\phi(d/s) =\sqrt{2}(0.5)\phi(1.4142) \approx 0.7071(0.1468) \approx 0.1038\text{ outcome units}. $$ Therefore, $$ E[\hat\theta_J]=E[\theta_J]+E[\hat\theta_J-\theta_J] \approx 0.9214+0.1038 =1.0251\text{ outcome units}. $$ The calculation uses the unrounded density and probability values before rounding the final result. The selected estimate is, on average, about $0.1038$ outcome units above the true effect of whichever arm was selected. It is not biased upward by $1.0251$ units: that number is the expected estimate itself.

This example separates two features of selection. The probability of identifying the truly superior arm is high because its effect gap is substantial relative to the standard error of the contrast. Nevertheless, whenever selection is based on the same noisy estimates being reported, there remains positive expected selected noise. If the gap grows further while the standard error stays fixed, the probability of selecting arm 1 approaches one and the winner’s-curse bias approaches zero.

Several Arms With Similar Effects

When several arms have equal true effects, there is no uniquely best arm. Selection still favors a positive noise realization, and adding arms increases the expected maximum. Equal effects provide a useful benchmark for several similarly effective treatments: small differences between their true effects may alter which arm wins, but noise can remain important when those differences are small relative to the standard errors.

Worked Example: Three Similarly Effective Arms

Suppose three hypothetical arms each have true effect $0.4$ outcome units. Their independent estimates have standard error $0.5$ outcome units, and the largest estimate is selected. Find the expected selected estimate and its bias.

Since all three effects are equal, selection does not change the true effect of the selected arm: $\theta_J=0.4$ outcome units. The expected maximum of three independent standard normal variables is $3/(2\sqrt{\pi})$, as in the preceding calculation. Therefore, $$ E[\hat\theta_J] =0.4+0.5\left(\frac{3}{2\sqrt{\pi}}\right) \approx 0.4+0.4231 =0.8231\text{ outcome units}. $$

The expected winner’s-curse bias is $0.8231-0.4=0.4231$ outcome units. Every arm is truly equally effective, so selecting the largest estimate cannot identify a truly superior arm; it identifies the arm with the most favorable observed estimate. The same noise-driven inflation appears in the all-ineffective example, but here the selected estimate is centered around a nonzero common effect before selection.

A pair of nearly tied arms illustrates how bias can remain substantial even when one is technically better. For example, with effects $0.2$ and $0$ outcome units and standard error $0.5$ for each independent estimate, $d/s=0.2/(\sqrt{2}\times0.5)=0.2828427$. The probability of selecting the higher-effect arm is about $0.6114$, while the bias is $\sqrt{0.5}\phi(0.2828427)\approx 0.2710$ outcome units. The expected true effect of the selected arm is approximately $0.2(0.6114)=0.1223$ outcome units, giving an expected selected estimate of about $0.1223+0.2710=0.3933$ outcome units. The small true-effect gap leaves considerable scope for noise to determine the winner.

Derivation and Practical Interpretation

The two-arm result follows by writing the selected estimate as the estimate from arm 2 plus the positive part of the estimated contrast: $\max(\hat\theta_1,\hat\theta_2)=\hat\theta_2+\max(\hat\theta_1-\hat\theta_2,0)$. Let $D=\hat\theta_1-\hat\theta_2$, so $D\sim N(d,s^2)$. For a normal variable, direct integration gives $E[\max(D,0)]=d\Phi(d/s)+s\phi(d/s)$. Adding $E[\hat\theta_2]=\theta_2$ yields $E[\hat\theta_J]=\theta_2+d\Phi(d/s)+s\phi(d/s)$, which is the formula above. The term $\theta_2+d\Phi(d/s)$ is $E[\theta_J]$; the remaining term is the expected excess from selection.

This decomposition helps compare effect patterns without conflating selection accuracy and estimation bias. When all arms are ineffective, the selected arm’s true effect is zero, so the positive expected estimate is entirely selection bias. With one clearly superior arm, the selected arm’s true effect is often large, and bias may be smaller when the true-effect gap is large relative to noise. With several similar effects, the selected arm may not be the truly best one, and the selected estimate can still be noticeably inflated. The expected true effect of the winner and the bias of its estimate answer different questions.

The calculations assume a fixed set of arms, equal standard errors, independent normal estimates, and selection by the largest estimate. They describe the expectation under repeated hypothetical repetitions of that same selection procedure; they do not say that every selected estimate will be too high. In MAMS settings, stagewise selection, stopping rules, unequal standard errors, and shared-control covariance can change the selection event and the distribution of the selected estimate. The equal, independent calculations are benchmarks, not substitutes for modeling the actual selection process.

Diagnostics, Alternatives, and a Common Pitfall

Before using these benchmarks, check that the selection rule really is “choose the largest estimate,” that the estimates being compared refer to the same estimand, and that the assumed uncertainty structure is plausible. In particular, estimates sharing a control group are generally correlated; treating them as independent can misstate the spread of their differences and hence the chance of each selection outcome. Unequal standard errors also make a common-$\tau$ comparison inappropriate. The earlier material on shared-control covariance and unequal standard errors addresses those features.

A useful diagnostic summary is to compare true-effect gaps (or plausible effect gaps) with the standard error of the relevant contrasts. Also report how many arms could have been selected: even with equal effects and equal standard errors, the expected maximum grows as the number of candidate arms increases. These summaries do not by themselves correct an estimate. They show whether the selection mechanism is likely to matter and which assumptions need closer scrutiny.

The main alternative to selecting and estimating from the same data is to use information not used to choose the arm, such as an independent validation sample. A prespecified arm avoids choosing a winner among the same noisy estimates, but it answers a different design question and may sacrifice the opportunity to identify a better treatment. Selection-adjusted estimation instead models the selection event. These approaches target related but distinct goals; this tutorial’s calculations describe the unadjusted selected estimate rather than a correction procedure.

Common mistake: “The selected arm has the highest estimate, so its true effect must be the highest.” The largest estimate is the observed winner, not necessarily the truly best arm. With similar effects, noise can readily change the ranking.
Correction: Report selection probability and expected bias as separate quantities. A high probability of choosing the truly best arm does not imply zero bias in its selected estimate.
Common mistake: “All arm estimates are unbiased, so the selected estimate is unbiased.” Unbiasedness applies before selection; conditioning the reported estimate on being largest favors positive noise.
Correction: Evaluate $E[\hat\theta_J-\theta_J]$ under the full selection rule and the relevant joint distribution of the estimates.

Key takeaway: All-ineffective arms can produce a positive selected estimate; a clearly superior arm tends to be selected more reliably and with less inflation; and several similarly effective arms leave the winner especially vulnerable to noise-driven ranking. The pattern of true effects shapes both which arm wins and how far its estimate is expected to exceed its own true effect.

Check Your Understanding

Answer each question, then compare with the short solutions below.

Three independent estimates have equal standard error and all three true effects are zero. Why can the selected estimate have positive expectation?
For two arms with a common standard error $\tau$, what quantity determines whether the effect gap is large relative to the noise in their contrast?
Two arms have effects $1.0$ and $0$ outcome units, with independent standard errors $0.5$. What is the approximate probability of selecting the higher-effect arm, and what is the winner’s-curse bias?
Three arms have equal true effects of $0.4$ outcome units and independent standard errors of $0.5$. What is the expected selected estimate?
Why might the independent-estimate benchmark fail for treatment-control contrasts using a shared control group?

Solutions. (1) Selection chooses the largest of several noisy estimates, favoring positive noise; with three equal-effect normal estimates, the expected maximum exceeds their common mean. (2) Use $d/(\sqrt{2}\tau)$, where $d$ is the true-effect difference; it compares the gap with the standard deviation of the estimated contrast. (3) The probability is $\Phi(1/(\sqrt{2}\times0.5))=\Phi(1.4142)\approx0.9214$. The bias is $\sqrt{2}(0.5)\phi(1.4142)\approx0.1038$ outcome units. (4) The expectation is $0.4+0.5[3/(2\sqrt{\pi})]\approx0.8231$ outcome units. (5) Shared-control contrasts are correlated, so the independent model does not represent the joint distribution that determines which estimate is largest.