Winner’s Curse In Selected Treatment Effect Estimates

Why the Largest Estimate Tends to Be Too Large

In a multiarm trial, investigators may compare several experimental treatments with a control and carry forward the arm with the largest estimated benefit. Even if every arm has the same true treatment effect, the estimates will differ because of sampling variation. The arm selected as the winner is, by definition, the one whose observed estimate happened to be largest. Its random error is therefore likely to be positive. This systematic optimism in the selected estimate is the winner’s curse.

The issue is distinct from whether a multiplicity-adjusted test controls a false-positive rate. Here the target is the expected size of the selected point estimate relative to the common true effect. The estimate for each arm can be unbiased before selection, while the estimate chosen after looking across arms is biased upward.

The effect scale must be stated in context: for example, a mean difference in points, a difference in response probability, or a log hazard ratio. The formulas below concern the estimate on its stated analysis scale. They do not imply that the true effect of the selected treatment is larger when all true effects are equal.

Normal Model and a General Result

Assume that the estimated effects are jointly normal, each estimate is unbiased for its arm’s true effect, and each has the same standard error \(SE\). Under the equal-effects illustration, \(\theta_j=\theta\) for all \(j\). Write \(\hat{\theta}_j=\theta+SE\,Z_j\), where each \(Z_j\) has mean zero and variance one. Dependence among estimates is allowed, but it matters: the expected maximum depends on their joint distribution, not just on the number of arms.

The selection optimism grows with uncertainty on the effect scale: multiplying all standard errors by a constant multiplies the expected bias by the same constant. It also tends to grow as more arms are considered, because taking a maximum over more noisy estimates creates more opportunity for an unusually high estimate. Correlation changes how much new information each additional estimate contributes.

Deriving the Two- and Three-Arm Results

For two independent standard normal errors, use the identity \(\max(Z_1,Z_2)=(Z_1+Z_2+|Z_1-Z_2|)/2\). The sum has expectation zero. Since \(Z_1-Z_2\sim N(0,2)\), its expected absolute value is \(2/\sqrt{\pi}\). Therefore:

For three independent standard normal errors, the density of their maximum at \(z\) is \(3\phi(z)\Phi(z)^2\), where \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. Hence its expectation is \(3\int_{-\infty}^{\infty}z\phi(z)\Phi(z)^2\,dz\). Since \(\phi'(z)=-z\phi(z)\), integration by parts gives:

For the last equality, symmetry gives \(\Phi(z)+\Phi(-z)=1\). Because \(e^{-z^2}\) is symmetric, the integral of \(\Phi(z)e^{-z^2}\) is half the integral of \(e^{-z^2}\), namely \(\sqrt{\pi}/2\). These constants apply only to independent standard normal errors; they are not universal correction factors.

Correlation From a Shared Control

Treatment-effect estimates often share a control group. This creates positive correlation: a control mean that is unusually low makes every treatment-minus-control estimate unusually high. For an exact, useful special case, suppose each standardized estimate error can be written as \(Z_j=\sqrt{\rho}\,Z_0+\sqrt{1-\rho}\,\varepsilon_j\), where \(Z_0,\varepsilon_1,\ldots,\varepsilon_K\) are independent standard normal variables and \(0\leq\rho\leq1\). The \(Z_j\) then have pairwise correlation \(\rho\).

The shared component \(\sqrt{\rho}Z_0\) shifts all estimates together and does not determine which arm is largest. Only the independent components distinguish the arms. Taking expectations yields the equicorrelated result:

This formula requires equal marginal standard errors and this equicorrelation normal structure. When \(\rho=0\), it reduces to the independent-arm result. When \(\rho=1\), all standardized errors move together, so there is no noise-based distinction among arms and the maximum has zero expected excess over the common mean.

Assumptions, Diagnostics, and Failure Modes

The closed-form calculations rely on a clearly specified selection rule and sampling model. In particular, the selected arm must be the one with the largest estimate on the same scale being analyzed; the arm estimates must be unbiased before selection; and the assumed normal distribution, standard errors, and dependence structure must be reasonable. The three-arm constants also require equal true effects and, for the independent formula, independent errors. The equicorrelation formula additionally requires the stated common-correlation structure.

Before using a formula, check how the trial actually generated the estimates. Shared controls, unequal allocation, different outcome variances, and interim selection can produce unequal variances or correlations that are not equicorrelated. If estimates have different standard errors, selecting the largest raw effect is not the same rule as selecting the largest standardized statistic, and neither constant above can simply be applied. Likewise, if arms have unequal true effects, the winner may be selected partly because it truly has a larger effect; the equal-effects bias calculation no longer isolates the full behavior of the selection rule.

Useful diagnostics include reviewing the planned randomization and analysis to identify shared data, checking the estimated covariance matrix of the arm contrasts, and comparing observed standard errors across arms. A simulation that reproduces the actual design and selection rule can estimate the expected optimism when the simple normal formulas do not apply. Such a simulation should specify the true effects and covariance structure, generate complete trial datasets, apply the selection rule to each dataset, and average the selected estimate minus the selected arm’s true effect.

Interpretation and Common Pitfalls

For equal true effects, selection does not create a real treatment difference. It creates a difference in observed estimates by choosing the arm with the most favorable noise realization. If all arm-specific estimates are unbiased, each one has expectation \(\theta\) before selection. But the maximum is a convex function, so Jensen’s inequality gives \(E(\max_j\hat{\theta}_j)\geq\max_j E(\hat{\theta}_j)=\theta\). This explains the direction of the bias even beyond the particular normal examples.

• Wrong reasoning: “Each treatment estimate is unbiased, so the selected estimate is unbiased.” Correction: unbiasedness of each estimate does not survive choosing the largest one using the same data; the selection rule favors positive errors.
• Wrong reasoning: “The observed winner is upward-biased by exactly the formula’s amount.” Correction: the formula describes an expectation over repeated trials under its assumptions. A particular selected estimate may be below, near, or far above that expected excess.
• Wrong reasoning: “Three arms always give 0.8463 standard errors of bias.” Correction: that factor is for three independent standard normal errors. Correlation and unequal standard errors change the result.
• Wrong reasoning: “A larger selected estimate proves that the selected treatment is truly better.” Correction: under the equal-effects illustration, no arm is truly better; the observed ranking is generated by sampling variation.

Selection-adjusted tests and confidence intervals address inferential error from searching across treatments; they answer a different question from the expected upward bias of the selected point estimate. The winner’s-curse calculation is an expectation under a specified model, not a universal correction to subtract from every observed estimate. It also does not provide a guarantee about the true effect in a particular trial. A calculation based on normality can be inaccurate with severe skewness, small samples, boundary-constrained estimates, or a selection rule involving several stages and outcomes.