Deriving Winner’s-Curse Bias for Normal Estimates

Why selecting the largest estimate creates bias

When several treatment arms are compared, the arm with the largest observed effect estimate can look unusually promising partly because its estimate includes random error. Even if the arms have identical true effects, selecting the maximum tends to select an estimate with a positive error. This is the winner’s curse: the selected estimate is, on average, higher than the common true effect.

Here we isolate that mechanism in a deliberately simple setting. Assume each arm’s estimate is normally distributed, all arms have the same true effect and standard error, and the estimates are independent. This is not a complete model of every multi-arm multi-stage design. It is a setting in which the selection effect can be derived directly and its size made explicit.

The model assumes a fixed common effect, the same known standard error for every arm, normal sampling distributions, and independence across arms. The selection rule is to choose the largest estimate, with no additional eligibility or stopping condition. Because independent continuous normal variables tie with probability zero, the largest estimate identifies a unique arm with probability one. The calculation concerns repeated applications of this model, not a claim that every selected estimate in an individual trial must exceed \(\theta\).

Deriving the expectation of the maximum

Standardize each estimate as \(Z_i=(X_i-\theta)/\sigma\). Under the assumptions above, the \(Z_i\) are independent standard normal variables. Define \(M_K=\max(Z_1,\ldots,Z_K)\). Since \(X_{(K)}=\theta+\sigma M_K\), it follows that

Thus the bias is the common standard error multiplied by a number that depends only on the number of arms. To derive that number, let \(\Phi(z)\) and \(\phi(z)\) denote the standard normal cumulative distribution function and density. The event \(M_K\le z\) occurs exactly when all \(K\) standardized estimates are at most \(z\). Independence gives

Differentiating this cumulative distribution function gives the density of the maximum. Integrating the value of the maximum against that density yields an expression for its expectation:

An equivalent expression uses the upper and lower tails. For an integrable random variable, its expectation is the integral of its positive-tail probabilities minus the integral of its negative-tail probabilities. Applying this identity to \(M_K\), and using \(P(M_K>t)=1-\Phi(t)^K\) and \(P(M_K<-t)=\Phi(-t)^K\) for \(t\ge0\), gives

These integrals can be evaluated numerically for a given \(K\). They show directly that the bias grows with the scale \(\sigma\), while the standardized amount \(E[M_K]\) is determined by how many estimates are compared.

What changes as the number of arms increases?

For \(K>2\), the same integral gives the standardized expected maximum. For example, numerical evaluation gives \(E[M_3]\approx0.8463\) and \(E[M_4]\approx1.029\). These are positive: with more independent estimates competing for selection, there are more opportunities for a large positive random error to produce the maximum. The expectation increases with \(K\), although it does not increase by a fixed amount for each additional arm.

Interpreting the result and checking its assumptions

The earlier material on representing arm selection as a statistical event expresses selection through the event that a particular arm has the largest estimate. The calculation here supplies the resulting expected estimate in the special case of independent, equal-precision normal estimates with a common true effect. Its central result is:

The positivity also has a direct justification. For \(K\ge2\), \(M_K\ge Z_1\) in every realization and \(P(M_K>Z_1)>0\). Since \(E[Z_1]=0\), taking expectations gives \(E[M_K]>0\). In addition, adding another independent estimate cannot reduce the maximum: \(\max(Z_1,\ldots,Z_{K+1})\ge\max(Z_1,\ldots,Z_K)\). Hence the expected standardized maximum is nondecreasing in the number of arms.

In practice, check whether the data analysis truly fits the assumptions before applying this result. In particular, ask whether arm estimates are approximately normal, whether their standard errors are the same, whether their errors are independent, and whether all arms can reasonably be treated as having a common true effect for this calculation. The formula is a reference calculation for this model, not a universal correction to apply mechanically.

• Dependence: estimates that share information may be correlated. Then the product \(\Phi(z)^K\) is not their joint probability of all being at most \(z\), so the derivation does not apply as written.
• Unequal standard errors: the largest estimate need not have the same distributional scale as the others. A single common \(\sigma\) cannot be factored out in the stated way.
• Unequal true effects: when arms have different true effects, selecting the largest estimate is not the same as selecting among estimates of one common \(\theta\). The bias relative to a particular target then needs to be defined for that setting.
• Different selection rules: selection based on a threshold, an interim stopping rule, or multiple stages is not simply selection of the maximum from \(K\) estimates under this model.

These are not small technical details: changing them can change the distribution of the selected estimate. The result also does not determine the variance or a confidence interval for the selected estimate. It quantifies one feature only: the expectation of the maximum under the specified model.

A common mistake and the appropriate comparison

A frequent error is to calculate the expected estimate of each arm first and then conclude that the expected selected estimate is also \(\theta\), because every arm has expectation \(\theta\). That argument overlooks that the selected arm’s index is itself chosen using the observed estimates. In general, the expectation of a maximum is not the maximum of the expectations: here, \(\max_i E[X_i]=\theta\), but \(E[\max_i X_i]=\theta+\sigma a_K>\theta\) for \(K\ge2\).

If one arm is specified in advance and its estimate is reported regardless of its observed value, its expectation under this model is \(\theta\); the maximum-selection bias does not arise from that fixed choice. If the largest estimate is chosen and reported, the expectation instead includes the positive selection term. These are different procedures and answer different questions. In a MAMS setting, the simple maximum calculation is useful for seeing why post-selection estimates can be optimistic, but a design with correlated contrasts, changing stages, or unequal arm effects requires a model aligned with those features.