Selection Bias in the Largest Observed Effect

Why the Largest Observed Effect Is Usually Too Large

When several treatments are estimated, selecting the one with the largest observed effect also selects partly on random error. Even if every estimate is unbiased before selection, the selected estimate tends to have a positive error: a treatment can win because its true effect is large, because its estimate happened to be high, or both. The resulting exaggeration depends on how the winner is chosen and on the variability and dependence of the estimates.

The calculation below isolates this selection effect for two estimates. It does not say that every selected estimate is exaggerated, nor that the winner’s true effect must exceed the other treatment’s true effect. It describes an expectation over repeated hypothetical trials conducted under the stated model.

Model and Assumptions

For the main result, suppose the estimates are jointly normal and unbiased before selection:

The errors may be correlated. Their variances and covariance are fixed, finite, and known for the calculation; write \(v_i^2=\operatorname{Var}(\varepsilon_i)\) and \(c=\operatorname{Cov}(\varepsilon_1,\varepsilon_2)\). For selection by the larger raw estimate, assume ties have probability zero, which holds when the difference of the estimates has positive variance. If its variance is zero, the estimates’ difference is deterministic and there is no random ranking to analyze.

Write \(\delta=\theta_1-\theta_2\), \(D=\hat{\theta}_1-\hat{\theta}_2\), and \(\tau^2=v_1^2+v_2^2-2c\). Then \(D\sim N(\delta,\tau^2)\). Selecting the largest raw estimate means \(J=1\) if \(D\geq 0\), and \(J=2\) otherwise. Normality is used for the exact formulas; unbiasedness alone does not generally determine the amount of selection bias.

Expected Maximum and Winner’s Noise Bias

The identity \(\max(\hat{\theta}_1,\hat{\theta}_2)=(\hat{\theta}_1+\hat{\theta}_2+|D|)/2\) gives the expected maximum. For \(a=\delta/\tau\), the absolute-value expectation for a normal variable is \(E|D|=2\tau\phi(a)+\delta\{2\Phi(a)-1\}\), where \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution functions. Therefore:

This is the expected reported effect when the largest raw estimate is reported. The selected treatment’s true effect is also random across repetitions because the selected index can change. Since \(P(J=1)=\Phi(a)\), its expected true effect is \(\theta_2+\delta\Phi(a)\). Subtracting this from the expected maximum gives a particularly direct expression for winner’s noise bias:

A short derivation shows why dependence enters through \(\tau\). Set \(V=\varepsilon_1-\varepsilon_2\), so \(D=\delta+V\), and \(J=1\) when \(V\geq-\delta\). For jointly normal errors, the conditional expectation of an error given \(V\) is linear in \(V\). With \(a=\delta/\tau\), the standard normal tail identity \(E[Z\,1(Z\geq-a)]=\phi(a)\) yields

Adding the two contributions gives the expected selected error. Since \(\operatorname{Cov}(\varepsilon_1,V)-\operatorname{Cov}(\varepsilon_2,V)=\operatorname{Var}(V)=\tau^2\), their sum is \(\tau\phi(a)\). Thus, positive dependence can reduce the noise in the ranking difference and hence reduce selection bias; the relevant variability is not either estimate’s standard error considered alone.

The Selection Rule Changes the Bias

The raw-estimate result applies only when selection is based on \(\hat{\theta}_1-\hat{\theta}_2\). Suppose instead the rule selects the larger standardized score, \(\hat{\theta}_i/SE_i\), while the reported quantity remains the selected raw effect estimate. If the standard errors are fixed, define the score difference \(L=\hat{\theta}_1/SE_1-\hat{\theta}_2/SE_2\), with mean \(m=\theta_1/SE_1-\theta_2/SE_2\), and let \(U=L-m\) have standard deviation \(\omega>0\). Joint normality gives the corresponding winner’s error:

This follows by writing the selected error as \(\varepsilon_1\) when \(L\geq0\) and \(\varepsilon_2\) otherwise, then applying the same normal tail identity used above. Unlike the raw-selection formula, the covariance in this expression need not equal \(\omega^2\): the score used to choose the winner is not generally the difference between the raw estimates. If standard errors are equal, the two rules rank the estimates identically. Otherwise they can choose different winners and produce different expected exaggeration.

Assumptions, Diagnostics, and Limits

The formulas are useful when two estimates have a defensible joint normal approximation, are unbiased before selection, and their error variances and covariance are adequately specified. The selection rule must be applied to the same estimates represented in the model. If the rule uses additional outcomes, constraints, or a more complicated ranking, these two-estimate expressions may not describe the resulting bias.

In an adaptive trial, dependence can arise when estimates share participants or data. A practical diagnostic is to check whether the assumed standard errors and covariance reflect that shared information and the actual analysis plan. If \(\tau\) is nearly zero for raw selection, the estimates move almost identically and their ranking is nearly determined by the true-effect difference; little bias is expected from noisy ranking. If the estimates are strongly dependent but that dependence is ignored, using \(\sqrt{v_1^2+v_2^2}\) instead of \(\sqrt{v_1^2+v_2^2-2c}\) can misstate the bias.

Normality can be doubtful with very small samples, highly skewed estimators, or boundary-constrained treatment-effect estimates. In those settings, the expected maximum may still be studied, but it should be calculated under a more appropriate joint sampling distribution, for example by simulation that reproduces both estimation and selection. With more than two candidate treatments, the maximum depends on the joint distribution of all estimates; applying a two-estimate formula to a large candidate set does not account for the additional selection.

A prespecified single treatment estimate avoids this particular selection step, though it does not remove other sources of bias or uncertainty. An independent confirmation estimate, not used to choose the winner, is another useful comparison: conditional on the selection data, an unbiased independent confirmation estimate does not inherit the same selection error. Neither alternative changes the fact that the selected estimate can be useful for choosing a treatment; it changes how cautiously its magnitude should be interpreted.

Common Mistake

A related mistake is to call \(\tau\phi(\delta/\tau)\) the expected excess over the best true effect. It is the expected error relative to the true effect of the selected treatment. The best-effect comparison also depends on the probability of selecting each treatment, as the unequal-effects example demonstrates.