Why Selection Changes the Meaning of an Estimate
Adaptive enrichment can use interim evidence to decide which biomarker-defined population to recruit or study further. That decision may be sensible for trial conduct, but it changes how the interim estimate should be read. If the population was chosen because its observed treatment effect looked especially large, the same estimate is not an ordinary, unbiased summary of the effect in that selected population. The selection rule preferentially keeps estimates with favorable random error.
This phenomenon is called selection-induced optimism, or the winner’s curse. It concerns estimation: the selected observed effect tends to exceed the corresponding true effect. It is distinct from whether the treatment comparison was randomized, whether a hypothesis test controls type I error, and whether the subgroup has a genuinely different treatment effect. A selected estimate can be based on a valid randomized comparison and still be optimistic.
The size of this tendency depends on the selection rule, the number and dependence of candidate estimates, their standard errors, and the pattern of true effects. It is not a universal fixed correction. In particular, when candidate populations have different true effects or the estimates are correlated, the amount of bias cannot be read off from the number of populations alone.
A Threshold-Selection Derivation
First consider one candidate population with an interim estimate \(X\). Suppose the true effect is \(\theta\), the standard error \(s\) is known, and the normal approximation is adequate:
Suppose the population is carried forward only when \(X>c\). This could represent a simple rule to continue only when the interim estimate exceeds a prespecified threshold. Define \(Z=(X-\theta)/s\) and \(a=(c-\theta)/s\). Then \(Z\sim N(0,1)\), and selection is the event \(Z>a\). The conditional mean error in the estimate is
where \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. The last equality follows because \(\phi'(z)=-z\phi(z)\), so \(\int_a^\infty z\phi(z)\,dz=\phi(a)\). The denominator is the probability of passing the threshold; the numerator is the average positive standardized error contributed by estimates in the selected tail. Thus, among selected trials, the average estimate exceeds the true effect even though the estimator was unbiased before selection.
This is a conditional statement. It does not say that every estimate passing the threshold is too large, or that the treatment has no effect. It says that, over repetitions under the stated model, the estimates that pass tend to include positive estimation error.
Worked Example: Continuing Only After a Large Interim Estimate
Suppose a hypothetical interim estimate of treatment benefit, measured in symptom-score points, has \(s=2.00\) points and true effect \(\theta=0\). The trial continues this population only if the estimate exceeds \(c=3.29\) points, corresponding approximately to a one-sided 5% standard-normal threshold when the true effect is zero. Assume the normal model and known standard error above.
Here \(a=(3.29-0)/2.00=1.645\). For a standard normal variable, \(\phi(1.645)\approx0.1031\) and \(1-\Phi(1.645)\approx0.0500\). Therefore:
Under this model, the average estimate among trials that pass the threshold is about 4.12 points above the true effect. Since the stipulated true effect is zero, the average selected estimate is about 4.12 points, despite the estimate being unbiased before the threshold is applied. The result illustrates the severity that conditioning on a rare favorable result can produce; it is not a correction to apply automatically to another design.
Choosing the Strongest of Several Subgroups
A related pattern arises when the trial compares several candidate subgroup estimates and selects the one with the largest apparent benefit. A particularly clear reference case assumes \(k\) independent estimates with a common true effect \(\theta\), equal known standard error \(s\), and normal sampling distributions:
If \(J\) is the index of the largest observed estimate, then \(X_J=\max_j X_j\). In this equal-effect case, the selected estimate’s expected excess over the common true effect is
The positivity follows because the maximum of several centered random errors is at least as large as each individual error, and it is strictly larger with positive probability. More explicitly, the maximum has distribution function \(P(\max_j Z_j\leq z)=\Phi(z)^k\), since the errors are independent. Its expected value can be obtained by integrating that distribution; its value increases with \(k\). The selection advantage therefore grows as more opportunities to find an unusually favorable estimate are considered.
Independence and equal effects make this a teaching model, not a universal formula for enrichment. Overlapping biomarker subgroups generally produce correlated estimates. Subgroups may also have unequal standard errors and different true effects. Under those conditions the selection bias depends on the joint distribution and the actual rule (for example, selecting the largest estimate or the largest standardized statistic). The core warning remains: an estimate chosen partly for being large contains the random fluctuation that helped it win.
Worked Example: Selecting the Largest of Three Estimates
In a hypothetical trial, three candidate biomarker subgroups have interim estimated benefits of 4.10, 1.80, and 3.00 symptom-score points. Suppose, for illustration, that their true effects are all 2.00 points, their standard errors are all 1.50 points, and their errors are independent and normal. The selection rule chooses the subgroup with the largest estimated benefit.
The observed selected estimate is 4.10 points, from the first subgroup. Its observed difference from the stipulated common true effect is \(4.10-2.00=2.10\) points. That particular difference is a feature of this hypothetical realization; it is not the expected selection bias.
For three independent standard normal variables, \(E[\max(Z_1,Z_2,Z_3)]\approx0.846\). The model’s expected excess of the selected estimate over the common true effect is therefore:
Thus, over repeated trials under the stated assumptions, the largest estimate averages about 1.27 points above the common true effect. The observed value of 4.10 points is not made unbiased by being the largest, nor does the calculation establish that the true effect in the selected subgroup is 2.00 points in an actual study. It quantifies optimism only under the stipulated reference model.
Interpreting Estimates and Intervals After Selection
A selected interim result should be described with its selection history. For example, “the estimated benefit in the subgroup selected for the strongest interim result was 4.10 points” is more informative than reporting “the subgroup effect was 4.10 points” without qualification. The first wording identifies an observed estimate and why that population was chosen. It does not claim that the estimated value is the subgroup’s true effect.
A conventional confidence interval calculated from the same data does not automatically retain its nominal coverage conditional on selection. Such an interval is ordinarily constructed for a prespecified parameter and analysis, with coverage evaluated over repetitions under those procedures. When the analyst reports an interval only for the subgroup selected because its result was large, the selection event changes the distribution of the reported estimate. Simply attaching an unadjusted interval does not undo that change. Valid inference may require a method that accounts for the selection rule, or data not used to make the selection.
This issue is separate from multiplicity control and type I error. A testing procedure may control the probability of a false-positive claim while the selected point estimate remains exaggerated. Conversely, selection-induced optimism does not by itself show that a test has an inflated type I error rate; that depends on the test and decision procedure. The earlier material on type I error under data-dependent population selection and on valid randomization comparisons addresses those distinct inferential questions.
Worked Example: Reading an Independent Validation Estimate
A hypothetical interim analysis selects a biomarker subgroup because its estimated treatment benefit is 5.20 symptom-score points. A new, independent randomized cohort is then enrolled in that selected population. In this validation cohort, the estimated benefit is 2.40 points with \(SE=1.70\) points. Assume the validation comparison is prespecified, its normal approximation is adequate, and no further selection is made using these validation results.
A conventional 95% interval based on the independent validation estimate is:
The validation estimate is 2.80 points lower than the interim estimate: \(5.20-2.40=2.80\) points. That difference is compatible with the possibility that the interim estimate was optimistic, but it does not by itself prove why the estimates differ. Under the stated independence and prespecification assumptions, the 95% interval describes uncertainty about the treatment benefit in the selected population using the validation data; it includes zero and ranges from a 0.932-point worsening to a 5.732-point benefit. It is not a 95% probability statement about the fixed true effect.
The validation estimate is not inflated merely because the population was selected using the independent interim data. However, if investigators choose among several validation estimates and report only the largest, selection is repeated and the same concern returns. Independent data protect estimation only to the extent that the reported validation analysis is not itself selected.
Diagnostics, Alternatives, and Limits
Selection-induced optimism is especially plausible when a report highlights the subgroup with the largest observed effect, when the selection threshold is stringent, or when many candidate populations were examined. A large gap between the selected estimate and estimates in other strata can also prompt scrutiny, but it is not a diagnostic proof of bias: real treatment-effect heterogeneity can produce genuine differences. Earlier material on diagnosing treatment-effect heterogeneity and unstable interaction evidence helps distinguish weak evidence of heterogeneity from stronger evidence; neither makes a selected estimate automatically unbiased.
Useful checks include documenting how many candidate populations were considered, stating whether selection used estimated effects, test statistics, or another interim criterion, and reporting the estimates for all relevant populations rather than only the winner. Also examine whether the reported standard error and interval account for the analysis actually performed. When the decision rule or interim analysis is unclear, the degree of optimism cannot be reliably quantified from the winning estimate alone.
The main alternatives address different goals. Continuing the selected population and reporting its same-data estimate is operationally efficient, but requires selection-aware inference if a calibrated estimate or interval is needed. A prespecified multiplicity-adjusted or selective-inference procedure can account for the selection rule, provided its assumptions match the design and the full rule is specified. An independent validation cohort provides a direct estimate not selected by its own interim result, but costs additional recruitment and supports inference in the validated population. These approaches are not interchangeable: adjustment targets inference under a known selection process, while validation obtains new information.
No single numerical “winner’s curse correction” applies to every enrichment design. Correlated subgroup estimates, changing standard errors, interim looks, and different true effects all alter the conditional distribution. Nor does an optimistic estimate imply that the selected population has no benefit: selection may correctly identify a population with a larger true effect, while still exaggerating its magnitude. The defensible interpretation separates the evidence for choosing a population from the accuracy of the selected effect estimate.
Check Your Understanding
Answer each question, then compare with the solutions below.
- If an unbiased normal estimate is reported only when it exceeds a fixed threshold, why is its mean among reported estimates above the true effect?
- Under the equal-effect, independent-normal model, what happens to expected optimism when the number of candidate subgroups increases?
- A subgroup is selected for a large interim estimate, and an ordinary confidence interval is calculated from that same estimate. Why is nominal conditional coverage not guaranteed?
- What assumption makes a new validation estimate useful for avoiding optimism caused by the interim selection?
Solutions. (1) Selection conditions on a positive-tail event, so positive estimation errors are overrepresented; under the normal model the conditional mean error is \(s\phi(a)/(1-\Phi(a))>0\). (2) It generally increases: the expected maximum of more independent standard normal errors is larger, so \(sE[\max_j Z_j]\) increases. (3) The interval’s ordinary coverage calculation does not condition on the event that this subgroup was chosen for a large estimate; the same data both selected and estimated the effect. (4) The validation data must be independent of the data used to select the population, and the validation estimate must not itself be selected from multiple results based on its observed magnitude.