Selection Bias in Reported Treatment Effects

Why the Selected Effect Can Look Too Large

In a seamless phase 2/3 trial, interim evidence may guide which treatment continues. That choice can be sensible: it concentrates later resources on a promising candidate. But if the selected treatment is reported with the same estimate that helped it win, the estimate has been chosen partly because it was unusually favorable. This is a form of selection bias, often called the winner’s curse.

The effect need not be exaggerated in every trial. A genuinely effective treatment can be selected, and its interim estimate can happen to be close to its true effect. The issue is repeated-sampling behavior: across trials following the same selection rule, the selected estimate tends to be too high relative to the selected treatment’s true effect. How large that tendency is depends on the number of candidates, how similar their true effects are, the variability of the estimates, and the selection rule.

The distinction between before selection and after selection matters. Unbiasedness of each candidate’s estimator means \(E(\hat{\theta}_j)=\theta_j\) when that candidate is considered without conditioning on its selection. It does not imply \(E(\hat{\theta}_J-\theta_J)=0\), because \(J\) depends on the observed estimates.

A Two-Candidate Derivation

A simple model shows the mechanism exactly. Suppose two treatments have the same true effect \(\theta\), and their interim estimates are independent with equal standard error \(\sigma\):

Each estimate is unbiased, since \(E(\hat{\theta}_j)=\theta\). Suppose the design selects the larger estimate and reports it, so \(\hat{\theta}_J=\max(\hat{\theta}_1,\hat{\theta}_2)\). The identity \(\max(a,b)=(a+b+|a-b|)/2\) gives a direct derivation:

The difference \(\hat{\theta}_1-\hat{\theta}_2\) is normally distributed with mean 0 and variance \(2\sigma^2\). For a zero-mean normal variable \(D\) with standard deviation \(\tau\), integration of its density gives \(E|D|=\tau\sqrt{2/\pi}\). Here \(\tau=\sqrt{2}\sigma\), so \(E|\hat{\theta}_1-\hat{\theta}_2|=2\sigma/\sqrt{\pi}\). Substitution yields:

Thus, in this particular model, the selected estimate exceeds the common true effect by \(\sigma/\sqrt{\pi}\) on average. The result relies on equal true effects, independent normal estimates, equal standard errors, and selection of the larger estimate. It is an illustration, not a universal correction factor.

Worked Estimates and Selection Rules

The magnitude and direction of selection effects are tied to what the rule rewards. Selecting the largest favorable effect estimate directly rewards positive estimation error. A rule based on a combination of effect estimates, safety, feasibility, and prespecified thresholds can produce a different pattern. Selection bias should therefore be assessed for the actual rule, not inferred from a generic formula alone.

Selection Bias Is Not the Same as Test Invalidity

An optimistic estimate and an invalid confirmatory test are different problems. Bias describes the repeated-sampling behavior of an estimator. Type I error is the probability that a testing procedure rejects \(H_0\) when \(H_0\) is true. An estimate can be selection-biased while a separately designed test still controls its Type I error.

For example, suppose an interim stage selects one candidate using only interim data. A later confirmatory stage then collects new, independent data and tests that selected candidate using a test with level \(\alpha\), under the model and assumptions specified for that test. Conditional on the interim data and hence on which treatment was selected, the new data retain their planned null distribution. If the selected treatment’s null hypothesis is true, the conditional probability of rejection is at most \(\alpha\). Averaging those conditional probabilities over possible interim outcomes also gives an unconditional probability of at most \(\alpha\).

provided the confirmatory test is calibrated under the null, the confirmatory data are independent of the data used for selection, and the tested hypothesis and analysis are specified appropriately. If several confirmatory claims are made, the relevant multiplicity requirements still apply. Independence of the fresh data does not make the interim estimate unbiased; it allows the later test to have a null distribution that is not distorted by the earlier selection.

By contrast, a nominal p-value calculated from a statistic that includes selection-favored interim evidence cannot automatically be interpreted as if all its inputs came from an unselected null distribution. For a numerical illustration, suppose an interim decision selected the candidate with the larger of two standardized interim statistics, observed as 2.1 and 0.8. A later statistic for that selected treatment is 0.8. If one mechanically forms \(Z_{\mathrm{pool}}=(2.1+0.8)/\sqrt{2}\), then:

The arithmetic is correct under the reference \(Z\sim N(0,1)\). But that tail area is not, by itself, a valid confirmatory p-value: the interim statistic 2.1 was selected as the larger of two values, so its distribution conditional on selection is not the same as an unselected standard normal statistic. This example is a warning about calibration, not a general testing recipe. A design must justify the null distribution of its actual test statistic.

Assessment, Alternatives, and Limitations

A practical assessment starts by describing the selection process, not just the final estimate. Record which candidates were eligible, what data and endpoints informed the choice, whether the rule selected the largest observed effect or used additional criteria, and whether the reported estimate comes from the selection data, fresh data, or both. Simulation under plausible treatment effects and covariance structures can quantify the behavior of the complete rule when a simple analytic formula does not apply.

Several reporting strategies answer different questions. They should not be treated as interchangeable:

Selection-aware estimation may use conditional likelihood, adjusted intervals, or other methods tailored to the selection rule. Such methods require assumptions and do not supply a universal adjustment that can be applied after the fact. Likewise, a valid confirmatory test does not prove that an effect is clinically important, that the model is adequate, or that every reported estimate is unbiased. Selection bias can also be small when candidates are clearly separated in true effects, but it need not disappear: noisy estimates can still influence which treatment wins.

Diagnostics and Common Mistakes

Warning signs include reporting only the winning interim estimate while omitting the competing estimates, presenting its standard error as if no selection occurred, or treating a nominal p-value as confirmatory without explaining how its null distribution accounts for the design. A useful diagnostic is to compare the selected estimate with estimates from independent confirmatory data and to simulate the full selection rule under scenarios where candidate effects are equal or close. Large discrepancies or inflated rejection rates under a global null indicate that the reporting or testing procedure needs further scrutiny.

• Mistake: “Each treatment estimator is unbiased, so the selected treatment’s estimate is unbiased.” Correction: Selection depends on the observed estimates. Choosing a large estimate tends to choose positive estimation error as well.
• Mistake: “The interim effect is optimistic, so any later confirmatory test must be invalid.” Correction: Bias in the selected estimate does not alone determine test validity. A properly calibrated test based on independent confirmatory data can retain its stated Type I error.
• Mistake: “A standard error or p-value computed by an ordinary formula automatically accounts for selection.” Correction: Its reference distribution must match the actual selection and analysis procedure. A numerical tail area is not evidence of calibration by itself.