Why selection changes what an estimate means
In a multi-arm, multi-stage trial, interim data may be used to decide which treatment arms continue. If an arm continues because its interim results look favorable, the estimate that helped it continue is not an ordinary, unselected estimate. It is one of the estimates that looked favorable enough to pass a comparison. Even when each arm's estimate is unbiased before selection, the selected estimate tends to be too high relative to that arm's true effect.
This phenomenon is often called selection bias or the winner's curse. It does not mean the estimate was calculated incorrectly. Nor does it say that every selected estimate must exceed its true effect. It describes a tendency over repeated trials or repeated applications of the selection rule: favorable random errors are more likely to help an arm get selected than unfavorable errors are.
That average is different from the error in one particular trial. A selected estimate might be below the true effect, just as an unselected estimate might be far above it. Selection bias is about how the selection process changes the distribution of errors, not about diagnosing the truth from one observed number.
Ordinary sampling error versus selection bias
Suppose an estimate is unbiased before selection: across repetitions, its errors average to zero, \(E(\hat{\theta}_j-\theta_j)=0\). A particular estimate can still be high or low because its data are a random sample. That deviation is ordinary sampling error. If the arm is selected using those same noisy estimates, however, high estimates have an advantage in being selected. The selected set of estimates therefore contains a disproportionate share of positive errors.
A useful, deliberately simplified argument makes the mechanism visible. Suppose two arms have the same true effect \(\theta\), and their interim estimates are \(\theta+\varepsilon_1\) and \(\theta+\varepsilon_2\). Assume the errors are independent, have mean zero, and have a continuous distribution, and select the arm with the larger estimate. The selected estimate is the maximum of the two. The identity
shows why the maximum tends to exceed \(\theta\). Taking expectations, the two mean-zero error terms contribute zero, while the absolute difference contributes a positive amount:
The strict inequality holds when there is a positive probability that the errors differ. No arm needs to have a larger true effect for the selected estimate to be optimistic. This argument illustrates the mechanism; it does not supply a general correction formula for a MAMS design.
Worked Example: One High Estimate Is Not Itself Proof of Bias
Consider a hypothetical treatment arm with a true effect of 2.0 points on a prespecified outcome scale. In one trial, its interim estimate is 4.1 points, with standard error 1.5 points. Assume, for this illustration, that the estimate is unbiased before any selection.
The observed estimation error in this trial is \(4.1-2.0=2.1\) points. This is a positive sampling error: this particular estimate is 2.1 points above the stipulated true effect. The standard error describes the estimate's sampling variability under the model; it is not the amount of bias and does not show that selection occurred.
If the arm was not chosen by comparing its interim result with other arms, the high value alone does not establish selection bias. If it was chosen precisely because its estimate was unusually favorable relative to competitors, then the selection mechanism gives a reason to expect upward bias across repetitions. The single observed excess of 2.1 points remains an error in this hypothetical realization, not a direct measurement of the long-run bias.
Selection at an interim analysis
The simple maximum argument captures an important feature of MAMS selection: an arm may continue because its interim estimate, test statistic, or other decision measure is favorable. When the same interim information is then presented as an estimate of the selected arm's effect, that estimate is being used both to select and to describe the selected arm.
The exact amount of exaggeration depends on the design and selection rule. The simple argument assumes equal true effects, equal precision, independent estimates, and selection by the largest raw estimate. Actual trials may have unequal standard errors, correlated treatment-control contrasts because arms share a control group, several stages of decisions, or selection based on a standardized statistic rather than the raw estimated effect. In those settings, the simple maximum calculation need not describe the amount of bias. The general warning still applies when favorable random variation in an estimate helps determine whether that estimate is retained.
It is also important to distinguish the estimate used to select an arm from an estimate based on new information collected after selection. If later observations are independent of the interim observations and the chosen arm's effect is estimated from those later observations alone, the selection decision does not by itself make that fresh-data estimate high on average. Combining the selected interim estimate with later data is different: the selected interim component can carry its favorable error into the combined estimate.
Worked Example: Selecting the Larger of Two Interim Estimates
In a hypothetical two-arm comparison, larger treatment effects are favorable. Both arms have a true effect of 2.0 points, and both interim estimates have standard error 1.0 point. The observed interim estimates are 4.0 points for arm A and 2.7 points for arm B. The rule continues the arm with the larger raw estimate.
Arm A is selected because \(4.0>2.7\). Its interim estimate exceeds its true effect by \(4.0-2.0=2.0\) points; arm B's estimate exceeds its true effect by \(2.7-2.0=0.7\) points. The selected estimate is therefore 4.0 points, although the stipulated effect for either arm is 2.0 points.
This one realization does not establish the expected bias or tell us the true effect in an actual study. It does show the selection logic: the rule chooses whichever estimate is higher, so it is more likely to choose an arm whose error happened to be positive (or less negative). Repeating the same procedure can produce upward bias in the selected estimate even though each arm's estimate is unbiased before selection.
What happens when interim data are combined with later data?
A common naive summary is to combine all available observations for the selected arm and report the resulting estimate as though the arm had been specified in advance. Whether that estimate inherits selection bias depends on what data entered the selection and how the estimate is formed. When selected interim data are included, the estimate can retain some of the favorable random error that helped the arm continue. New data can dilute that contribution, but do not retroactively make the interim selection disappear.
Worked Example: A Selected Interim Estimate in a Combined Estimate
Suppose a hypothetical MAMS trial compares arms A and B with a common control. For simplicity, assume the interim contrasts are independent, have equal precision, and are compared by their raw estimates; larger effects are favorable. Both arms have a true effect of 2.0 points. At interim, A is estimated at 5.0 points and B at 3.0 points, so A continues. An independent, later-stage estimate for A is 2.5 points. Assume the interim and later-stage estimates have equal information and target the same effect.
With equal information, the naive combined estimate for A is their average:
The later-stage estimate alone is 2.5 points, which is 0.5 points above the stipulated true effect in this realization. The combined estimate is 3.75 points, or 1.75 points above that true effect. The interim estimate's contribution is conspicuously favorable because A was selected using the interim comparison.
This example does not show that every combined estimate is more biased than every later-stage estimate. It illustrates why using the selected data as if they were unselected can be misleading. In a real trial, information weights, correlations, the selection rule, and the target effect all require appropriate treatment; the simple average is justified here only by the equal-information assumption.
Diagnostics, alternatives, and limitations
Selection bias is a feature of the design and analysis, not a defect that can be diagnosed just by looking for an unusually large estimate. Useful checks start by reconstructing what information determined continuation and what information is included in the reported estimate.
- Trace the data use. Record which interim estimates or statistics drove arm selection, and whether those same data contribute to the final point estimate.
- Check the selection rule. Confirm whether the decision used a raw effect estimate, a standardized statistic, a boundary crossing, or a combination of criteria. These rules select different features of the data.
- Check comparability and dependence. Unequal precision and shared controls can make arm estimates correlated or change which arm is most likely to continue. Independence and equal standard errors should not be assumed without justification.
- Compare estimates from different data sources cautiously. A later-stage-only estimate and an all-stage estimate may differ partly because they use different observations and have different precision, as well as because of selection.
The main alternatives to a naive, unadjusted estimate have different costs. A later-stage-only estimate avoids reusing the selection data if its observations are genuinely independent of the interim data, but it discards information and is usually less precise. Selection-aware estimation or inference accounts for the fact that the arm was chosen using observed data; the appropriate method depends on the selection rule and dependence structure. Simply reporting a standard error or confidence interval as though the arm had been prespecified does not, by itself, correct the point estimate's selection bias.
These distinctions matter when interpreting a selected treatment effect. An estimate can be useful for describing the data observed in a trial while still being optimistic as an estimate of the selected arm's underlying effect. The magnitude and adjustment of that optimism are not determined by the word “selection” alone; they depend on the full design, the estimand, and the analysis.
Common mistake
Incorrect reasoning: “Each arm's interim estimator is unbiased, so the estimator for the arm we continued is unbiased too.” This treats the selected arm as though it had been specified before observing the interim data.
Correction: Unbiasedness before selection does not imply unbiasedness after selection. If an estimate's favorable random error makes selection more likely, the estimates that survive selection are enriched for favorable errors. The conclusion is a tendency over repetitions, not a guarantee that the selected estimate in this particular trial is too high.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In your own words, distinguish ordinary sampling error from selection bias.
- Two arms have the same true effect and unbiased interim estimates. Why can selecting the larger estimate create an optimistic estimate even when neither estimator was biased before selection?
- A selected arm has an interim estimate of 6.0 points and a later-stage estimate of 3.0 points. The true effect is not known. What can and cannot be concluded from these numbers?
- Name two assumptions in the simple equal-arm argument that may fail in a MAMS trial, and explain why they matter.
Solutions. (1) Sampling error is the deviation in a particular estimate; selection bias is a systematic change in the average error caused by choosing an estimate using favorable data. (2) The larger estimate is more likely to include a positive random error, so the selected estimate's average can exceed the common true effect. (3) The estimates differ by 3.0 points, but without knowing the true effect, the selection rule, their precision, and their dependence, these values do not quantify the bias or prove that either estimate is too high. (4) For example, estimates may have unequal standard errors or be correlated because of a shared control. Either feature changes the behavior of the selection rule, so the simple independent, equal-precision argument cannot be used as an exact description.