Sample Splitting for Unbiased Post-Selection Estimation

Why Separate Selection From Estimation?

In a multi-arm multi-stage trial, interim data may determine which treatment arms continue. If the same noisy estimates both determine which arms look promising and supply the reported effect estimates, the selected estimates tend to be high relative to their true effects. This is selection on noise: among several arms, a high estimate can reflect a favorable random error as well as a large treatment effect.

Sample splitting addresses this particular problem by assigning independent data to two tasks. One subset determines which arm or arms are selected; a separate subset estimates the effects of those selected arms. The selected arms may still be unusual because of the selection data, but the estimation errors come from data that did not help choose them.

Definition: Let $\theta_j$ be the treatment effect for arm $j$. A selection subset is the data used to choose a set of arms $S$, or a selection path. An estimation subset is data not used in that selection decision, from which an estimator $\hat{\theta}^{\,E}_j$ is calculated for a selected arm. Sample splitting uses the first subset for selection and the second for estimation.

Model and Unbiasedness Result

For a simple formulation, write an estimate from the selection subset as $Y^S_j=\theta_j+\varepsilon^S_j$, and an estimate from the estimation subset as $Y^E_j=\theta_j+\varepsilon^E_j$. The errors need not be normally distributed for the main result below. What matters is that, conditional on the treatment effects and design, the estimation data are independent of the selection data, and $E(Y^E_j\mid\theta)=\theta_j$. Here $\theta$ denotes the vector of treatment effects.

The selected set $S$ is determined using the selection data. It can be a complicated function of interim estimates, stopping rules, or other prespecified selection information. If the estimation data are independent of all information used to form $S$, then conditioning on the selected set does not change the mean of an estimation-subset estimate.

Theorem: Suppose $S$ is determined by the selection subset and $\hat{\theta}^{\,E}_j$ is an unbiased estimator of $\theta_j$ based only on an independent estimation subset. For any arm $j$ that can be selected, $$ E\!\left(\hat{\theta}^{\,E}_j\mid S,\theta\right)=\theta_j. $$ Thus, for a selected arm $j\in S$, its estimation-subset estimate is conditionally unbiased given the selected set. The same conclusion holds when conditioning on the full selection path, provided that path is determined without using the estimation subset.

The statement is about estimating the effect of an arm whose identity has already been determined by independent data. It does not say that the arm selected by the first subset is the truly best arm, or that the estimated effect equals the true effect in a particular trial. Nor does it guarantee unbiasedness if the analysis chooses a new winner using the estimation-subset estimates themselves.

Why the Conditional Mean Is Unchanged

Let $D_S$ denote all selection-subset data and $D_E$ all estimation-subset data. Since $S$ is a function of $D_S$, conditioning on $S$ reveals information about $D_S$, but not about the independent estimation data $D_E$. If the estimator is unbiased for each possible selected arm, then for any selection event with positive probability,

\begin{aligned} E(\hat{\theta}^{\,E}_j\mid S,\theta) &= E\{E(\hat{\theta}^{\,E}_j\mid D_S,\theta)\mid S,\theta\}\\ &= E(\theta_j\mid S,\theta)\\ &= \theta_j. \end{aligned}

The second equality follows because $D_E$ is independent of $D_S$ conditional on the design and effects, and its estimator has mean $\theta_j$. The last equality holds because $\theta_j$ is a fixed parameter, not a random quantity in this model. This is the core justification: conditioning on the selection event does not condition on the estimation error.

In a randomized trial, effects are often estimated by treatment-control differences. If the estimation subset has independent treatment and control observations with means $\mu_j$ and $\mu_0$, respectively, then $\theta_j=\mu_j-\mu_0$ and $\hat{\theta}^{\,E}_j=\bar{x}^{\,E}_j-\bar{x}^{\,E}_0$. Its expectation is $\mu_j-\mu_0$. The arm-specific difference is unbiased even when several selected arms share the same estimation-subset control group; sharing the control induces correlation between differences, but does not change their individual expectations.

Worked Example: Estimating an Arm Chosen by Interim Data

Suppose three treatment arms are screened using one set of participants. The screening estimates are 0.7, 2.4, and 1.1 units of benefit for arms 1, 2, and 3. The prespecified rule selects the arm with the largest screening estimate, so arm 2 is selected. A separate estimation subset has 25 participants on arm 2 and 25 on control. Their hypothetical mean outcomes are 7.2 and 5.9 outcome units.

The estimation-subset effect estimate is $$ \hat{\theta}^{\,E}_2=\bar{x}^{\,E}_2-\bar{x}^{\,E}_0 =7.2-5.9=1.30\text{ outcome units}. $$

The value 2.4 from screening is not added to, averaged with, or substituted for the estimation-subset estimate. It determined which arm to estimate; the independent data supply the estimate. If the estimation-subset difference is unbiased for arm 2's effect and the subset was not involved in screening, then its conditional expectation given the selection of arm 2 is $\theta_2$. The observed value, 1.30 units, may still differ from $\theta_2$ through ordinary sampling error.

For example, under the additional hypothetical assumption that individual outcomes in each group have variance $9$ square outcome units and observations are independent, the standard error is $$ SE(\hat{\theta}^{\,E}_2)=\sqrt{\frac{9}{25}+\frac{9}{25}} =\sqrt{0.72}=0.849\text{ outcome units}. $$ This calculation describes sampling variability; it does not alter the conditional-unbiasedness argument.

What Sample Splitting Removes—and What It Does Not

A short calculation illustrates why reuse of the same data can create upward bias. Suppose two arms have equal true effects, both zero, and their estimates are $Y_1=\varepsilon_1$ and $Y_2=\varepsilon_2$, where the errors are independent $N(0,1)$. If the arm with the larger estimate is selected and that same estimate is reported, the reported value is $\max(Y_1,Y_2)$. Using $\max(a,b)=(a+b+|a-b|)/2$,

\begin{aligned} E\{\max(Y_1,Y_2)\} &=\frac{E(Y_1)+E(Y_2)+E|Y_1-Y_2|}{2}\\ &=\frac{0+0+2/\sqrt{\pi}}{2} =\frac{1}{\sqrt{\pi}} \approx 0.564. \end{aligned}

The reported estimate has positive expectation even though both true effects are zero. With sample splitting, the selection subset can still choose the larger screening estimate, but an independent estimate for that chosen arm has mean zero conditional on which arm was chosen. The selection-induced bias is removed under the stated independence and unbiasedness conditions; ordinary estimation uncertainty remains.

Sample splitting does not correct bias from a misspecified outcome model, confounding in a nonrandomized comparison, informative missingness, or an estimator that is biased even before selection. It also does not justify taking the largest estimate from the estimation subset after the split and treating that maximum as an unbiased estimate. Once that second selection is made, the estimation data have again been used to choose what to report.

Worked Example: Conditioning on a Randomly Selected Arm

Suppose a selection rule chooses arm 1 with probability 0.30 and arm 2 with probability 0.70. The true effects are $\theta_1=1.0$ and $\theta_2=2.0$ benefit units. For each arm that is chosen, an independent estimation-subset estimator has conditional mean equal to that arm's true effect.

Let $J$ denote the selected arm. Conditional unbiasedness means $$ E(\hat{\theta}^{\,E}_J-\theta_J\mid J=1)=0 \quad\text{and}\quad E(\hat{\theta}^{\,E}_J-\theta_J\mid J=2)=0. $$

Averaging over the two possible choices gives $$ E(\hat{\theta}^{\,E}_J-\theta_J) =0.30(0)+0.70(0)=0. $$ The expected selected true effect is $0.30(1.0)+0.70(2.0)=1.70$ benefit units, and the expected estimate is also 1.70 units. The estimate is unbiased for the effect of the selected arm, averaged over the selection process.

It would be incorrect to claim this estimate is unbiased for arm 1's effect: its expected target is the effect of whichever arm was selected, not always $\theta_1$. The target must be stated to match the selection rule.

Multi-Arm Estimation With a Shared Control

In many MAMS designs, several selected arms are compared with a common control. A split must preserve the separation between selection and estimation for the entire relevant data vector—not just for each treatment arm separately. If the control observations used for estimation also contributed to selecting the arms, the treatment-control estimates may retain selection dependence through that shared control.

Worked Example: Two Selected Arms and an Independent Shared Control

Suppose screening data select arms 1 and 3. In an independent estimation subset, 40 participants are assigned to each selected arm and 40 to a common control. Hypothetical mean outcomes are 5.6 for arm 1, 6.0 for arm 3, and 4.5 for control, measured in outcome units.

The estimates are $$ \hat{\theta}^{\,E}_1=5.6-4.5=1.10, \qquad \hat{\theta}^{\,E}_3=6.0-4.5=1.50 \quad\text{outcome units}. $$

Assume independent participant outcomes across the three groups, each with variance $4$ square outcome units. For either treatment-control difference, $$ SE=\sqrt{\frac{4}{40}+\frac{4}{40}} =\sqrt{0.20}=0.447\text{ outcome units}. $$ The estimates are individually conditionally unbiased for their respective effects because the whole estimation subset, including the common control, is independent of screening.

The two estimates are not independent: both subtract the same control mean. Their covariance is $$ \operatorname{Cov}(\hat{\theta}^{\,E}_1,\hat{\theta}^{\,E}_3) =\operatorname{Var}(\bar{x}^{\,E}_0) =\frac{4}{40}=0.100 \quad\text{square outcome units}. $$ This dependence matters for joint calculations, but it does not create bias in either estimate by itself.

Designing the Split and Checking Its Assumptions

The split should be defined before looking at outcomes that will be used for selection. A clean design assigns distinct participants, or otherwise demonstrably independent observations, to selection and estimation roles. In a multi-stage trial, one can reserve an independent subset for estimating effects after selection; the essential requirement for the result is independence from every datum used to choose the arms and the validity of the estimator on the estimation subset.

Specify the target.
State whether the target is an effect for a particular arm, the effects for all selected arms, or the effect corresponding to a selection rule. These are not interchangeable targets.

Separate the information.
Identify exactly which participants, outcomes, and interim summaries determine selection. Keep the estimation data independent of that information.

Use an unbiased estimator on the estimation subset.
For a treatment-control effect, use the appropriate difference of means or another estimator whose expectation is the stated effect under its model and design.

Preserve the separation in reporting.
Do not use estimation-subset outcomes to redefine the selected set or to choose which selected estimate is reported as the winner.

Diagnostics should focus on possible information leakage and on whether the assumptions are plausible. Check data provenance: were any estimation-subset outcomes, interim summaries, or outcome-informed exclusions used in the selection rule? Check whether the split was made at the independent sampling unit; splitting repeated measurements from the same participant across subsets may leave dependence. Check missingness and protocol deviations: if inclusion in the estimation analysis depends on an outcome in a way that changes the mean, the estimator may no longer be unbiased. Finally, verify that the treatment-control contrast and its standard error use the actual estimation-subset allocation and variance structure.

Trade-Offs and Common Pitfalls

A split spends some information on selection and leaves less information for estimating effects. With a fixed total sample size, an estimate based on only the estimation subset will generally have a larger standard error than one using all observations. That loss in precision is the direct cost of the simple independence guarantee. The split also does not automatically provide a valid interval or a simultaneous statement for several selected arms; those require their own coverage arguments.

Compared with reusing all data and applying a selection-adjusted estimator, sample splitting is conceptually direct: the selection event does not truncate the independent estimation errors. Its disadvantages are reduced precision and the need to preserve a genuinely independent subset. Methods that reuse or condition on selection data can potentially use more information, but their justification and implementation are more involved. A separately collected later cohort can also provide independent estimates, but it is not the same operational choice as dividing a fixed data set; differences in population, calendar time, or conduct may affect how well its estimate targets the original trial effect.

Pitfall: “The selected estimate is unbiased because I used a split.” The correction is that the result holds only if the estimation data are independent of all selection information and the estimation procedure is itself unbiased under its assumptions.
Pitfall: “I can choose the largest estimate in the estimation subset.” That creates a new selection step using the estimation noise. Report prespecified arm-specific estimates, or use a method that accounts for the additional selection.
Pitfall: “A shared control makes the arm estimates biased.” A shared control generally makes estimates correlated, not individually biased, provided the entire estimation subset is independent of selection and the contrasts are correctly formed.
Pitfall: “Unbiased means precise or close to the truth in this trial.” Unbiasedness is an expectation over repeated samples under a model. A particular estimate can be far from its target, especially when the estimation subset is small.

Key takeaway: Independent estimation data break the link between selecting an arm for a favorable noisy estimate and estimating that arm from the same favorable noise. Conditional unbiasedness follows when selection uses only the separate selection subset and the estimation procedure is unbiased for its stated target.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A trial uses interim data to select arm 2. Independent estimation data give means of 8.1 units for arm 2 and 6.7 units for control. What is the estimated effect, and what does conditional unbiasedness say about it?
Why does independence of the estimation subset matter when conditioning on which arms were selected?
Two selected arms use the same independent estimation control group. Does this invalidate marginal unbiasedness? What dependence does it create?
After splitting, an analyst reports whichever selected arm has the larger estimation-subset estimate. Is that estimate protected from selection-on-noise? Explain.
With two equal true effects of zero and independent $N(0,1)$ errors, what is the expected maximum of the two same-data estimates?

Solutions. (1) The estimate is $8.1-6.7=1.4$ outcome units. Conditional unbiasedness says its expectation, over repeated independent estimation subsets, equals arm 2's true effect given that arm 2 was selected; it does not say the observed 1.4 equals that effect. (2) Selection reveals information about the selection data, but independent estimation errors retain their mean-zero property under that conditioning. If the estimation data helped select the arms, this argument would fail. (3) No. Each treatment-control difference can remain marginally unbiased; the shared control induces covariance between the differences. (4) No. Choosing the largest estimate uses estimation-subset noise for a second selection, so the maximum can be upward biased. (5) The maximum has expectation $1/\sqrt{\pi}\approx0.564$, since it equals $(Y_1+Y_2+|Y_1-Y_2|)/2$, with $Y_1-Y_2\sim N(0,2)$.