Estimating Treatment Effects After Data-Driven Selection

Why Estimation Changes After Selecting an Arm

When several treatments are compared at an interim analysis, the arm with the largest estimated effect is often carried forward. Its estimate is then not an ordinary estimate from a randomly chosen arm: it was selected partly because its observed value was high. Even when each arm-specific estimate is unbiased before selection, the winning estimate can be systematically optimistic.

This tutorial compares three responses to that problem: reporting the selected arm’s ordinary estimate, shrinking estimates toward a common or prior mean, and subtracting an estimate of the bias induced by the selection rule. These approaches answer different questions and rely on different assumptions. A correction that reduces average selection bias is not automatically a confidence interval, and it does not guarantee that the estimate for a particular selected arm is close to its true effect.

The model is an illustrative starting point, not a universal description of an adaptive trial. Its assumptions include unbiased arm-specific estimates, a common known standard error, independent errors, and selection of the largest interim estimate. Shared controls, unequal standard errors, correlated outcomes, changing analysis populations, or a different selection rule require a model that represents those features. The selection procedures discussed earlier in the course address testing and error-rate control; estimation requires attention to the magnitude and uncertainty of the selected effect as well.

Three Estimates, Three Interpretations

The ordinary selected estimate is \(Y_J\). It retains the familiar arm-level meaning, but selection can make its error positive on average. For a fixed vector of arm effects, its unconditional selection bias is $$ E_{\theta}(Y_J-\theta_J), $$ where the expectation is over repeated trial data under that fixed vector. This definition compares the selected estimate with the true effect of the arm selected in each repetition. It does not subtract a random \(\theta_J\) from an expected estimate. A conditional bias for a particular selected arm \(j\) can instead be defined as \(E_{\theta}(Y_j-\theta_j\mid J=j)\), when that conditional quantity is relevant.

A shrinkage estimate pulls an arm estimate toward a specified center, often because estimates across arms are believed to come from a distribution. For example, if the arm effects have a normal prior distribution and the sampling model is normal, the posterior mean is a weighted average of the observed estimate and the prior mean. Shrinkage can reduce variability and extreme estimates, but ordinary per-arm shrinkage is not by itself a correction for the selection rule.

A selection-adjusted estimate explicitly accounts for optimism induced by the rule that selected the arm. One simple form is the selected estimate minus an estimated selection bias. The correction depends on the assumed model, the effects across arms, and the selection rule. If those quantities are misspecified or estimated imprecisely, the adjusted estimate can itself be biased or unstable.

Shrinkage: Useful Regularization, Not Automatic Selection Correction

A standard shrinkage calculation follows from a normal prior. Suppose, for each arm before observing the estimates, \(\theta_j\sim N(\mu,\tau^2)\), independently, and \(Y_j\mid\theta_j\sim N(\theta_j,\sigma^2)\). Here \(\mu\) is the prior mean and \(\tau^2\) describes prior variation in arm effects. Completing the square in the prior density multiplied by the sampling density gives a normal posterior whose mean is $$ E(\theta_j\mid Y_j)=\mu+B(Y_j-\mu),\qquad B=\frac{\tau^2}{\tau^2+\sigma^2}. $$ Because \(0\le B\le1\), this posterior mean lies between the prior mean and the observed estimate. When \(\tau^2\) is small relative to \(\sigma^2\), the data are noisy relative to expected between-arm differences and shrinkage is stronger.

Shrinkage and selection adjustment can move an estimate in the same direction, but for different reasons. Shrinkage uses information about plausible arm effects to moderate an extreme estimate. Selection adjustment estimates how much the selection procedure inflates the estimate on average. Shrinkage may reduce mean squared error under an appropriate prior or hierarchical model; a bias correction may reduce a particular expected bias while increasing variability. Neither has a universal advantage.

A Two-Arm Selection-Bias Derivation

A simple two-arm result shows why the selection correction depends on the difference between the true effects. Let \(D=Y_1-Y_2\), \(\delta=\theta_1-\theta_2\), and select arm 1 when \(D>0\). With independent errors of variance \(\sigma^2\), \(D=\delta+U\), where \(U=\varepsilon_1-\varepsilon_2\sim N(0,2\sigma^2)\). The error of the selected estimate can be written $$ Y_J-\theta_J=\varepsilon_2+(\varepsilon_1-\varepsilon_2)I(D>0), $$ where \(I(D>0)\) equals 1 when \(D>0\) and 0 otherwise. Taking expectations, the first term has mean zero, so the selection bias is \(E[U I(U>-\delta)]\). For \(U\sim N(0,v)\), integrating its density gives \(E[U I(U>a)]=\sqrt{v}\,\phi(a/\sqrt{v})\), where \(\phi\) is the standard normal density. Substituting \(v=2\sigma^2\) and \(a=-\delta\), and using the symmetry \(\phi(-x)=\phi(x)\), yields $$ E_{\theta}(Y_J-\theta_J)=\sqrt{2}\,\sigma\,\phi\!\left(\frac{\delta}{\sqrt{2}\sigma}\right). $$

For completeness, the integral identity follows by writing \(U=\sqrt{v}Z\), with \(Z\sim N(0,1)\), and integrating: $$ E[U I(U>a)]=\sqrt{v}\int_{a/\sqrt{v}}^\infty z\phi(z)\,dz =\sqrt{v}\,\phi(a/\sqrt{v}), $$ because \(\phi'(z)=-z\phi(z)\) and \(\phi(z)\) tends to zero as \(z\) tends to infinity. The formula shows that the bias is largest when the true effects are equal (\(\delta=0\)); when one arm is much better than the other relative to the standard error, selection is less often driven by noise and the bias is smaller.

Constructing and Checking a Selection Adjustment

For a general selection rule, define its bias at a fixed effect vector \(\boldsymbol{\theta}=(\theta_1,\ldots,\theta_K)\) by $$ b(\boldsymbol{\theta})=E_{\boldsymbol{\theta}}(Y_J-\theta_J). $$ If this bias were known, \(Y_J-b(\boldsymbol{\theta})\) would have zero expected error relative to \(\theta_J\), across repetitions under that fixed vector. Usually the vector is unknown, so a practical correction substitutes an estimate, or estimates the bias by repeatedly simulating data under a fitted model and applying the actual selection rule each time. That procedure must simulate all features that affect selection, including correlations, standard errors, interim timing, and any thresholds or tie-breaking rules.

The resulting adjusted estimate can be sensitive to the fitted effects. A useful assessment is to repeat the calculation over plausible effect vectors and model assumptions, rather than reporting a single plug-in value as if it were certain. Simulation error should also be small relative to the correction: increasing the number of simulated trials reduces Monte Carlo error, but does not fix model misspecification.

Warning signs include a correction that changes dramatically under small changes in estimated arm differences, a simulated selection frequency that does not match the implemented rule, and a model that ignores shared-control correlations or unequal standard errors. An adjusted estimate may also fall outside a plausible clinical range. That does not prove the arithmetic is wrong, but it can indicate that the correction is unstable or that its model does not adequately describe the design. In such cases, report the instability and consider a more appropriate model rather than truncating the estimate without justification.

Choosing Among the Approaches

These estimates are not substitutes for a prespecified inferential procedure. A bias-adjusted point estimate does not by itself provide a valid standard error, confidence interval, or confirmatory test. Likewise, a multiplicity-adjusted test addresses error rates for hypotheses, not necessarily the bias or accuracy of a selected point estimate. Conditional inference given the selected arm and formal coverage guarantees require additional analysis beyond this comparison.

Common Pitfalls

• Subtracting the wrong quantities. Because \(J\) is random, the fixed-effect selection bias is \(E_{\boldsymbol{\theta}}(Y_J-\theta_J)\), not \(E(Y_J)-\theta_J\) with an unspecified fixed \(\theta_J\).
• Calling shrinkage a selection correction. Pulling an estimate toward a prior mean can moderate an extreme value, but it only has the interpretation supported by its prior and sampling model.
• Treating a plug-in correction as exact. Replacing unknown effects by observed estimates adds uncertainty and can leave residual bias.
• Assuming the winner’s estimate is always too high by a fixed amount. The expected bias depends on the number of arms, their true effects, the precision and correlation of estimates, and the selection rule.
• Confusing point-estimate bias with inference. Subtracting a bias estimate does not automatically produce a confidence interval with a stated coverage probability.