Why Tuning Can Reintroduce Selection Bias
Shrinkage pulls an estimated treatment effect toward a target, often zero or a common mean. With a fixed factor, the pull is transparent: a factor between zero and one reduces the magnitude of the estimate. But in a multi-arm multi-stage trial, the arms available for estimation have often been selected using interim or final results. If the shrinkage factor is also estimated from those selected results, it becomes data-dependent. Large, selection-inflated estimates can then influence both the estimate being shrunk and how much shrinkage is applied.
The central issue is not that estimating a shrinkage factor from data is always wrong. It is that a tuning procedure must reflect how the data were generated and selected. A tuning rule that ignores adaptive selection may target the wrong amount of between-arm variation and may produce a shrinkage estimate with substantial conditional bias. A factor that looks reasonable in the observed data is not, by itself, evidence that the resulting estimate is well calibrated.
For a simple illustration, let \(X\) be an estimate of a treatment effect \(\theta\), with \(X\sim N(\theta,v)\), where \(v\) is known. A shrinkage estimator toward a target \(\mu_0\) is \( \delta=\mu_0+\lambda(X-\mu_0) \), where \(0\leq\lambda\leq1\). A smaller \(\lambda\) pulls more strongly toward \(\mu_0\); \(\lambda=1\) leaves \(X\) unchanged. This normal model is a teaching approximation. In a MAMS trial, stagewise estimates can have different standard errors and can be correlated, for example because arms share a control group.
Selection changes the relevant distribution. If an estimate is reported only when it crosses a threshold, its distribution among reported estimates is not its original sampling distribution. A fixed factor does not remove that selection effect automatically, and a random factor adds another feature to account for.
What a Data-Dependent Factor Adds
Let \(S\) denote the event that an arm is selected, and suppose \(\lambda\) is estimated from the same data as \(X\). Conditional on selection, the mean of the shrunken estimate is \( E(\delta\mid S)=\mu_0+E\{\lambda(X-\mu_0)\mid S\} \). If both \(\lambda\) and \(X\) vary, the product cannot generally be replaced by the product of their conditional means. The exact decomposition is:
The covariance term is the extra feature introduced by data-dependent tuning. If the factor tends to be larger for larger selected estimates, that term is positive when the target is fixed. This can offset the intended shrinkage. If the factor tends to be smaller for larger estimates, the covariance can instead add shrinkage. Its direction and size depend on the tuning rule and the selection mechanism.
This identity is a justification, not a recipe for correcting bias: it involves conditional quantities that may be unknown. Its practical message is that evaluating a random shrinkage factor as if it were a fixed constant can be misleading. The full procedure—including arm selection and factor estimation—needs to be assessed together.
Worked Example: A Fixed Factor After Threshold Selection
Suppose a hypothetical arm has true effect \(\theta=0.5\) units, and its estimate satisfies \(X\sim N(0.5,1)\). The arm is selected if \(X>1\). Shrink the selected estimate toward zero using a fixed factor \(\lambda=0.70\). Find the conditional mean and bias of the shrunken estimate among selected arms.
The selection threshold is one standard error above zero but only \(0.5\) standard errors above the true mean. For a normal variable, the mean above a threshold \(c\) is \( E(X\mid X>c)=\theta+\sigma\phi(a)/\{1-\Phi(a)\} \), where \(a=(c-\theta)/\sigma\), and \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. Here \(a=(1-0.5)/1=0.5\), so:
Because the shrinkage factor is fixed and the target is zero, \(E(\delta\mid X>1)=0.70E(X\mid X>1)\approx0.70(1.6411)=1.1488\) units. Its conditional bias is approximately \(1.1488-0.5=0.6488\) units. Using the displayed rounded mean instead gives \(1.149-0.5=0.649\) units. Thus, shrinkage reduces the selected estimate’s conditional mean relative to the unshrunk mean of about \(1.6411\), but it does not eliminate the upward conditional bias.
Why Naïve Plug-In Tuning Can Mislead
A common tuning idea comes from a normal hierarchical model: treatment effects vary around a common target with between-arm variance \(\tau^2\), while estimates have sampling variance \(v\). If the model and data structure are appropriate, a familiar shrinkage factor is \( \lambda=\tau^2/(\tau^2+v) \). A larger estimated \(\tau^2\) implies less shrinkage; an estimate near zero implies strong shrinkage.
The difficulty is that the variation among selected estimates is not generally the variation among all candidate-arm estimates. Selection may truncate the observed distribution, and selection decisions can depend on several stages and correlated estimates. Estimating \(\tau^2\) from the selected estimates alone and treating that estimate as though it came from an unselected sample can therefore misstate the amount of heterogeneity. With few selected arms, the estimate can also be unstable for the ordinary reason that a small sample contains little information about a variance.
Worked Example: A Naïve Variance Estimate From Selected Arms
Suppose, hypothetically, that two selected arms have estimated effects of \(2.0\) and \(2.4\) units, each with known sampling variance \(v=1\) square units. Consider a naïve method-of-moments estimate that subtracts the sampling variance from the sample variance of these two estimates, truncates the result at zero, and then uses \(\hat\lambda=\hat\tau^2/(\hat\tau^2+v)\). Calculate the factor and explain the limitation.
The sample mean is \((2.0+2.4)/2=2.2\) units. With two observations, the sample variance has denominator \(2-1=1\): \(s^2=\{(2.0-2.2)^2+(2.4-2.2)^2\}/1=(0.04+0.04)=0.08\) square units. The proposed plug-in estimate is therefore \( \hat\tau^2=\max(0,0.08-1)=0 \) square units, and \( \hat\lambda=0/(0+1)=0 \).
Under this rule, both estimates would be shrunk all the way to the target. That is an arithmetic consequence of the stated rule, not evidence that the true treatment effects are identical. The calculation has only two selected estimates and uses no adjustment for how they were selected. In particular, the sample variance of selected estimates need not estimate the variance of all candidate-arm effects. A selection-aware analysis could reach a different conclusion, but its answer would depend on the specified model and the selection information available.
This example also shows why “estimate the factor from the trial data” is not a complete method description. One must specify which arms and stages enter the tuning calculation, how their dependence is handled, and whether selection is represented. In MAMS settings, ignoring shared-control correlations or the distinction between interim and final estimates can further distort a variance estimate.
Defensible Ways to Tune Shrinkage
Several approaches can be defensible, depending on the estimand, available information, and purpose of the analysis. None makes shrinkage automatically unbiased. The key distinction is whether the rule accounts for the adaptive selection that generated the estimates.
| Strategy | How it is chosen | Main strength | Main caution |
|---|---|---|---|
| Prespecified factor | Choose the factor before observing selected effects, using subject-matter knowledge or external information. | Avoids direct tuning to the selected estimates. | May be poorly matched to the trial’s true heterogeneity. |
| Design-calibrated factor | Choose a rule or factor by evaluating the full design under specified effect patterns, selection rules, and loss function. | Can target performance across realistic selection scenarios. | Depends on the scenarios and loss function used. |
| Selection-aware model-based tuning | Estimate model parameters using a likelihood or hierarchical model that represents candidate arms and the selection process. | Uses data while accounting for selection and dependence under the model. | Can be sensitive to model assumptions and limited selected data. |
| Naïve selected-sample plug-in | Estimate heterogeneity from selected estimates as if they were an ordinary unselected sample. | Simple to compute. | Can misestimate heterogeneity and the shrinkage factor after adaptive selection. |
A selection-aware tuning procedure needs a clear account of what was observed and what determined selection. For example, if a likelihood is conditioned on selection, its distribution must reflect the selection event; if the full candidate-arm data and path are modeled jointly, the selection rule and relevant dependence must be included in the joint model. The appropriate construction depends on the design and inferential target. Merely applying a familiar variance formula to selected results does not make it selection-aware.
Worked Example: Choosing Between Prespecified and Selection-Aware Tuning
A hypothetical trial team expects moderate variation in treatment effects across arms and wants to shrink selected estimates toward zero. Before the trial, its external information suggests \(\tau^2=1\) square units. Suppose a typical arm estimate has sampling variance \(v=1\) square units. Compare the resulting prespecified factor with the factor from the naïve selected-sample calculation in the preceding example, and state what can and cannot be concluded.
Using the external value in the normal hierarchical-model formula gives \( \lambda=1/(1+1)=0.50 \). This prespecified factor shrinks an estimate halfway toward zero, provided the team fixes the value before seeing the selected effects. It does not guarantee unbiasedness conditional on selection, as the first example demonstrated for a different factor.
The naïve calculation from the two selected estimates gave \(\hat\lambda=0\), which would shrink completely to zero. The discrepancy, \(0.50\) versus \(0\), arises because the two procedures use different information and assumptions. It is not resolved by choosing whichever factor appears more plausible after seeing the results. The team should either justify the external factor in advance or specify and evaluate a model that accounts for arm selection and the relevant data structure. The example supplies no selection-aware estimate, so it does not establish which factor is optimal for the hypothetical trial.
Diagnostics, Limitations, and Common Pitfalls
A useful diagnostic is to examine the whole tuning procedure under the selection mechanism, rather than just inspect the fitted factor. In design-based simulations, specify plausible patterns of true arm effects, sampling variances, correlations, interim decisions, and final selection. For each simulated trial, repeat both selection and tuning, then summarize conditional bias among selected arms as well as unconditional performance over all simulated trials. These are different operating characteristics: a rule can behave differently for selected arms than it does on average across all trials.
Other warning signs include a factor at a boundary (near zero or one), large changes in the factor when one selected arm is removed, implausibly little estimated heterogeneity with very few selected arms, and differences between analyses that include or omit shared-control dependence. Such signs do not prove that tuning is invalid, but they indicate that the result may be driven by a fragile estimate or a misspecified model.
- Pitfall: “The factor is below one, so the selected estimate is unbiased.” Shrinkage reduces the magnitude of an estimate relative to its target, but the selected distribution may still be shifted. The first worked example has a fixed factor of \(0.70\) and conditional bias of about \(0.6488\) units.
- Pitfall: “A plug-in factor is fixed once calculated, so its randomness does not matter.” In a repeated-trial evaluation, the calculated factor varies and may be associated with the selected estimates. Its covariance contribution cannot generally be ignored.
- Pitfall: “A selected-sample variance is automatically a between-arm variance estimate.” Selection changes the distribution being summarized. A variance estimator justified for an unselected sample may not retain that justification after adaptive selection.
The alternatives also have limitations. A prespecified factor avoids tuning directly to the selected estimates but can be wrong if external information is not transportable. Design calibration depends on which effect patterns and losses are considered important. Selection-aware models can use trial information efficiently, but their conclusions rely on the assumed effect distribution, standard errors, dependence structure, and selection model. Shrinkage is therefore best understood as a way to trade variance against bias under explicit assumptions, not as a universal correction for winner’s-curse bias.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In \( \delta=\mu_0+\lambda(X-\mu_0) \), what does \(\lambda=0.4\) mean, and why does it not imply conditional unbiasedness after selection?
- Suppose \(\lambda\) is random and tends to be larger for larger selected estimates. Which term in the conditional-bias decomposition captures this relationship?
- Two selected estimates have sample variance \(0.30\) square units and known sampling variance \(0.80\) square units. Under the naïve rule \(\hat\tau^2=\max(0,s^2-v)\), find \(\hat\tau^2\) and the corresponding \(\hat\lambda\).
- Name one advantage and one limitation of choosing a shrinkage factor through design calibration.
- Why should simulations of a tuning rule repeat arm selection as well as factor estimation?
Solutions. (1) The estimate is moved 40% of the way from the target toward \(X\), equivalently retaining 40% of its deviation from \(\mu_0\). Selection can shift the distribution of \(X\), so this linear pull need not make its conditional mean equal to \(\theta\). (2) The term is \(\operatorname{Cov}(\lambda,X-\mu_0\mid S)\); under the stated tendency, it is generally positive. (3) \(\hat\tau^2=\max(0,0.30-0.80)=0\) square units, so \(\hat\lambda=0/(0+0.80)=0\). (4) Calibration can assess performance under the adaptive design; however, the result depends on the effect patterns and loss function chosen for calibration. (5) Both selection and tuning vary across repeated trials, and their dependence affects the selected estimates’ bias and the estimated factor. Fixing selection in the simulation would fail to assess that joint procedure.