Common Mistakes in Post-Selection Estimation

Why Post-Selection Errors Matter

In a multi-arm multi-stage (MAMS) trial, interim results may determine which treatment arms continue. The treatment effect ultimately reported for a retained or selected arm is therefore linked to a data-dependent decision. The central caution is that a familiar estimator or interval does not become selection-adjusted merely because it is calculated after the selection decision.

Let \(J\) denote the index of the arm selected under the trial's prespecified rule, and let \(\hat{\theta}_j\) denote an estimate of the treatment effect for arm \(j\). The reported estimate is \(\hat{\theta}_J\). Its interpretation and sampling properties depend on how \(J\) was chosen, which data contributed to \(\hat{\theta}_J\), and how estimates across arms are dependent. Confusing any of these can produce overstated effects, incorrect uncertainty, or intervals with misleading coverage.

Definition: A post-selection estimate is an estimate reported for an arm chosen by a rule that depends on trial data. The selection event for arm \(j\) is the full set of conditions on the data under which the rule selects arm \(j\), including any eligibility, stopping, or ranking conditions.

The examples below use normal models to make the calculations transparent. Unless an example says otherwise, assume the model is correctly specified, observations are independent across treatment groups, and the variance used in a calculation is known. In an actual MAMS analysis, the selection rule and any shared-control dependence must also be represented correctly; normality alone does not remove selection effects.

Mistake 1: Treating the Selected Estimate as Prespecified

A prespecified arm and a selected arm are not interchangeable. When an arm is selected because its observed estimate is favorable, its estimate tends to include favorable random error. This does not imply that every selected estimate is exaggerated, but it means that ordinary reasoning for a fixed arm does not automatically apply to the selected one.

Worked Example: The Winner of Two Noisy Estimates

Suppose two arms have the same true effect, \(\theta=0.4\) units, and independent interim estimates \(X_1,X_2\sim N(0.4,1)\). The arm with the larger estimate is selected. What is the expected value of the selected estimate?

Write \(X_1=0.4+Z_1\) and \(X_2=0.4+Z_2\), where \(Z_1,Z_2\) are independent \(N(0,1)\) variables. The selected estimate is \(\max(X_1,X_2)\). Using \(\max(a,b)=(a+b+|a-b|)/2\),

E\{\max(X_1,X_2)\} =0.4+\frac{1}{2}E|Z_1-Z_2| =0.4+\frac{1}{\sqrt{\pi}} \approx 0.964.

Here \(Z_1-Z_2\sim N(0,2)\), whose mean absolute value is \(2/\sqrt{\pi}\). Thus, although each arm-specific estimate is unbiased before selection, the expected selected estimate is about \(0.564\) units above the common true effect. This calculation illustrates selection-induced optimism; it is not a correction formula for every MAMS design.

The issue is especially relevant when an estimate combines data from the selection stage with later data. If the estimate uses only new, independent stage-two observations for a fixed retained arm, selection based on stage one does not by itself make those new observations biased for that arm's effect under the stated model. But it remains important to account for the design when interpreting the overall analysis; pooling stage-one data, selecting on a statistic correlated with the reported estimate, or selecting among several arms can change the relevant sampling distribution.

Mistake 2: Ignoring the Covariance From a Shared Control

Treatment-control contrasts in a multi-arm trial commonly reuse the same control observations. The resulting estimates are correlated even if the treatment groups are mutually independent. Treating those contrasts as independent can give an incorrect standard error for comparisons between arms or for a selection-related calculation.

Worked Example: Two Contrasts With One Control Group

Suppose each of two treatment groups and the shared control group contains 40 independent observations. The outcome has variance \(\sigma^2=100\) in every group. Let \(\hat{\theta}_1=\bar{X}_1-\bar{X}_C\) and \(\hat{\theta}_2=\bar{X}_2-\bar{X}_C\), where the bars are group means.

The treatment groups and control group are independent of one another, but both contrasts contain the same control mean. Therefore,

\operatorname{Cov}(\hat{\theta}_1,\hat{\theta}_2) =\operatorname{Cov}(\bar{X}_1-\bar{X}_C,\bar{X}_2-\bar{X}_C) =\operatorname{Var}(\bar{X}_C) =\frac{100}{40}=2.50.

Each contrast has variance \(100/40+100/40=5.00\), so their correlation is \(2.50/\sqrt{5.00(5.00)}=0.500\). Now consider their difference. The shared control cancels:

\hat{\theta}_1-\hat{\theta}_2=\bar{X}_1-\bar{X}_2, \qquad \operatorname{Var}(\hat{\theta}_1-\hat{\theta}_2)=\frac{100}{40}+\frac{100}{40}=5.00.

If the two contrasts were incorrectly treated as independent, the calculated variance of their difference would be \(5.00+5.00=10.0\), twice the correct value. The numerical direction of an error depends on the calculation: omitting a positive covariance can inflate or deflate a variance depending on the signs with which the estimates enter it. The covariance structure, rather than a general rule about whether dependence is conservative, must determine the calculation.

Mistake 3: Conditioning on the Wrong Event

A selection adjustment must reflect the event that actually selected the arm. If the rule selects the largest eligible interim statistic, the event for arm 1 is that its statistic exceeds the other eligible statistics, together with any required eligibility conditions. The event is not simply that arm 1's statistic is positive, nor is it necessarily that it exceeds a fixed threshold.

Worked Example: Winner Versus Positive Score

For a simple illustration, let two interim selection scores \(S_1,S_2\) be independent \(N(0,1)\) variables, and select arm 1 exactly when \(S_1>S_2\). Compare the conditional mean of \(S_1\) under the actual selection event with its conditional mean given only \(S_1>0\).

For the actual event, set \(D=S_1-S_2\). Then \(D\sim N(0,2)\), and \(S_1=(S_1+S_2)/2+D/2\). The sum and difference are independent, so

E(S_1\mid S_1>S_2) =\frac{1}{2}E(D\mid D>0) =\frac{1}{2}\sqrt{\frac{4}{\pi}} =\frac{1}{\sqrt{\pi}} \approx 0.564.

By contrast, conditioning only on a positive score gives

E(S_1\mid S_1>0)=\sqrt{\frac{2}{\pi}}\approx 0.798.

These are different conditional distributions because they refer to different events. In the actual selection rule, arm 1 can be selected even when both scores are negative, provided \(S_1>S_2\). Conversely, \(S_1\) can be positive without arm 1 being selected. In a MAMS analysis, the event must follow the rule actually used, including shared-control dependence and any dropping or stopping conditions.

A useful diagnostic is to write the selection event in words and then as inequalities before doing any conditional calculation. Check that the event includes every comparison and threshold that determined the selected arm, and that it does not add conditions that were absent from the rule.

Mistake 4: Reporting an Unadjusted Interval as Though Selection Did Not Occur

A confidence interval can have its advertised coverage for a prespecified arm under a model and still fail to have that coverage for an arm chosen using the same estimates. The following example isolates the issue. It uses independent estimates rather than shared-control contrasts, so it does not claim to give the exact coverage for a particular MAMS design.

Worked Example: Coverage After Selecting the Largest of Three Estimates

Suppose three independent arm estimates \(X_1,X_2,X_3\) have a common true effect \(\theta=0\), known standard error 1, and normal distributions \(N(0,1)\). Select the arm with the largest estimate, and report the usual nominal 95% interval \(X_J\pm1.96\). For a prespecified arm, this interval has approximately 95% coverage.

By symmetry, the probability that arm 1 is selected is \(1/3\). Conditional on \(X_1=x\), it is selected when both other estimates are less than \(x\), an event with probability \(\Phi(x)^2\), where \(\Phi\) is the standard normal cumulative distribution function. Therefore, its conditional coverage given that arm 1 is selected is

\begin{aligned} P(-1.96<X_1<1.96\mid J=1) &=3\int_{-1.96}^{1.96}\phi(x)\Phi(x)^2\,dx\\ &=\Phi(1.96)^3-\Phi(-1.96)^3\\ &\approx 0.9750^3-0.0250^3\\ &\approx 0.9268. \end{aligned}

The integral uses \(d\{\Phi(x)^3\}/dx=3\phi(x)\Phi(x)^2\). Thus, in this example, the ordinary interval covers the selected arm's true effect about 92.68% of the time under repeated sampling, not 95%. This is a repeated-sampling statement under the specified model; it is not a probability that the fixed parameter is inside the interval after the data have been observed.

A selection-aware interval or analysis must use a method appropriate to the design and inferential target. Depending on the goal, relevant alternatives include an estimate based on independent post-selection data, a method that accounts for the selection event, or simultaneous inference across candidate arms. These approaches answer different questions and rely on different assumptions; none should be chosen by label alone. In particular, using a wider interval does not automatically make it valid if its construction ignores the actual selection rule or dependence.

Common Mistakes and Practical Checks

Wrong: “The estimate is unbiased for each arm, so it is unbiased after I select the largest one.” Correction: Unbiasedness before selection does not generally carry over to an estimate selected for an extreme observed value.
Wrong: “The two treatment-control estimates use different treatment groups, so they are independent.” Correction: A shared control mean induces covariance; include it in the joint variance calculation.
Wrong: “The arm was selected because its score was positive, so conditioning on a positive score describes the selection.” Correction: State the actual rule, including comparisons with other arms and any eligibility or stopping boundaries.
Wrong: “A 95% interval for a fixed arm is automatically a 95% interval for whichever arm is selected.” Correction: Check coverage under the selection procedure and the intended inferential target.

When reviewing an analysis, compare the recorded selection rule with the event used in the estimator or interval calculation. Also inspect the covariance matrix of the treatment contrasts and verify which stage data enter the reported estimate. If a simulation diagnostic is used, it should reproduce the allocation, shared controls, interim decisions, stopping, and estimation steps relevant to the claim. A simulation that omits the actual selection process cannot diagnose selection-related error.

These checks do not remove the limitations of any particular post-selection method. Results may depend on the outcome model, variance assumptions, and exact selection rule; estimates can also be unstable when few arms remain eligible or selection is highly competitive. The key is to report what was selected, how it was selected, what data were used for estimation, and what uncertainty calculation corresponds to that procedure.

Check Your Understanding

Answer each question using the assumptions and calculations in this tutorial.

Two independent \(N(\theta,1)\) estimates are available and the larger is selected. What is the expected selected estimate relative to \(\theta\)?
In the shared-control example, what is the covariance between the two treatment-control contrasts?
Why is the event \(S_1>0\) not the same as selecting arm 1 when the rule chooses the larger of \(S_1\) and \(S_2\)?
What coverage did the unadjusted nominal 95% interval have in the three-arm example?

Solutions. (1) The expected selected estimate is \(\theta+1/\sqrt{\pi}\), approximately \(\theta+0.564\), because selection favors positive noise. (2) The covariance is \(\sigma^2/n_C=100/40=2.50\). (3) A positive \(S_1\) need not exceed \(S_2\), and \(S_1\) can be selected even when both scores are negative; the selection event is \(S_1>S_2\). (4) Its coverage was \(\Phi(1.96)^3-\Phi(-1.96)^3\approx0.9268\), or about 92.68%, under the stated independent normal model.