Interim Estimates Versus Final Selected-Arm Estimates

Why the Final Estimate Need Not Be Free of Selection Bias

In a multi-arm multi-stage design, interim outcomes can determine which treatment arm continues. The final estimate for that arm may then use both the interim data that drove the selection and additional outcomes collected afterward. These two sources of information play different roles: the first is involved in selecting the arm, while the second may be independent of that decision. Adding new data can reduce the influence of the selection-affected estimate, but it does not automatically erase it.

The key question is therefore not simply whether more outcomes were collected. It is how the final estimator weights the interim estimate and the post-selection estimate, and whether the latter is independent of the selection decision. The discussion below focuses on a single selected arm and a common treatment-effect estimand across stages. It does not derive a general selection correction.

Definition: For candidate arm $k$, let $\hat\theta_{k,1}$ be its treatment-effect estimate from the interim data, with variance $V_1$. Let $\hat\theta_{k,2}$ be an estimate from additional post-selection data, with variance $V_2$. The selected arm is $J$, determined using interim information. The pooled final estimate $\hat\theta_{J,F}$ combines the interim and post-selection estimates for arm $J$. Selection bias conditional on an event $A$ is $E(\hat\theta_{J,F}-\theta_J\mid A)$, where $\theta_J$ is the true effect of the arm selected under that event.

Model and Conditions for the Pooling Argument

Assume that both stages estimate the same effect on the same scale and in the same target population. For each arm $k$, suppose $\hat\theta_{k,1}\sim N(\theta_k,V_1)$ and $\hat\theta_{k,2}\sim N(\theta_k,V_2)$. For the basic result, assume that the post-selection estimate is independent of all interim data used to choose $J$, and that the stage-specific estimates have known variances. In particular, conditional on the interim data and the selected arm, the post-selection estimate has mean $\theta_J$.

These assumptions describe a common setting where stage data come from disjoint participants, outcome measurement and analysis are comparable across stages, and the selected arm is chosen before the additional outcomes are observed. In practice, the variance estimates may be estimated rather than known; the inverse-variance formula is then a model-based approximation whose accuracy depends on those estimates. The independence and common-estimand conditions remain central.

When the stage estimates are independent and unbiased before selection, the inverse-variance weighted pooled estimate is

\hat\theta_{k,F} = \frac{\hat\theta_{k,1}/V_1+\hat\theta_{k,2}/V_2} {1/V_1+1/V_2} = w_1\hat\theta_{k,1}+w_2\hat\theta_{k,2}, \qquad w_1=\frac{V_2}{V_1+V_2}, \quad w_2=\frac{V_1}{V_1+V_2}.

The weights sum to one and give more weight to the estimate with smaller variance. The pooled variance under the model is $V_1V_2/(V_1+V_2)$. After selection, the estimate of interest is $\hat\theta_{J,F}$, using the data for the arm that was actually selected.

Key result: If the selection event $A$ is determined entirely by interim data and the post-selection estimate is conditionally unbiased and independent of those data, then $$ E(\hat\theta_{J,F}-\theta_J\mid A) = w_1 E(\hat\theta_{J,1}-\theta_J\mid A). $$ Thus pooling attenuates the conditional bias of the interim estimate by the interim weight $w_1$. Unless that weight is zero or the interim conditional bias is zero, the pooled estimate can remain biased after selection.

Why the Conditional Bias Is Weighted Down

The result follows by subtracting the selected arm’s true effect from the pooled estimate and taking a conditional expectation. Since $J$ and $A$ are determined by interim information, the post-selection estimate has conditional mean $\theta_J$. Therefore,

\begin{aligned} E(\hat\theta_{J,F}-\theta_J\mid A) &= w_1 E(\hat\theta_{J,1}-\theta_J\mid A) + w_2 E(\hat\theta_{J,2}-\theta_J\mid A)\\ &= w_1 E(\hat\theta_{J,1}-\theta_J\mid A) + w_2(0). \end{aligned}

This is a conditional statement: it describes average bias among repetitions satisfying event $A$, not the error in one particular trial. It also does not say that the selected interim estimate is biased for every possible selection rule or event. The earlier material on conditioning on the winner or the full selection path distinguishes those events; whichever event is used, the same decomposition applies if the post-selection estimate is conditionally unbiased.

Worked Example: Pooling an Observed Interim Estimate

Suppose two candidate arms are compared with a control at an interim analysis. Arm A is selected. Its interim estimated effect is $0.62$ percentage points, with standard error $0.30$ percentage points. After selection, new participants provide an independent estimate for Arm A of $0.18$ percentage points, with standard error $0.20$ percentage points. Assume both estimates target the same effect and use inverse-variance weighting.

The variances are $V_1=0.30^2=0.09$ and $V_2=0.20^2=0.04$, in squared percentage points. The interim weight is $w_1=V_2/(V_1+V_2)=0.04/0.13=0.3077$, and the post-selection weight is $w_2=0.09/0.13=0.6923$.

\hat\theta_{A,F} =0.3077(0.62)+0.6923(0.18) =0.1908+0.1246 =0.3154\text{ percentage points}.

The pooled estimate is lower than the interim estimate because the independent post-selection estimate is lower and receives more weight. It is not justified to conclude from this one calculation that selection bias has been removed: the true effect is unknown, and a single observed change is not a bias estimate. Under the stated model, the pooled estimator retains $30.77\%$ of the interim estimate’s conditional bias.

How More Post-Selection Information Changes Expected Bias

A simple selection rule makes the attenuation transparent. Suppose there are two arms with the same true effect $\theta$, independent interim estimates with common standard error $\sigma_1$, and the arm with the larger interim estimate is selected. This is a deliberately simple model: it isolates the effect of choosing a winner and does not account for shared-control correlation.

Write the interim estimates as $\theta+\epsilon_1$ and $\theta+\epsilon_2$, where the errors are independent $N(0,\sigma_1^2)$. For the selected arm, the interim error is $\max(\epsilon_1,\epsilon_2)$. Using $\max(a,b)=(a+b+|a-b|)/2$, and noting that $\epsilon_1-\epsilon_2\sim N(0,2\sigma_1^2)$, we obtain

E\{\max(\epsilon_1,\epsilon_2)\} =\frac{1}{2}E|\epsilon_1-\epsilon_2| =\frac{1}{2}\left(\frac{2\sigma_1}{\sqrt{\pi}}\right) =\frac{\sigma_1}{\sqrt{\pi}}.

The selected interim estimate therefore has expected excess $\sigma_1/\sqrt{\pi}$ over the common true effect. If the independent post-selection estimate is unbiased and the pooled interim weight is $w_1$, the expected excess of the pooled estimate is $w_1\sigma_1/\sqrt{\pi}$. The added data reduce the expected bias in proportion to their influence on the final estimate; they do not make the first-stage selection disappear.

Worked Example: Expected Bias Under a Two-Arm Winner Rule

Continue with two equally effective arms, but now consider repeated hypothetical trials rather than the one observed pair of estimates above. Let both true effects be $0.20$ percentage points, let the interim standard error for each arm be $0.30$ percentage points, and select whichever arm has the larger interim estimate. Suppose independent post-selection data give the selected arm an estimate with standard error $0.20$ percentage points.

The expected interim winner’s excess is $0.30/\sqrt{\pi}=0.1693$ percentage points. The inverse-variance interim weight is $0.04/(0.09+0.04)=0.3077$. Hence the expected excess of the pooled final estimate, conditional on selecting the interim winner, is

0.3077(0.1693)=0.0521\text{ percentage points}.

In this model the expected selected interim estimate is $0.20+0.1693=0.3693$ percentage points, while the expected pooled estimate is $0.20+0.0521=0.2521$ percentage points. The figures describe expectations across repetitions under the specified equal-effects, independent-normal model. They are not a correction to apply automatically in a real MAMS design, where arm estimates may be correlated and selection rules may be more complex.

When the Attenuation Result Can Fail

The key condition is not merely that outcomes are collected later in calendar time. The post-selection estimate must carry no conditional bias induced by the selection information. If it shares participants, outcomes, or other noise with the interim estimate, independence may fail. If the estimand changes—for example, because the target population or outcome definition differs between stages—then combining estimates with a single common $\theta_J$ is not appropriate.

For a general linear combination with weights $a$ and $1-a$, the conditional bias is

E\{a\hat\theta_{J,1}+(1-a)\hat\theta_{J,2}-\theta_J\mid A\} = aE(\hat\theta_{J,1}-\theta_J\mid A) +(1-a)E(\hat\theta_{J,2}-\theta_J\mid A).

If the second term is nonzero, the post-selection estimate’s bias contributes as well; the simple attenuation formula no longer follows. Dependence may also alter the variance of the pooled estimator, so the independent-stage variance formula should not be used without justification.

Worked Example: A Correlated Later Estimate

Suppose two equally effective arms have interim errors $\epsilon_{1,1}$ and $\epsilon_{2,1}$, each $N(0,1)$, and the arm with the larger interim estimate is selected. For the selected arm, suppose the later estimate’s error has conditional expectation $0.5\epsilon_{J,1}$, as occurs in a jointly normal model with equal error variances and correlation $0.5$ between the interim and later errors for each arm. This is a hypothetical dependence structure, not an independent validation stage.

For two independent interim errors, the expected interim winner’s excess is $1/\sqrt{\pi}=0.5642$. The later estimate’s expected excess is therefore $0.5(0.5642)=0.2821$. If the two estimates receive equal weights, the pooled expected excess is

\frac{1}{2}(0.5642)+\frac{1}{2}(0.2821) =0.4232.

The pooled estimate still has less expected excess than the interim estimate in this example, but not the $0.5(0.5642)=0.2821$ predicted by the independent-stage formula. The discrepancy arises because the later estimate also carries selection-related excess. Before treating new data as an unbiased contribution, verify their relationship to the data used for selection.

Diagnostics, Alternatives, and Practical Limits

A useful analysis checks the design and data-generating process before relying on the attenuation argument:

Trace the selection information. Confirm exactly which interim outcomes, contrasts, and rules determined the selected arm.
Check data separation. Establish whether post-selection participants and outcomes are genuinely new, and whether any shared controls or reused observations induce dependence.
Check the estimand. Confirm that both stages estimate the same treatment contrast in a comparable population with compatible outcome definitions and follow-up.
Revisit precision weights. Check that the stage variances and any covariance assumptions match the actual design. If stages are dependent, the independent inverse-variance formula is not generally justified.

There are three useful reference points. An interim-only estimate uses the information most directly affected by selection and can retain the full conditional bias of that interim estimate. A pooled estimate usually has greater precision than an estimate based only on the added data, but it retains a weighted contribution from the interim estimate. A post-selection-only estimate can avoid selection bias under the independence and unbiasedness conditions described above, but it discards interim information and may be less precise. These are comparisons of properties, not a universal ranking: the appropriate estimator depends on the design, estimand, and inferential goal.

The result is limited to the estimator’s conditional mean under the stated model. It does not guarantee unbiased confidence intervals or tests after selection, and it does not say that the pooled estimate is unbiased conditional on every possible selection path. Unequal standard errors, correlated treatment-control contrasts, stopping rules, or selection based on several stages require their own conditioning and variance analysis. In particular, do not infer that a smaller final estimate proves the interim estimate was exaggerated in that trial; selection bias is a repeated-sampling property.

Common pitfall: “The final estimate includes new participants, so selection bias is gone.” This is wrong when the final estimate still gives positive weight to the interim estimate that selected the arm. Under independent, unbiased post-selection data, the correct conclusion is that the pooled estimate’s conditional bias is the interim conditional bias multiplied by its pooling weight.

Check Your Understanding

Answer each question using the independence and common-estimand assumptions where needed. Brief solutions follow.

An interim estimate has conditional bias $0.12$ units. Its weight in the pooled estimate is $0.25$. What is the pooled conditional bias if the added estimate is independent and unbiased?
Why does collecting twice as many post-selection outcomes not necessarily make the pooled estimate unbiased?
State two reasons why a later estimate might not be conditionally unbiased given the selected arm.
For interim and post-selection variances $0.16$ and $0.04$, respectively, calculate the inverse-variance weight on the interim estimate and the conditional bias multiplier.

Solutions. (1) The bias is $0.25(0.12)=0.030$ units. (2) The interim estimate can still receive positive weight; more post-selection information reduces that weight but does not generally set it to zero. (3) Examples include overlapping or shared data that create dependence with the selection statistic, and a change in population or outcome definition that changes the estimand. (4) The interim weight is $0.04/(0.16+0.04)=0.20$, so the pooled conditional bias is $20\%$ of the interim conditional bias, under the stated assumptions.