Worked Example: Correcting a Selected Arm in a Two-Arm Trial

Why the Selected Estimate Needs Its Own Analysis

Suppose a two-arm trial compares one active treatment with control at an interim analysis. Let \(X\) be the estimated treatment effect on a scale where larger values are more favorable, and let \(\theta\) be the true treatment effect on that scale. If the active treatment is carried forward only when its interim estimate exceeds a threshold, the estimate being reported comes from a selected set of possible results. Even when the estimate is unbiased before selection, its average among selected results will generally be too favorable.

The calculations below concern one active-versus-control contrast and one interim estimate. They do not combine interim and later-stage data. The selection rule is assumed to be known and to depend only on this estimate.

Model and assumptions: Before selection, \(X\sim N(\theta,s^2)\), where \(s\) is a known standard error and \(\theta\) is the true treatment effect. The active arm is selected if and only if \(X>c\), for a prespecified threshold \(c\). The estimate is unbiased before selection, the normal model and standard error are appropriate, and no other outcome or decision rule affects selection. Write \(S=\{X>c\}\) for the selection event.

The threshold could be zero if the arm is retained whenever the estimated effect is favorable, or a positive value if stronger evidence is required. The correction and interval depend on the actual rule: changing \(c\) changes the distribution of estimates conditional on selection.

How Selection Creates Overestimation

Let \(\phi\) and \(\Phi\) denote the standard normal density and cumulative distribution functions. Standardizing \(X\) gives \(Z=(X-\theta)/s\sim N(0,1)\). Selection \(X>c\) is equivalent to \(Z>a\), where \(a=(c-\theta)/s\). The conditional mean of a standard normal variable above \(a\) is \(\phi(a)/[1-\Phi(a)]\), since \(\int_a^\infty z\phi(z)\,dz=\phi(a)\). Therefore,

E(X\mid S)=\theta+s\frac{\phi(a)}{1-\Phi(a)},\qquad E(X-\theta\mid S)=s\frac{\phi(a)}{1-\Phi(a)},\qquad a=\frac{c-\theta}{s}.

The second expression is the winner’s-curse bias: the expected amount by which the estimate exceeds the true effect among trials in which the arm is selected. It is positive for a finite threshold. This is a property of repeated estimates under the model, not a known amount that can simply be subtracted from every observed estimate: the expression depends on the unknown \(\theta\).

Worked Example: Expected Bias Under a Zero Threshold

Suppose the true effect is \(\theta=0.5\) treatment-effect units, the interim standard error is \(s=2\) units, and the arm is selected when \(X>0\). Calculate the selection probability and the expected overestimation among selected trials.

Here \(a=(0-0.5)/2=-0.25\). Thus \(P(S)=1-\Phi(-0.25)=\Phi(0.25)\approx0.5987\). Using \(\phi(-0.25)=\phi(0.25)\approx0.3867\), the expected bias is \(2(0.3867)/0.5987\approx1.292\) treatment-effect units.

Consequently, the model’s mean interim estimate among selected trials is approximately \(0.5+1.292=1.792\) units, although the true effect is \(0.5\) units. This is an average over repeated trials meeting the selection rule; it does not say that every selected estimate is biased upward by exactly 1.292 units.

A Selection-Conditional Estimate

One model-based correction is to maximize the likelihood conditional on selection. For an observed estimate \(x>c\), the conditional density is the ordinary normal density divided by the probability of selection under the candidate effect \(\theta\). The resulting log likelihood, apart from constants not depending on \(\theta\), is

\ell_c(\theta;x) =-\frac{(x-\theta)^2}{2s^2} -\log\left\{1-\Phi\left(\frac{c-\theta}{s}\right)\right\}.

The first term rewards values of \(\theta\) near the observed estimate. The second accounts for the fact that the observation was only seen because it passed the threshold. Differentiating and setting the score equal to zero gives the conditional-likelihood equation:

x-\theta =s\frac{\phi((c-\theta)/s)} {1-\Phi((c-\theta)/s)}.

The right side is the model’s expected selection bias at the candidate \(\theta\). The corrected estimate is found by solving this equation, generally numerically. Unlike subtracting a bias evaluated at an arbitrary assumed true effect, this approach estimates the effect and its associated selection bias together under the conditional likelihood.

Worked Example: Correcting an Observed Estimate

Suppose the interim estimate is \(x=1.5\) units, its known standard error is \(s=1\) unit, and the arm was selected whenever \(X>0\). The naive estimate is 1.5 units. Find the conditional-likelihood estimate.

With \(c=0\), the equation becomes \(\theta+\phi(\theta)/\Phi(\theta)=1.5\). At \(\theta=1.31\), \(\phi(1.31)\approx0.1690\) and \(\Phi(1.31)\approx0.9049\), giving \(1.31+0.1690/0.9049\approx1.497\). At \(\theta=1.314\), the corresponding values are approximately \(0.1683\) and \(0.9056\), giving \(1.314+0.1683/0.9056\approx1.500\).

The conditional-likelihood estimate is therefore about \(1.314\) treatment-effect units. It is below the observed estimate because some of the observed favorability is attributed to selection. The correction is modest here; it is not a claim that the true effect is known to be 1.314.

For comparison, a simple plug-in calculation evaluates the expected bias at the naive estimate: \(b(1.5)=\phi(-1.5)/\Phi(1.5)\approx0.1295/0.9332=0.1388\) units, producing \(1.5-0.1388=1.361\) units. This is a convenient approximation, but it is not the same as maximizing the selection-conditional likelihood. The analysis should state which correction is used.

Selection-Adjusted Interval by Conditional Inversion

An ordinary normal interval uses the distribution of \(X\) before selection. For an arm known to have passed \(X>c\), the relevant distribution is instead the truncated normal distribution conditional on \(S\). For any observed \(x>c\), its conditional cumulative distribution function under candidate effect \(\theta\) is

F_\theta(x\mid S) = \frac{\Phi((x-\theta)/s)-\Phi((c-\theta)/s)} {1-\Phi((c-\theta)/s)}.

A two-sided equal-tailed 95% selection-adjusted interval is obtained by finding the two values of \(\theta\) for which this conditional CDF is 0.975 and 0.025. The CDF decreases as \(\theta\) increases: a larger effect shifts the selected estimate distribution to the right, making a fixed observed \(x\) less central. Accordingly, the lower endpoint solves \(F_\theta(x\mid S)=0.975\), and the upper endpoint solves \(F_\theta(x\mid S)=0.025\). This inversion yields an interval with 95% conditional coverage under the stated model and selection rule.

Worked Example: Constructing the Conditional 95% Interval

Use the observed estimate \(x=1.5\) units, standard error \(s=1\) unit, and selection threshold \(c=0\). For these values, \(F_\theta(1.5\mid S)=[\Phi(1.5-\theta)-\Phi(-\theta)]/\Phi(\theta)\).

To find the lower endpoint, solve \(F_\theta(1.5\mid S)=0.975\). At \(\theta=-1.34\), the numerator is approximately \(\Phi(2.84)-\Phi(1.34)=0.9977-0.9099=0.0878\), and the denominator is \(\Phi(-1.34)=0.0901\). The ratio is approximately \(0.975\), so the lower endpoint is about \(-1.34\) units.

For the upper endpoint, solve \(F_\theta(1.5\mid S)=0.025\). At \(\theta=3.455\), the numerator is approximately \(\Phi(-1.955)-\Phi(-3.455)=0.0253-0.0003=0.0250\), and the denominator is \(\Phi(3.455)\approx0.9997\). The ratio is approximately \(0.025\). Thus the selection-adjusted 95% interval is approximately \([-1.34,\,3.46]\) treatment-effect units.

The interval includes zero and is much wider than the naive interval \(1.5\pm1.96(1)=[-0.46,\,3.46]\). The two intervals need not have matching endpoints or widths: the conditional interval reflects that only estimates above zero could have been observed for this analysis. Under repeated applications of this same model and rule, the conditional procedure covers the true effect in 95% of selected cases; it does not assign a 95% probability to this particular interval containing a fixed effect.

A More Stringent Rule and Practical Comparisons

Worked Example: The Effect of Raising the Threshold

Suppose \(X\sim N(0,1)\), so the true effect is zero and the standard error is one unit, but selection requires \(X>1\). The selection probability is \(1-\Phi(1)\approx0.1587\). Since \(\phi(1)\approx0.2420\), the expected estimate among selected trials is \(0+1(0.2420/0.1587)\approx1.525\) units.

The selected estimates average about 1.525 units even though the true effect is zero. Relative to a zero-threshold rule, the stricter rule selects less often but produces a larger average estimate among those selected. This illustrates why a correction based on the wrong threshold can be materially misleading.

The conditional-likelihood estimate and interval answer different questions from common alternatives. A naive estimate and ordinary interval ignore the selection event. Plug-in bias subtraction is simple but depends on how the unknown effect is inserted into the bias formula. A bootstrap can represent a more complicated selection process, but it must reproduce the actual rule and the relevant model; resampling without selection can miss the bias. For this single normal contrast, the conditional likelihood and conditional-CDF inversion make the selection adjustment explicit.

The method also has important limits. Its calibration depends on a suitable normal approximation, a credible standard error, and an accurately specified selection threshold. If the rule also depends on another endpoint, an interim safety decision, or a competing treatment arm, conditioning only on \(X>c\) does not represent the full selection event. The resulting correction and interval then need a model that captures those additional decisions.

Diagnostics and Common Pitfalls

Before reporting a corrected estimate, verify that \(x>c\) and that the specified rule really was the rule used. Check that the estimate is approximately normal at the interim stage and that \(s\) reflects its sampling variability. If \(s\) is estimated imprecisely, treating it as known can understate uncertainty. Sensitivity analyses using plausible alternative thresholds, standard errors, or outcome-model assumptions can show how dependent the answer is on these inputs.

Subtracting the bias as if it were fixed. The conditional mean bias depends on the unknown \(\theta\). A plug-in subtraction is an approximation, not an exact universal correction.
Using the unselected interval after selection. The usual normal interval does not have the stated coverage conditional on passing the threshold.
Using the wrong selection event. A threshold of zero and a threshold of one standard error generate different conditional distributions. The correction must match the actual rule and scale.
Overstating what the result establishes. A corrected estimate remains uncertain and model-dependent. It is not proof of treatment benefit or a probability statement about the fixed effect.

Key takeaway: Under a normal model with selection \(X>c\), correct the selected estimate using a method that accounts for the truncated sampling distribution, and construct the interval by conditioning on that same event. Report the rule, model assumptions, estimate, and interval together.

Check Your Understanding

Answer each question, then compare with the short solutions.

For \(X\sim N(\theta,s^2)\) and selection when \(X>c\), write the expected overestimation among selected estimates.
If selection is \(X>0\), \(s=1\), and the observed estimate is \(x=2\), should the conditional-likelihood estimate generally be above or below 2? Explain.
For the selection rule \(X>c\), which conditional CDF equation gives the lower endpoint of an equal-tailed 95% interval?
Why is an ordinary interval not automatically a 95% interval conditional on selection?

Solutions. (1) It is \(s\phi((c-\theta)/s)/[1-\Phi((c-\theta)/s)]\). (2) Below 2, because the conditional likelihood accounts for the positive selection bias; the exact value requires solving the score equation. (3) Solve \(F_\theta(x\mid X>c)=0.975\), with \(F_\theta\) given by the truncated-normal CDF. (4) The ordinary interval is calibrated under the unselected sampling distribution, whereas selected estimates follow a distribution conditional on exceeding \(c\).