Post-Selection Estimation for Binary Outcomes

Why Selection Changes Binary-Outcome Estimates

In a multi-arm, multi-stage trial, an arm may continue, stop, or be selected for further study because its interim results look promising. If the same noisy results also contribute to the reported treatment estimate, the selected estimate tends to look more favorable than the underlying effect. This is selection bias: it arises from conditioning attention or analysis on a data-dependent event, even when treatment assignment was randomized.

For a binary endpoint, the effect can be summarized on several scales. We will use a direction in which positive values favor treatment: the control event risk minus the treatment event risk for the risk difference, and the control log odds minus the treatment log odds for the log odds benefit. The estimand and its scale must be stated before interpreting any correction.

A simple starting model assumes independent binomial outcomes in randomized arms, a fixed analysis time, and complete outcome ascertainment. For sufficiently large event and non-event counts, estimated risk differences and log odds ratios are approximately normal. The log-odds approximation is unreliable with sparse cells; in that setting, use a binomial likelihood rather than treating a normal approximation as dependable. If selection uses interim data and estimation uses additional stages, the dependence between stages must also be represented.

Conditional Likelihood for a Selected Estimate

Let \(Y\) denote an estimate of a scalar benefit \(\psi\), with approximate sampling model \(Y\sim N(\psi,s^2)\), where \(s\) is its standard error. Suppose the arm is reported only if \(Y>c\), for a prespecified threshold \(c\). Given that selection event, the density of the observed estimate is the original normal density divided by the probability of passing the threshold:

The conditional-likelihood estimate maximizes this density as a function of \(\psi\). Equivalently, for this normal threshold model it solves \(y=\psi+s\lambda((c-\psi)/s)\), where \(\lambda(a)=\phi(a)/\{1-\Phi(a)\}\), and \(\phi\) and \(\Phi\) are the standard normal density and distribution functions. The extra term is the expected upward shift caused by retaining only estimates above the threshold.

The score equation follows directly by differentiating the conditional log likelihood. Write \(a=(c-\psi)/s\). Apart from constants, the log likelihood is \(-(y-\psi)^2/(2s^2)-\log\{1-\Phi(a)\}\). Since \(da/d\psi=-1/s\), its derivative is \((y-\psi)/s^2-\lambda(a)/s\). Setting this derivative to zero gives \(y=\psi+s\lambda(a)\). Thus, for a positive threshold, the observed estimate is expected to exceed the parameter under the conditional model. The correction is obtained by solving for \(\psi\), not by subtracting a fixed amount independent of \(\psi\).

Log Odds Benefits and Their Approximation

For independent binomial arms with nonzero event and non-event counts, a common large-sample estimate of the log odds benefit is the difference between the control and treatment sample log odds. Its approximate variance is the sum of the reciprocals of the four cell counts. This is a sampling approximation, not a selection correction; selection must still be included in the likelihood or another correction procedure.

In a real MAMS analysis, selection may depend on several interim statistics, may require an arm to beat a comparator, or may include dropping arms at multiple looks. Then the denominator in the conditional likelihood is the probability of the observed selection path under the model, not merely \(P_\psi(Y>c)\). For binary outcomes, that probability can be computed from a binomial model or an appropriate joint approximation. The model must preserve the dependence between interim and final data.

Selection Among Competing Arms

Selection of the largest estimate is different from selection by a fixed threshold. Even if every candidate has the same true benefit, the winning estimate tends to be above that common value because it is the maximum of noisy estimates. A compact derivation shows the size of this effect in a simple two-arm model.

Suppose \(Y_1\) and \(Y_2\) are independent estimates with \(Y_j\sim N(\theta_j,s^2)\), and arm 1 is selected when \(Y_1>Y_2\). Let \(D=Y_1-Y_2\). Then \(D\sim N(\theta_1-\theta_2,2s^2)\), and \(Y_1=(Y_1+Y_2+D)/2\). The sum \(Y_1+Y_2\) is independent of \(D\), so conditioning on \(D>0\) gives

In particular, if \(\theta_1=\theta_2=\theta\), then \(\lambda(0)=\sqrt{2/\pi}\), and the expected winning estimate is \(\theta+s/\sqrt{\pi}\). This is a bias statement under a specified equal-effects model; it is not a universal amount to subtract from a selected estimate.

Choosing and Checking a Correction

The main alternatives differ in how they represent selection. A naive estimate uses the observed data without conditioning on the selection event. Conditional likelihood explicitly divides by the model probability of that event. A selection-aware parametric bootstrap instead repeatedly generates data under a fitted outcome model, reapplies the trial’s selection rule, and evaluates the resulting selected estimates. The bootstrap must reproduce the actual path, including interim looks and data reuse; resampling only the final selected arm does not do so.

Useful diagnostics include checking observed and expected event counts, verifying that the selection event in the analysis matches the actual rule, and examining whether the normal approximation is plausible on the chosen scale. For log odds estimates, sparse or zero cells signal that the reciprocal-count variance and normal approximation may be poor. Compare the corrected estimate with the naive estimate and inspect whether the change is directionally consistent with the selection mechanism. A correction that changes greatly under modest, scientifically plausible model variations deserves cautious interpretation.

A frequent error is to say, “the arm passed the efficacy threshold, so its estimate is unbiased conditional on success.” The reasoning reverses the conditioning: among estimates that pass, positive sampling errors are overrepresented. The correction addresses that conditional distribution; it does not repair every source of bias, such as outcome-model misspecification, missing outcomes, or a selection rule that was recorded incorrectly. Nor does correction convert a selected-arm estimate into proof of treatment benefit.