Binary-Outcome Inference After Adaptation

Why Binary Outcomes Need Selection-Aware Inference

A two-stage trial may begin with several treatment arms, use first-stage response rates to select one, and then allocate the second stage to that treatment and a control group. The selected arm’s first-stage response rate helped determine which estimate would be reported. Consequently, treating that arm as if it had been specified in advance can make an ordinary interval too optimistic: it ignores the selection step.

This tutorial develops a transparent, finite-sample solution based on simultaneous binomial intervals. It is deliberately conservative, but its coverage guarantee does not depend on the details of the selection rule beyond recording which arm was selected. More efficient selection-adjusted methods are possible, but they require more detailed calibration.

Setup and assumptions: There are \(K\) candidate treatments and one control. In stage 1, each arm has a prespecified sample size and independent binary outcomes. A recorded rule selects one treatment \(S\) using stage-1 information. In stage 2, only \(S\) and control are observed, with sample sizes fixed once stage-1 information is available. Within each arm, outcomes are independent Bernoulli observations. The arm-specific response probabilities are stable across stages, and the outcome, follow-up, and analysis population are defined consistently. Under random assignment, the risk contrasts describe treatment effects for the randomized population, subject to these assumptions.

Write \(p_k\) for the response probability under candidate treatment \(k\) and \(p_0\) for the control response probability. The reported target is the effect for the selected arm, so it is \(p_S-p_0\) on the risk-difference scale or the selected-arm odds ratio on the odds-ratio scale. Because \(S\) depends on stage-1 data, this is a selected target, not a treatment arm fixed independently of the data.

Definition: The risk difference for treatment \(k\) is \(\delta_k=p_k-p_0\). The odds ratio is \(\operatorname{OR}_k=\{p_k/(1-p_k)\}/\{p_0/(1-p_0)\}\), for probabilities strictly between 0 and 1. The corresponding selected-arm estimands are \(\delta_S\) and \(\operatorname{OR}_S\).

A Simultaneous-Interval Construction

For one arm with \(x\) responses among \(n\) participants, let \(\hat p=x/n\). Hoeffding’s inequality gives, for any \(r>0\), \(P(|\hat p-p|\ge r)\le 2\exp(-2nr^2)\). Thus, choosing \(r=\sqrt{\log(2/a)/(2n)}\) gives a binomial interval with coverage at least \(1-a\):

I(\hat p,n,a)= \left[ \max(0,\hat p-r),\ \min(1,\hat p+r) \right], \qquad r=\sqrt{\frac{\log(2/a)}{2n}}.

This interval is conservative, but its guarantee is finite-sample and does not require a normal approximation. To control the chance of any arm-level interval failing, allocate a total error allowance across the observed design. For example, with total \(\alpha=0.05\), allocate \(\alpha/2\) to stage 1 and \(\alpha/2\) to stage 2. Divide the stage-1 allowance across the \(K+1\) arms, and the stage-2 allowance across the selected treatment and control. Stage-2 intervals have their stated coverage conditional on stage-1 data because, given that data and the resulting allocation, the stage-2 sample sizes are fixed and stage-2 outcomes follow the assumed binomial model.

For any particular stage \(j\), if the treatment and control risk intervals are \([L_{Tj},U_{Tj}]\) and \([L_{Cj},U_{Cj}]\), subtraction gives a valid risk-difference interval:

I_{\delta,j}= [L_{Tj}-U_{Cj},\ U_{Tj}-L_{Cj}].

If both arm risks lie in their respective intervals, their difference must lie in this interval. To combine stage estimates, choose prespecified weights \(w_1,w_2\) with \(w_1,w_2\ge 0\) and \(w_1+w_2=1\). The estimate is \(w_1\hat\delta_1+w_2\hat\delta_2\), and its interval is formed by applying the same weights to the lower and upper stage limits.

Coverage justification: If the stage-1 intervals for all arms cover their risks, then they cover the risk of whichever treatment is selected. The probability that any stage-1 interval fails is at most \(\alpha/2\), by the union bound. Conditional on stage 1, the probability that either stage-2 interval fails is at most \(\alpha/2\); averaging over stage-1 outcomes preserves that bound. Therefore the probability that any interval needed for the selected-arm comparison fails is at most \(\alpha\). When both stage-specific risk-difference intervals contain the same stable effect \(\delta_S\), any convex combination of their lower limits is no greater than \(\delta_S\), and any convex combination of their upper limits is no less than \(\delta_S\). The combined interval therefore has coverage at least \(1-\alpha\).

Worked Example: Following the Selection Path

Suppose two candidate treatments, A and B, and a control each have 30 stage-1 participants. The numbers of responses are 12 for A, 15 for B, and 6 for control. The rule selects the treatment with the larger observed response rate. Stage 2 then enrolls 20 participants on B and 20 on control, with 8 and 4 responses respectively. Calculate the stage-specific risk differences and an equally weighted descriptive estimate.

Stage 1 selects B because \(15/30=0.500\) exceeds \(12/30=0.400\). The control response rate is \(6/30=0.200\), so the stage-1 difference for the selected treatment is \(0.500-0.200=0.300\).

In stage 2, the treatment response rate is \(8/20=0.400\) and the control response rate is \(4/20=0.200\). The stage-2 risk difference is \(0.400-0.200=0.200\). With equal stage weights, the descriptive estimate is \((0.300+0.200)/2=0.250\), or 25 percentage points.

This calculation makes the selected target and the two data contributions explicit. By itself, the estimate is not a selection-adjusted interval; using it in an ordinary unadjusted Wald interval would not account for choosing B based on stage-1 results.

Worked Risk-Difference Interval

Worked Example: A Conservative Interval After Selecting the Highest Response Rate

Consider a hypothetical trial with two candidate treatments and a control. In stage 1, each arm has \(n=40\): control has 8 responses, A has 14, and B has 18. The rule selects B, whose observed rate is higher than A’s. In stage 2, B and control each have \(n=30\); B has 15 responses and control has 6. Construct a 95% interval using equal stage weights and the simultaneous Hoeffding construction.

Allocate \(\alpha/2=0.025\) to stage 1, divided across its three arms, so \(a_1=0.025/3=0.008333\). The common stage-1 radius is \(r_1=\sqrt{\log(2/0.008333)/(2\cdot40)} =\sqrt{\log(240)/80}=0.2617\). The stage-1 rates are \(18/40=0.450\) for B and \(8/40=0.200\) for control. Their intervals are \([0.450-0.2617,0.450+0.2617]=[0.1883,0.7117]\) and \([\max(0,0.200-0.2617),0.200+0.2617]=[0,0.4617]\). Thus the stage-1 risk-difference interval is \([0.1883-0.4617,\ 0.7117-0]=[-0.2734,0.7117]\).

Allocate the other \(0.025\) across the two stage-2 arms, so \(a_2=0.0125\). The radius is \(r_2=\sqrt{\log(2/0.0125)/(2\cdot30)} =\sqrt{\log(160)/60}=0.2908\). The stage-2 rates are \(15/30=0.500\) for B and \(6/30=0.200\) for control. Their risk intervals are \([0.2092,0.7908]\) and \([0,0.4908]\), giving the difference interval \([0.2092-0.4908,\ 0.7908-0]=[-0.2816,0.7908]\).

The stage-specific point estimates are \(0.450-0.200=0.250\) and \(0.500-0.200=0.300\). Their equal-weight estimate is \(0.275\). Weighting the interval limits equally gives \([(-0.2734-0.2816)/2,\ (0.7117+0.7908)/2]=[-0.2775,0.7513]\). Under the stated assumptions, this interval has at least 95% coverage for B’s risk difference. Its width reflects the deliberately conservative construction, not a claim that the effect is known precisely.

Odds Ratios and Their Intervals

For odds ratios, first form each stage’s risk bounds. Since odds \(p/(1-p)\) increase with \(p\), the lower odds-ratio limit uses the treatment’s lower risk and the control’s upper risk; the upper limit uses the treatment’s upper risk and the control’s lower risk. It is often easier to combine stage estimates on the log scale: calculate each stage’s log odds ratio, combine them with fixed convex weights, and combine the log interval limits in the same way. Exponentiating the resulting limits gives an odds-ratio interval.

Worked Example: Combining Stage-Specific Odds-Ratio Intervals

In a hypothetical two-stage trial, two treatments and control each have 500 stage-1 participants. B, selected by the prespecified rule, has 160 responses; control has 100. In stage 2, B and control each have 400 participants, with 120 and 80 responses. Use the same 95% allocation as above: stage-1 allowance \(0.025\), divided among three arms, and stage-2 allowance \(0.025\), divided between two arms.

The stage-1 radius is \(r_1=\sqrt{\log(240)/(2\cdot500)}=0.07403\). The B and control rates, \(0.320\) and \(0.200\), have intervals \([0.2460,0.3940]\) and \([0.1260,0.2740]\). The stage-1 odds ratio estimate is \((0.320/0.680)/(0.200/0.800)=1.882\). Its interval is \([ (0.2460/0.7540)/(0.2740/0.7260),\ (0.3940/0.6060)/(0.1260/0.8740) ]=[0.864,4.51]\), approximately.

The stage-2 radius is \(r_2=\sqrt{\log(160)/(2\cdot400)}=0.07965\). The rates \(0.300\) for B and \(0.200\) for control have intervals \([0.2204,0.3796]\) and \([0.1204,0.2796]\). The stage-2 odds ratio estimate is \((0.300/0.700)/(0.200/0.800)=1.714\), and its interval is \([ (0.2204/0.7796)/(0.2796/0.7204),\ (0.3796/0.6204)/(0.1204/0.8796) ]=[0.728,4.47]\), approximately.

With equal weights, the combined log-odds-ratio estimate is \(\{\log(1.882)+\log(1.714)\}/2\), so the combined odds ratio is \(\exp(\{\log(1.882)+\log(1.714)\}/2)=1.796\). Combining the log limits and exponentiating gives \([\sqrt{0.864\cdot0.728},\sqrt{4.51\cdot4.47}]\), approximately \([0.793,4.49]\). The interval includes 1, so these data do not rule out equal odds at the 95% coverage level. An odds ratio of 1.796 compares response odds; it is not a 79.6 percentage-point increase in response probability.

Diagnostics, Alternatives, and Limitations

The guarantee is only as credible as the design information and model assumptions used to construct it. Check the recorded selection rule, stage-specific sample sizes, outcome definition, and analysis population. Confirm that the second-stage allocation was fixed by the adaptation and that outcomes were not selectively omitted. If response probabilities plausibly change across calendar time or stage, a single common risk difference or odds ratio may not describe both stages; report stage-specific effects or define a different target rather than combining incompatible estimates.

The simultaneous construction is robust to the particular way stage-1 data select an arm, but can be wide because it protects all candidate arms and uses a conservative inequality. A selection-conditional approach instead conditions on the event that a particular arm was selected and calibrates inference using that conditional distribution. A test-inversion approach can likewise obtain an interval by inverting tests that account for selection. These methods may be more efficient, but require the selection rule and a valid treatment of nuisance parameters to be specified and calibrated. An ordinary Wald interval is simpler and often narrower, but its usual approximation does not automatically account for selection or guarantee coverage in small samples.

For odds ratios, risk intervals that reach 0 or 1 can produce an odds-ratio limit of 0 or infinity. That is a legitimate indication of limited information, not a reason to insert an arbitrary continuity correction and present the result as unchanged. Also distinguish interval coverage from point-estimate bias: simultaneous intervals protect coverage for the selected target, but do not make a selected-arm point estimate unbiased.

Common mistake: “B had the largest stage-1 response rate, so I can calculate a standard 95% interval for B as if it had been prespecified.” The selection makes B’s observed stage-1 result unusually favorable on average. A valid interval must account for that selection, for example by simultaneous coverage across candidate arms or by a correctly calibrated selection-conditional method.

Check Your Understanding

Use the assumptions and procedures in this tutorial to answer each question.

With four candidate treatments and one control, how should the stage-1 error allowance be split if half of a total 0.05 allowance is assigned to stage 1?
A stage-specific treatment risk interval is \([0.30,0.50]\) and its control interval is \([0.10,0.25]\). What risk-difference interval follows?
Why does a stage-2 interval retain its conditional coverage when its sample size is set using stage-1 data?
Suppose an odds-ratio interval has a control risk upper limit of 1. What should be reported about the corresponding lower odds-ratio limit?

Solutions. (1) There are five stage-1 arms, so divide \(0.025\) by five: each arm receives \(0.005\). (2) Subtract opposite endpoints: \([0.30-0.25,\ 0.50-0.10]=[0.05,0.40]\). (3) Conditional on the stage-1 data, the selected arm and stage-2 sample sizes are fixed; under the model, the new stage-2 outcomes have the stated binomial distribution. (4) The control upper risk limit of 1 has infinite odds, so the lower odds-ratio limit is 0; report the boundary rather than applying an unreported correction.