Small-Sample Exact Intervals After Selection

Why Selection Changes an Exact Interval

When several treatment arms are compared, the arm with the most favorable observed result may be selected for further study or reporting. Even if each arm’s response count follows a binomial model, the count for the selected arm does not have its ordinary, unselected distribution: selection has required that it meet a threshold set by the competing arms. A confidence interval that ignores this event can therefore have poor coverage conditional on the arm having been selected.

For small samples and discrete outcomes, a direct solution is to condition on the selection information and invert tests using the resulting conditional distribution. The method below gives a finite-sample coverage guarantee under its stated model. Because the distribution is discrete, coverage is generally at least, rather than exactly, the nominal level.

Definition: A selection-conditioned exact confidence interval for a parameter is obtained by inverting tests whose reference distribution is the finite-sample distribution conditional on the event that led to selection. “Exact” means the coverage guarantee follows from that specified distribution without a large-sample approximation; discreteness can make the interval conservative.

Model and Conditional Distribution

Suppose a trial has fixed sample sizes \(n_0,n_1,\ldots,n_K\), with one control arm and \(K\) candidate treatment arms. Let \(X_j\) be the number of responders among the \(n_j\) participants assigned to candidate arm \(j\), and let \(X_0\) be the control response count. Assume

within each arm, participant outcomes are independent Bernoulli outcomes;
the response probability is constant within arm, with probability \(p_j\) in candidate arm \(j\); and
the arm counts are independent under the sampling model, conditional on the fixed allocation.

For the worked examples, take the control response probability \(p_0\) as known. The treatment effect of interest is the selected arm’s risk difference, \(p_j-p_0\). Treating \(p_0\) as known isolates the selection issue; if it is estimated from the same trial, its uncertainty must also be incorporated for an interval on the risk difference.

Suppose selection chooses the candidate arm with the largest response count, with a prespecified rule for resolving ties. After arm \(j\) is selected, condition on the observed response counts in the competing arms. Let \(c\) be the smallest count arm \(j\) could have had and still be selected given those competitor counts and the tie rule. In the examples, ties are awarded to the arm under consideration, so selection requires \(X_j\ge c\).

Before conditioning, \(X_j\sim\mathrm{Binomial}(n_j,p_j)\). Given the competitor counts and the event \(X_j\ge c\), its conditional probability mass function is

P_{p_j}(X_j=k\mid X_j\ge c) = \frac{\binom{n_j}{k}p_j^k(1-p_j)^{n_j-k}} {\displaystyle\sum_{r=c}^{n_j}\binom{n_j}{r}p_j^r(1-p_j)^{n_j-r}}, \qquad k=c,\ldots,n_j.

The denominator is the probability of satisfying the selection threshold under \(p_j\). The other arms’ probabilities do not appear: after conditioning on their observed counts, their likelihood factors cancel. This does not mean selection is irrelevant; it is represented by the truncated support \(c,\ldots,n_j\).

Inverting Conditional Tests

Let \(x\) be the selected arm’s observed count and let \(0<\alpha<1\) be the desired error level. For each candidate value of \(p_j\), calculate the two inclusive conditional tail probabilities

U(p_j)=P_{p_j}(X_j\ge x\mid X_j\ge c), \qquad L(p_j)=P_{p_j}(X_j\le x\mid X_j\ge c).

A central equal-tailed interval retains parameter values for which neither conditional tail is at most \(\alpha/2\). Its lower endpoint is found by solving \(U(p_j)=\alpha/2\), and its upper endpoint by solving \(L(p_j)=\alpha/2\), when those roots lie in \([0,1]\). If an endpoint reaches a boundary, use 0 or 1 as appropriate. The risk-difference interval is obtained by subtracting \(p_0\) from both endpoints.

Justification of coverage: Under a fixed \(p_j\), the conditional distribution above is the true distribution of the count given the selection information. For a discrete distribution, an inclusive-tail test based on a tail probability of at most \(\alpha/2\) rejects with probability no greater than \(\alpha/2\): the outcomes in its rejection region have total probability at most that threshold. Applying this to both tails, the probability that either test excludes the true \(p_j\), conditional on the specified selection information, is at most \(\alpha/2+\alpha/2=\alpha\). Therefore the interval covers \(p_j\) with conditional probability at least \(1-\alpha\). Subtracting the known \(p_0\) preserves that coverage for \(p_j-p_0\).

The inequality is often strict because a discrete count cannot generally produce a tail probability equal to the chosen cutoff. If the conditioning event has probability zero at a parameter boundary, the conditional distribution is undefined there; use limits from parameter values for which selection has positive probability, and report boundary handling explicitly.

Worked Example: Two Candidate Arms and a Four-Patient Sample

Suppose two candidate treatments each have \(n=4\) participants, and the arm with the higher response count is selected; ties favor arm A. The hypothetical counts are \(X_A=3\) and \(X_B=2\). For arm A, the selection condition given \(X_B=2\) is \(X_A\ge2\). Let the known control response probability be \(p_0=0.20\). Find a 95% conditional exact interval for arm A’s risk difference.

Write \(p=p_A\). Conditional on selection, the probability of a count of at least 3 is

U(p)= \frac{P_p(X_A\ge3)}{P_p(X_A\ge2)} = \frac{4p^3(1-p)+p^4}{6p^2(1-p)^2+4p^3(1-p)+p^4} = \frac{p(4-3p)}{6-8p+3p^2}.

For the lower endpoint, set \(U(p)=0.025\). Cross-multiplication gives \(3.075p^2-4.2p+0.15=0\); the root in \([0,1]\) that gives the lower endpoint is \(p=0.03670\). For the upper endpoint, the only conditional outcome above 3 is 4, so

L(p)=1-\frac{P_p(X_A=4)}{P_p(X_A\ge2)} =1-\frac{p^2}{6-8p+3p^2}.

Setting \(L(p)=0.025\) gives \(1.925p^2-7.8p+5.85=0\). The root in \([0,1]\) is \(p=0.9937\). Thus the interval for \(p_A\) is approximately \([0.03670,0.9937]\). Subtracting the known control probability gives the risk-difference interval

[0.03670-0.20,\;0.9937-0.20]=[-0.1633,\;0.7937].

This very wide interval is still informative about the method: with only four participants and conditioning on having at least two responses, the data give limited precision about the response probability.

More Selection Thresholds

The same construction applies when there are more than two candidate arms. Once the competing counts are observed, the selection rule determines the threshold for the selected arm. For example, if the selected arm must have at least as many responses as every competitor and ties favor it, \(c\) is the largest competitor count. If ties are resolved in favor of another arm, the threshold may instead be one higher. The rule must be specified before calculating the conditional distribution.

Worked Example: Three Candidate Arms

Consider three hypothetical candidate arms, each with \(n=5\). Their response counts are \(X_A=4\), \(X_B=3\), and \(X_C=1\); arm A is selected, and ties favor the selected arm. Given the competing counts, arm A must have at least \(c=3\) responses. Let \(p_0=0.25\) be known. Calculate the 95% conditional exact interval for arm A’s risk difference.

For \(X_A=4\), the conditional upper tail is

U(p)=\frac{P_p(X_A\ge4)}{P_p(X_A\ge3)} =\frac{5p^4(1-p)+p^5}{10p^3(1-p)^2+5p^4(1-p)+p^5} =\frac{p(5-4p)}{10-15p+6p^2}.

Solving \(U(p)=0.025\) gives \(4.15p^2-5.375p+0.25=0\). The root in \([0,1]\) is \(p=0.04831\). The conditional lower tail through 4 is

L(p)=1-\frac{P_p(X_A=5)}{P_p(X_A\ge3)} =1-\frac{p^2}{10-15p+6p^2}.

Solving \(L(p)=0.025\) gives \(4.85p^2-14.625p+9.75=0\), with root \(p=0.9949\) in \([0,1]\). The response-probability interval is therefore approximately \([0.04831,0.9949]\), and the risk-difference interval is

[0.04831-0.25,\;0.9949-0.25]=[-0.2017,\;0.7449].

The calculation conditions on the observed competing counts, not just on the statement “arm A was selected.” That distinction determines the threshold and hence the reference distribution used for inference.

Worked Example: A Selection Boundary at One Response

Suppose a hypothetical selected arm has \(n=3\), observed count \(x=2\), and was selected because it had at least one response; thus \(c=1\). Let the known control response probability be \(p_0=0.30\). For a 95% interval, the conditional upper tail is

U(p)=\frac{P_p(X\ge2)}{P_p(X\ge1)} =\frac{p(3-2p)}{3-3p+p^2}.

Solving \(U(p)=0.025\) gives \(2.025p^2-3.075p+0.075=0\); the root in \([0,1]\) is \(p=0.02480\). Since the only outcome above 2 is 3, the lower conditional tail is

L(p)=1-\frac{p^3}{P_p(X\ge1)} =1-\frac{p^2}{3-3p+p^2}.

Setting this equal to 0.025 gives \(p^2+117p-117=0\), whose root in \([0,1]\) is \(p=0.9916\). The interval for the selected risk difference is \([0.02480-0.30,\;0.9916-0.30]=[-0.2752,\;0.6916]\). The selection threshold is essential here: the relevant distribution is conditional on at least one response, not the ordinary three-trial binomial distribution.

Diagnostics, Alternatives, and Limitations

Before reporting an interval, verify that the conditional calculation matches the trial and its selection rule. The method depends on the selection event being known and correctly recorded, the arm counts following the stated binomial model, and the allocation sizes being fixed as assumed. It is also important that the conditioning event has positive probability under the parameter values considered.

Check the threshold. Reconstruct \(c\) from the recorded competitor counts and the prespecified tie rule. A wrong threshold means the conditional reference distribution is wrong.
Check the outcome model. Dependence between participant outcomes, varying response probabilities within an arm, or unmodeled changes in allocation can invalidate the binomial distribution.
Inspect discreteness and width. With small samples, endpoints can be near 0 and 1 and coverage can exceed 95%. This is a consequence of discreteness, not necessarily a calculation error.
Check what was conditioned on. Conditional coverage for a selected arm at specified competitor counts is not the same claim as simultaneous coverage for every arm or coverage averaged over all possible selection outcomes.

An unconditional interval instead averages over the possible selection outcomes, while a conditional interval targets coverage given the specified selection information. A conservative multiplicity-adjusted interval can avoid modeling the detailed selection-conditioned distribution, but may be wider. Large-sample intervals use approximations that can be unreliable with very small counts. These approaches answer related but distinct questions; the conditioning event and the desired coverage guarantee should guide the choice.

A further limitation is the known-control assumption in the risk-difference examples. If \(p_0\) is estimated, simply subtracting the observed control proportion from both endpoints does not establish exact coverage for \(p_j-p_0\). A valid procedure must account for uncertainty in both probabilities and any dependence induced by the trial’s selection rule. Similarly, conditioning on the competitor counts gives an interval for the selected arm’s effect, not a simultaneous guarantee across all candidate arms.

Common pitfall: “Arm A had the largest count, so I can use an ordinary binomial interval for its response probability.” This ignores the fact that the selected count was required to exceed the competitors’ counts. The correction is to derive the count distribution under the actual selection condition and invert tests using that conditional distribution.

Check Your Understanding

Answer each question, then compare with the solutions below.

A selected arm has \(n=6\), observed count \(x=4\), and the largest competing count is 2. Ties favor the selected arm. What selection threshold \(c\) should be used?
Why does the conditional mass function contain a denominator summing from \(c\) to \(n_j\)?
If the selected arm’s response-probability interval is \([0.12,0.78]\) and the known control probability is \(0.20\), what is the risk-difference interval?
Does a 95% conditional interval necessarily have exactly 95% coverage? Explain.
What changes if ties are resolved in favor of a competing arm and the observed maximum competitor count is \(c\)?

Solutions. (1) Use \(c=2\), since the selected arm can tie the largest competitor and still be selected. (2) It normalizes the binomial probabilities over outcomes that satisfy selection, giving the distribution conditional on \(X_j\ge c\). (3) Subtract \(0.20\) from both endpoints: \([0.12-0.20,\;0.78-0.20]=[-0.08,\;0.58]\). (4) No. Discreteness generally makes the inverted procedure conservative, so coverage is at least 95% under the stated assumptions. (5) The selected arm must exceed the competitor count, so use threshold \(c+1\), provided that outcome is possible.