Conditional Confidence Intervals After Selection

Why Condition on Selection?

When a trial selects an arm using its observed results, the estimate for the selected arm is not distributed as it would be before selection. For example, if the arm with the larger observed treatment contrast is carried forward, its contrast is necessarily positive. An interval that ignores this selection may therefore have a different coverage rate among the trials in which that arm is selected than its nominal rate suggests.

A conditional confidence interval makes a specific promise: for a fixed true parameter, its repeated-sampling coverage is controlled among repetitions satisfying a prespecified selection event. This is not the same target as coverage averaged across all possible adaptation outcomes. The event must be stated precisely, and the probability of that event can depend strongly on the true treatment effect.

Definition: Let $\theta$ be a parameter, $C(X)$ an interval calculated from data $X$, and $A$ a prespecified event that determines selection. The interval has $1-\alpha$ conditional coverage on $A$ if $$ P_\theta\{\theta\in C(X)\mid A\}=1-\alpha $$ for every $\theta$ in the model, or at least is no smaller than $1-\alpha$ if the procedure is conservative. The conditioning event is part of the coverage statement, not an informal description of the analysis.

Two-Arm Normal Model and Selection Rule

Consider two fixed treatment arms with independent estimated mean outcomes $\bar{x}_1$ and $\bar{x}_2$. Assume the observations in each arm are normally distributed, the arms are independent, sample sizes are fixed, and the outcome variance $\sigma^2$ is known and common. The target is the signed treatment contrast $\delta=\mu_1-\mu_2$, measured in outcome units. Then the observed contrast $D=\bar{x}_1-\bar{x}_2$ satisfies

D\sim N(\delta,\tau^2), \qquad \tau=\sigma\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}.

Suppose the prespecified rule selects arm 1 when $D>c$, where $c$ is a fixed threshold. The common best-observed-arm rule is the special case $c=0$. The interval below estimates the signed contrast $\delta$, with arm labels fixed before observing outcomes; if the rule selects arm 2 instead, the contrast can be reoriented and the corresponding event stated.

For any possible observed value $d>c$, the conditional distribution function is obtained by dividing the probability of observing a contrast no greater than $d$ by the probability of satisfying the selection rule:

F_\delta(d\mid D>c) = \frac{P_\delta(c<D\le d)}{P_\delta(D>c)} = \frac{ \Phi\!\left(\frac{d-\delta}{\tau}\right) -\Phi\!\left(\frac{c-\delta}{\tau}\right) }{ 1-\Phi\!\left(\frac{c-\delta}{\tau}\right) }.

Here $\Phi$ is the standard normal cumulative distribution function. The formula uses the normal model for $D$ and conditions on the same event that triggers selection. It is not the ordinary normal cumulative probability evaluated as if selection had not occurred.

Worked Example: A Conditional Percentile After Positive Selection

Suppose $\tau=1$ outcome point, the selection rule is $D>0$, and the observed contrast is $d=1$ point. To see the effect of conditioning, evaluate the conditional percentile when the true contrast is $\delta=0$.

Under this null value, $D\sim N(0,1)$, and the selection probability is $P(D>0)=0.5$. Therefore,

F_0(1\mid D>0) = \frac{\Phi(1)-\Phi(0)}{1-\Phi(0)} = \frac{0.841344746\ldots-0.500000000\ldots}{0.500000000\ldots} = 0.68268949\ldots \approx 0.6827.

Thus, conditional on a positive contrast, an observed value of 1 standard error is at about the 68.27th percentile when $\delta=0$. The exact normal probabilities determine the reported value; rounding $\Phi(1)$ early would give a less accurate result. Without conditioning, the percentile would instead be $\Phi(1)$, about 84.13%.

Inverting the Conditional Distribution

For a continuous distribution, the probability integral transform says that its cumulative distribution function evaluated at a draw from that distribution is uniform on $(0,1)$. Applying this result under the conditional distribution gives, for each fixed $\delta$,

F_\delta(D\mid D>c)\mid(D>c)\sim \operatorname{Uniform}(0,1).

Consequently, the event that this conditional percentile lies between $\alpha/2$ and $1-\alpha/2$ has conditional probability $1-\alpha$. Inverting that event over possible values of $\delta$ gives an equal-tailed interval. For an observed $d>c$, its endpoints solve the two equations below.

F_{\delta_L}(d\mid D>c)=1-\frac{\alpha}{2}, \qquad F_{\delta_U}(d\mid D>c)=\frac{\alpha}{2}.

The order of the two tail probabilities may at first seem reversed. For fixed $d$, increasing $\delta$ shifts the conditional distribution of $D$ toward larger values, so its cumulative probability at $d$ decreases. The larger conditional percentile therefore gives the lower parameter endpoint. This monotonicity also follows from the conditional density: the ratio of the density at $d$ under $\delta_2$ to that under $\delta_1$, for $\delta_2>\delta_1$, is a positive constant times $\exp\{(\delta_2-\delta_1)d/\tau^2\}$, which increases with $d$. The conditional family has a monotone likelihood ratio and is stochastically increasing in $\delta$.

The coverage justification is direct. Under the model, conditional on selection, the conditional percentile is uniform. Thus the probability that it falls in the central interval $[\alpha/2,1-\alpha/2]$ is $1-\alpha$. The set of parameter values satisfying that percentile inequality is exactly the interval bounded by the two roots above. This establishes

P_\delta\{\delta_L(D)\le \delta\le \delta_U(D)\mid D>c\}=1-\alpha.

In general the roots do not have a simple closed form. They can be found by numerical root-solving, evaluating the conditional distribution function for candidate parameter values. The probability of selection must be positive at the parameter values under consideration; under the normal model it is positive for every finite $\delta$, although it can be very small.

Worked Example: A 95% Interval After Selecting the Larger Contrast

Assume independent normal outcomes in two arms, common known standard deviation 5 points, and 50 observations per arm. The standard error of the difference is $5\sqrt{1/50+1/50}=1$ point. Suppose the observed arm means are 12 and 11 points, so $d=1$, and the prespecified rule selects arm 1 whenever $D>0$. All assumptions used in the normal contrast model hold by construction in this hypothetical example.

For a 95% equal-tailed interval, set $\alpha=0.05$, $\tau=1$, $c=0$, and $d=1$ in the conditional CDF. Solve

\frac{\Phi(1-\delta)-\Phi(-\delta)}{\Phi(\delta)} =0.975 \quad\text{for }\delta_L, \qquad \frac{\Phi(1-\delta)-\Phi(-\delta)}{\Phi(\delta)} =0.025 \quad\text{for }\delta_U.

Numerical root-solving gives approximately $\delta_L=-2.93$ points and $\delta_U=2.93$ points. Hence the conditional 95% interval is about $[-2.93,\,2.93]$ points, with width $2.93-(-2.93)=5.86$ points. Small differences in the last decimal can arise from numerical root tolerance.

For comparison, the ordinary unadjusted Wald interval would be $d\pm1.96\tau=1\pm1.96$, or approximately $[-0.96,\,2.96]$ points, with width 3.92 points. That interval is not obtained by conditioning on $D>0$; its usual coverage statement is unconditional under the fixed normal model. The conditional interval is wider here because it uses the selected-data distribution and promises 95% coverage specifically among repetitions in which the contrast is positive. It can include negative values of $\delta$: selecting a positive estimate does not prove that the true contrast is positive.

Worked Example: A More Stringent Prespecified Threshold

Suppose the same standard error $\tau=1$ point is used, but arm 1 is selected only when $D>0.5$ points. Consider a hypothetical observed contrast $d=1$ point and evaluate its conditional percentile at $\delta=0$. First, the selection probability under $\delta=0$ is

P_0(D>0.5)=1-\Phi(0.5)=1-0.69146246\ldots=0.30853754\ldots.

Since $d=1$ exceeds the threshold, its conditional percentile is

F_0(1\mid D>0.5) = \frac{\Phi(1)-\Phi(0.5)}{1-\Phi(0.5)} = \frac{0.84134475\ldots-0.69146246\ldots}{0.30853754\ldots} \approx 0.4858.

So the observed contrast is near the conditional median under $\delta=0$ for this stricter rule, even though its unconditioned standard normal percentile is about 84.13%. The event defining selection must therefore appear in both the conditional distribution and the coverage interpretation.

Width, Diagnostics, and Alternatives

Conditional intervals need not be narrower or wider in every possible data configuration; their width depends on the observed contrast, threshold, standard error, and confidence level. Selection can make a nominally ordinary interval poorly calibrated for the selected sample. In the numerical example, the conditional interval was wider, but width alone is not evidence that one interval is more appropriate: the two intervals address different coverage statements.

Before using this construction, verify the following:

The event is prespecified and represented correctly. If the actual rule includes a threshold, tie-breaking rule, additional interim statistic, or other condition, conditioning only on $D>0$ may not match the selection process.
The distributional model is credible. The derivation assumes a normal contrast with the stated variance. Dependence between arm estimates, an estimated variance, or a non-normal outcome may require a different conditional distribution.
The statistic used in the interval matches the selection statistic. If selection depends on more information than $D$, the distribution of the reported estimate conditional only on $D>c$ may fail to reflect the actual selection event.
Numerical roots are checked. Confirm that the lower endpoint gives conditional percentile $1-\alpha/2$, the upper gives $\alpha/2$, and both are on the correct side of one another. Use adequate numerical precision, especially when the selection event is rare.

An ordinary Wald interval is simpler and may be appropriate when the intended guarantee is unconditional under a fixed analysis, but it does not automatically deliver conditional coverage after selection. Test inversion provides a general way to build intervals from acceptance rules; here the tests are based specifically on the conditional distribution. Stagewise-ordering limits are another ordering-based construction, but they need not produce the same endpoints as conditional equal-tailed inversion. The choice depends on the inferential target and ordering, not solely on which interval is shortest.

A conditional interval also has limitations. It answers a conditional question and does not state what coverage averaged over selected and unselected trials would be. If selection is extremely unlikely at a parameter value, conditioning focuses on a rare subset of repetitions, and the resulting interval may be wide or numerically sensitive. Exact conditional coverage is only as reliable as the model, selection rule, and variance assumptions. It is not a correction for a misspecified endpoint, an unrecorded adaptation, or an inaccurately modeled outcome distribution.

Key takeaway: To obtain conditional coverage after a prespecified selection event, derive the distribution of the inferential statistic given that event, then invert its conditional CDF. State the event and the conditional coverage target explicitly when reporting the resulting interval.

Check Your Understanding

Answer each question, then compare with the solutions below.

In the two-arm model, what parameter does the interval based on $D=\bar{x}_1-\bar{x}_2$ estimate, and what is the selection event when $c=0$?
If $\tau=1$, $c=0$, $\delta=0$, and $d=0.5$, calculate the conditional percentile $F_0(d\mid D>0)$.
For a 90% equal-tailed conditional interval, which conditional percentiles determine the lower and upper parameter endpoints?
Why does a positive observed contrast not imply that the conditional interval must exclude zero?
Name one model or design feature that can invalidate the conditional CDF used in this tutorial.

Solutions. (1) It estimates the signed contrast $\delta=\mu_1-\mu_2$; at $c=0$, arm 1 is selected when $D>0$. (2) $F_0(0.5\mid D>0)=[\Phi(0.5)-0.5]/0.5=(0.69146246\ldots-0.5)/0.5\approx0.3829$. (3) With $\alpha=0.10$, the lower endpoint solves $F_{\delta_L}(d\mid D>c)=0.95$, and the upper solves $F_{\delta_U}(d\mid D>c)=0.05$. (4) The selection rule constrains the observed estimate, not the true parameter; sampling variation can produce a positive selected contrast when $\delta\le0$. (5) Examples include dependent arm estimates when independence was assumed, an estimated rather than known variance, or a selection rule based on additional statistics not represented in the conditional event.

Conditional Confidence Intervals After Selection

What You'll Learn