Confidence Interval Coverage and Its Conditioning

Why the Conditioning Event Matters

An interval procedure is often described as having 95% coverage. In an adaptive trial, that statement is incomplete unless we know over which trial outcomes the coverage probability is calculated. Is it averaged over every possible outcome, including outcomes that do not lead to selecting or advancing a treatment? Or is it calculated only among outcomes that do lead to that adaptation decision? Those are different probabilities.

The distinction is important when selection and estimation use the same data. An estimate that passes a high selection threshold tends to be unusually large relative to its true value. An interval centered on that estimate may therefore miss the true value more often among selected outcomes than across all outcomes. This tutorial isolates that issue in a simple selection design. It does not claim that one coverage target is always preferable: the relevant target depends on the inferential question and on when the interval will be used.

Definition: Let $\theta$ be a fixed parameter, $C(X)$ an interval computed from trial data $X$, and $S$ an event describing an adaptation decision, such as selecting a treatment. Unconditional coverage is $P_\theta\{\theta\in C(X)\}$, calculated over the full trial-outcome distribution. Conditional coverage given selection is $P_\theta\{\theta\in C(X)\mid S\}$, calculated only among outcomes for which $S$ occurs. The joint probability $P_\theta(\{\theta\in C(X)\}\cap S)$ is the probability of both coverage and selection; it is the numerator in the conditional-coverage calculation, not the conditional probability itself.

The subscript $\theta$ emphasizes that these probabilities are evaluated under a specified true parameter value. Unconditional coverage averages over selected and nonselected outcomes under that value. Conditional coverage answers a narrower question: among trials that meet the selection rule, how often does the interval contain $\theta$? If an interval is produced only after selection, report the conditional probability when describing coverage among reported intervals. The probability of reporting an interval that covers is instead the joint probability.

A Normal-Theory Selection Model

Assume a hypothetical trial produces an estimate $\hat{\theta}$ that, at the parameter value under study, has a normal sampling distribution with known standard error $\tau$:

\hat{\theta}\sim N(\theta,\tau^2), \qquad Z=\frac{\hat{\theta}-\theta}{\tau}\sim N(0,1).

Consider the usual two-sided normal interval

C(X)=\left[\hat{\theta}-1.96\tau,\ \hat{\theta}+1.96\tau\right].

Under this model, before conditioning on any adaptation, the interval covers $\theta$ exactly when $-1.96\leq Z\leq1.96$, an event with probability approximately 0.9500. The selection decision in our example is to advance the treatment if $\hat{\theta}$ exceeds a prespecified benchmark $b$. We evaluate coverage at the true value $\theta=b-\tau$, so the selection event is $Z>1$. The decision rule uses the observed estimate and a fixed benchmark; the true parameter is used only to evaluate the resulting probability.

Result: If selection is $S=\{Z>c\}$, and $-1.96<c<1.96$, then for this interval $$ P_\theta\{\theta\in C(X)\mid S\} =\frac{\Phi(1.96)-\Phi(c)}{1-\Phi(c)}, $$ where $\Phi(z)=P(Z\leq z)$ for $Z\sim N(0,1)$. The numerator is the joint probability of coverage and selection, $P_\theta(\{\theta\in C(X)\}\cap S)$. The denominator is the probability of selection, $P_\theta(S)$.

Deriving the Conditional-Coverage Formula

The conditional probability definition gives $P(A\mid S)=P(A\cap S)/P(S)$ when $P(S)>0$. Here $A$ is the coverage event $\{-1.96\leq Z\leq1.96\}$. If $S=\{Z>c\}$ and $c$ lies between the interval limits, both events occur exactly when $c<Z\leq1.96$. Therefore:

\begin{aligned} P_\theta(A\cap S) &=P(c<Z\leq1.96)\\ &=\Phi(1.96)-\Phi(c),\\ P_\theta(S)&=P(Z>c)=1-\Phi(c),\\ P_\theta(A\mid S) &=\frac{P_\theta(A\cap S)}{P_\theta(S)} =\frac{\Phi(1.96)-\Phi(c)}{1-\Phi(c)}. \end{aligned}

This derivation also shows why a joint probability alone is not conditional coverage. The joint probability counts covering outcomes among all possible trials; dividing by the selection probability changes the reference set to selected trials only. If selection is so stringent that $c\geq1.96$, no selected outcome has coverage (apart from a probability-zero boundary), so conditional coverage is zero. If the cutoff is outside the interval’s lower limit, the numerator must be computed from the actual intersection of the two events rather than by applying the displayed formula outside its stated range.

Worked Example: Coverage Among Selected Trials

Suppose a hypothetical trial has $\tau=2$ outcome units and the true parameter is $\theta=10$ units. The treatment is advanced if its estimate exceeds $b=12$ units. Thus selection is $\hat{\theta}>12$, or $Z>1$. The interval is $\hat{\theta}\pm1.96(2)$, with a half-width of 3.92 units. Find its unconditional coverage, its joint probability of coverage and selection, and its conditional coverage among selected trials.

The interval covers when $|Z|\leq1.96$, so unconditional coverage is $P(-1.96\leq Z\leq1.96)=\Phi(1.96)-\Phi(-1.96)\approx0.975002-0.024998=0.9500$.

Both selection and coverage occur when $1<Z\leq1.96$. Using $\Phi(1)=0.841345$, the joint probability is $P(1<Z\leq1.96)=0.975002-0.841345=0.133657$. This is the numerator, not the conditional coverage.

The probability of selection is $P(Z>1)=1-\Phi(1)=1-0.841345=0.158655$. Therefore, $P(\theta\in C(X)\mid S)=0.133657/0.158655\approx0.8424$. The interval covers in about 84.3% of selected trials in this model, despite having 95.0% unconditional coverage. These are different probability statements, not contradictory calculations.

How Coverage Changes Across Adaptation Groups

Coverage conditional on not selecting can also differ from the unconditional value. This is not a separate interval procedure: it is the same interval and the same trial-outcome distribution, examined within the complementary event. The law of total probability requires the unconditional coverage to be the selection-probability-weighted average of the two conditional coverages.

Worked Example: Checking the Weighted-Average Identity

Use the preceding hypothetical design. Among nonselected trials, $Z\leq1$. Coverage and nonselection occur when $-1.96\leq Z\leq1$, whose probability is $\Phi(1)-\Phi(-1.96)=0.841345-0.024998=0.816347$. The probability of nonselection is $\Phi(1)=0.841345$, so coverage conditional on nonselection is $0.816347/0.841345=0.9703$.

Weight each conditional coverage by the probability of its group: $0.158655(0.8424)+0.841345(0.9703)\approx0.133651+0.816357=0.950008\approx0.9500$. More exactly, the weighted terms are the joint coverage probabilities in each disjoint group, and their sum is the unconditional coverage. The selected group has lower coverage here, while the nonselected group has higher coverage; neither conditional value alone equals the overall 95.0%.

Worked Example: A More Stringent Selection Cutoff

Keep the same normal model and interval, but suppose the hypothetical advancement rule requires $Z>1.5$, corresponding to a fixed benchmark $b=\theta+1.5\tau$ at the true parameter value being evaluated. Since $1.5<1.96$, coverage and selection occur when $1.5<Z\leq1.96$. Using $\Phi(1.5)=0.933193$ and $\Phi(1.96)=0.975002$, their joint probability is $0.975002-0.933193=0.041809$.

The selection probability is $1-0.933193=0.066807$, so conditional coverage is $0.041809/0.066807=0.6258$. Thus about 62.6% of selected outcomes cover the true parameter in this example. The nominal unconditional coverage remains 95.0%, because the interval and sampling model have not changed. The conditional coverage is lower than in the first example because the stricter rule retains a more extreme upper tail, much of which lies beyond the upper coverage limit.

Choosing and Reporting a Coverage Target

Unconditional and conditional coverage answer different questions. Unconditional coverage describes performance over repeated full trials under the specified model, including all adaptation outcomes. Conditional coverage describes performance in the subset of trials satisfying a specified event. When a treatment is advanced only after selection, conditional coverage may be especially relevant to the interval presented for that selected treatment. When the goal is to assess a procedure across the full design, unconditional coverage is relevant. The report should name the event and the probability being summarized rather than using “coverage” without qualification.

There is also a useful distinction when an interval is not reported after nonselection. One may define a full-data interval procedure for all outcomes and calculate its unconditional coverage; alternatively, one may define the reporting rule and calculate coverage among reported intervals. The probability that a trial both reports an interval and that interval covers is a joint probability. It equals the probability of reporting multiplied by coverage conditional on reporting. It is not itself the coverage proportion among reported intervals.

In practice, coverage calculations can be checked analytically for simple models or estimated by simulation for a more complex adaptation rule. In a simulation, generate trials under a stated $\theta$, apply the complete selection and interval procedure, and record both selection and whether the interval covers. Estimate unconditional coverage as the number covering divided by all simulated trials. Estimate conditional coverage as the number both selected and covering divided by the number selected. Record the selection count: a small number of selected replicates makes the conditional estimate imprecise even if the total simulation count is large.

These calculations rely on the assumed sampling distribution and on the stated interval rule. In the examples, the estimate is exactly normal, $\tau$ is known, and the interval uses the corresponding 1.96 multiplier. In applications, an estimated standard error, nonnormal outcomes, additional adaptation decisions, or a different estimand can alter both probabilities. The event being conditioned on must also be specified precisely: conditioning on selection of an arm is not automatically the same as conditioning on an early stopping decision or on selection of a subgroup.

Common Mistakes and Diagnostics

Calling the joint probability conditional coverage. For the first example, 0.133657 is the probability of both selection and coverage across all trials. Divide it by $P(S)=0.158655$ to obtain selected-trial coverage of approximately 0.8424.
Using the wrong denominator. Conditional coverage among selected trials divides by the number or probability of selected trials, not by all trials. Unconditional coverage uses all trial outcomes in its denominator.
Assuming a nominal interval retains nominal coverage after selection. The 95% result applies to the unconditioned normal sampling distribution here. Conditioning changes the distribution of $Z$; the selected outcomes do not represent an unselected standard normal sample.
Failing to state what is selected. Give the actual event, such as $S=\{\hat{\theta}>b\}$, and the parameter value at which the probabilities are evaluated. Without those details, a conditional-coverage number is not interpretable.
Ignoring model and simulation uncertainty. Check that the assumed estimator distribution and standard error match the design being evaluated. In simulation, verify that the selected-replicate count is adequate before interpreting a conditional estimate.

Key takeaway: Unconditional coverage is $P_\theta\{\theta\in C(X)\}$. Conditional coverage among selected trials is $P_\theta\{\theta\in C(X)\mid S\}=P_\theta(\{\theta\in C(X)\}\cap S)/P_\theta(S)$. Always distinguish the joint numerator from the conditional probability and state the event that defines the selected group.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the normal example with selection $Z>1$, what event represents joint coverage and selection?
If the joint probability is 0.133657 and selection probability is 0.158655, what is conditional coverage among selected trials?
Why can unconditional coverage be 0.95 while selected-trial coverage is about 0.8424?
For the cutoff $Z>1.5$, identify the numerator and denominator used to calculate conditional coverage.

Solutions. (1) The intersection is $1<Z\leq1.96$, because those outcomes both pass the cutoff and fall inside the interval’s coverage range. (2) Divide the joint probability by the selection probability: $0.133657/0.158655\approx0.8424$. (3) The probabilities use different reference sets: unconditional coverage averages selected and nonselected trials, whereas conditional coverage uses selected trials only. (4) The numerator is $P(1.5<Z\leq1.96)=0.041809$; the denominator is $P(Z>1.5)=0.066807$, giving conditional coverage $0.6258$.