Conditional and Marginal Coverage After Selection

Why the Conditioning Event Matters

A confidence interval can have good coverage on average across repeated trials and still cover poorly when attention is restricted to trials in which a particular arm was selected. This distinction matters in a multi-arm, multi-stage trial because the reported treatment effect is not fixed in advance: it depends on the interim data and the selection rule. Coverage describes repeated-sampling performance, so the event on which that performance is conditioned must be stated.

Here, conditional coverage means the probability that the reported interval covers its target, given a specified selection outcome. Marginal coverage averages the coverage event over all possible selection outcomes under the model. Neither is automatically the relevant guarantee in every application. The question is whether the scientific claim concerns overall repeated-trial performance or performance specifically after a named arm or path has been selected.

Definition: Let \(S\) denote the selected arm, \(\theta_k\) the treatment effect for arm \(k\), and \(I_S\) the interval reported for the selected arm. The coverage event is \(C=\{\theta_S\in I_S\}\). For an arm \(k\) with \(P(S=k)>0\), conditional coverage is \(P(C\mid S=k)\). Marginal coverage is \(P(C)=\sum_k P(C\mid S=k)P(S=k)\), when exactly one arm is selected and an interval is reported for it.

In a multi-stage design, the selection outcome can be more detailed than a single arm label: it may include which arms continued, when stopping occurred, and which arm was ultimately reported. The same probability identity applies to any mutually exclusive, exhaustive set of outcomes. If some outcomes produce no interval, define the target performance explicitly—for example, coverage among trials that report an interval—and average over the outcome probabilities conditional on reporting.

The Probability Identity and Its Interpretation

The marginal probability follows by partitioning the coverage event according to which arm was selected. For each \(k\), \(C\cap\{S=k\}\) is the event that arm \(k\) is selected and its interval covers \(\theta_k\). These events are disjoint, so their probabilities add.

P(C)=\sum_{k=1}^{K}P(C\cap\{S=k\}) =\sum_{k=1}^{K}P(C\mid S=k)P(S=k).

This is the law of total probability, not an assumption that the conditional coverages are equal. A marginal coverage near 95% does not establish 95% coverage after each arm-specific selection outcome. Conversely, one arm’s conditional coverage can be below 95% while the marginal rate is near 95%, if other outcomes have higher coverage and sufficient probability.

The selection rule and the interval procedure jointly determine conditional coverage. If the data used to select an arm are independent of the data used to construct its interval, conditioning on selection need not distort the interval’s error distribution. If the same noisy estimates drive both selection and interval construction, selected estimates tend to be unusually high, and ordinary fixed-arm intervals can have different coverage after selection than before it.

Worked Examples

Worked Example: Independent Validation Data

Suppose a trial screens three arms and selects one using interim data. The final estimate for the selected arm comes from independent validation data, with a normal error of known standard error 2 points. For whichever arm is selected, the interval is the estimate plus or minus \(1.96\) standard errors. Under the model, the validation error for each fixed arm is normal with mean zero and standard deviation 2, and is independent of the screening data. Find conditional and marginal coverage.

Because the screening data are independent of validation errors, conditioning on \(S=k\) does not change the standardized validation error \(Z_k=(\hat{\theta}_k-\theta_k)/2\). Thus \(Z_k\sim N(0,1)\) conditional on selecting arm \(k\). The interval covers when \(-1.96\leq Z_k\leq1.96\), giving

P(C\mid S=k)=\Phi(1.96)-\Phi(-1.96) \approx 0.975002105-0.024997895 =0.950004210.

So conditional coverage is approximately 95.0004% for each arm. The exact central 95% normal interval would use \(c=\Phi^{-1}(0.975)\approx1.959964\), rather than the rounded value 1.96. Since every possible selected arm has the same conditional coverage, the weighted average is also approximately 95.0004%, whatever the arm-selection probabilities are. Independence of validation and screening is the key assumption; without it, this argument does not follow.

Worked Example: Reusing the Estimates That Select the Arm

Now suppose three treatment arms have equal true effects, and their selection estimates \(X_1,X_2,X_3\) are independent \(N(0,1)\) variables. Select the arm with the largest estimate, \(S=\arg\max_i X_i\), and report \(I_S=[X_S-1.96,\ X_S+1.96]\). Calculate conditional and marginal coverage. Ties have probability zero in this continuous model.

For a fixed arm, before selection, an interval of this form covers its true effect with probability \(\Phi(1.96)-\Phi(-1.96)\), approximately 0.9500042. After selection, however, the selected estimate is the maximum. The interval covers the common true effect zero precisely when the maximum lies between \(-1.96\) and \(1.96\). By independence,

P(C)=P(-1.96\leq \max_i X_i\leq1.96) =\Phi(1.96)^3-\Phi(-1.96)^3 \approx 0.9268498.

The subtraction uses \(\Phi(1.96)\approx0.975002105\) and \(\Phi(-1.96)\approx0.024997895\); the resulting coverage is about 92.685%, not 95%. For example, the probability that all three estimates are at most \(1.96\) is \(\Phi(1.96)^3\); the probability that all three are below \(-1.96\) is \(\Phi(-1.96)^3\). Their difference is the probability that the maximum falls within the interval.

By symmetry each arm is selected with probability \(1/3\), and its conditional coverage given selection is the same as for the other arms. Therefore each conditional coverage equals the marginal coverage, approximately 0.9268498. More explicitly, the total coverage is the sum of three equal joint probabilities \(P(C,S=k)\), and each is one third of the total; dividing by \(P(S=k)=1/3\) gives \(P(C\mid S=k)=P(C)\). Symmetry makes the conditional rates equal here; it does not restore the fixed-arm 95% rate.

Worked Example: Marginal Coverage Can Hide Different Conditional Rates

Consider a two-arm screening process that selects arm A with probability 0.70 and arm B with probability 0.30. Assume the validation estimate for the selected arm is independent of screening and normally distributed about its true effect, with known standard error 1 unit. The prespecified interval for A uses the central 90% normal critical value \(z_{0.95}\), while the interval for B uses the central 98% critical value \(z_{0.99}\). Find the conditional and marginal coverage.

Under the stated normal and independence assumptions, the probability of coverage for A conditional on its selection is 0.90; for B it is 0.98. The intervals have different nominal levels by design, so equal conditional coverage is not expected. Applying the probability-weighted average gives

P(C)=P(C\mid S=A)P(S=A)+P(C\mid S=B)P(S=B) =(0.90)(0.70)+(0.98)(0.30)=0.924.

The marginal coverage is 92.4%, while the two conditional rates are 90% and 98%. This example is deliberately simple: it shows that an overall rate is a weighted average, not a description of each arm’s performance. In practice, arm-specific conditional rates arise from the selection mechanism, interval construction, or both; they need not be deliberately assigned different nominal levels.

Deriving Conditional Coverage Under Normal Selection

The preceding same-data example has a useful generalization. Suppose \(X_i\) are independent estimates with \(X_i\sim N(\theta_i,v^2)\), where the common standard error \(v\) is known, and the largest estimate is selected. For a particular arm \(k\), set \(Z_k=(X_k-\theta_k)/v\), so \(Z_k\sim N(0,1)\). Conditional on \(Z_k=z\), arm \(k\) is selected exactly when every other estimate is no larger than \(X_k=\theta_k+vz\). Independence gives the conditional probability of that event as a product of normal CDF values.

P(S=k\mid Z_k=z) =\prod_{j\ne k}\Phi\!\left(z+\frac{\theta_k-\theta_j}{v}\right).

A fixed-width interval \(X_k\pm cv\) covers \(\theta_k\) exactly when \(-c\leq Z_k\leq c\). Integrating the joint probability of selection and coverage over this range, then dividing by the probability of selection, yields

P(C\mid S=k)= \frac{\displaystyle\int_{-c}^{c}\phi(z) \prod_{j\ne k}\Phi\!\left(z+\frac{\theta_k-\theta_j}{v}\right)\,dz} {\displaystyle\int_{-\infty}^{\infty}\phi(z) \prod_{j\ne k}\Phi\!\left(z+\frac{\theta_k-\theta_j}{v}\right)\,dz}.

Here \(\phi\) and \(\Phi\) are the standard normal density and CDF. The denominator is \(P(S=k)\); the numerator is \(P(C,S=k)\). The ratio makes the source of the change explicit: selection weights values of \(Z_k\) according to how likely they are to make arm \(k\) the winner. When true effects differ, these weights differ across arms, so conditional coverages need not be equal. When estimates are correlated, as can occur with shared controls, the product formula does not apply; the joint distribution must account for that correlation. The probability-weighted identity for marginal coverage still applies.

Diagnostics, Alternatives, and Limitations

A useful diagnostic is to tabulate or estimate coverage separately by the outcome on which the analysis conditions: selected arm, selection rank, or complete selection path. A single marginal rate can mask a weak conditional rate. In simulation studies, report the number of replicates in each outcome as well as its estimated coverage; a conditional estimate based on very few replicates is itself imprecise. Check that the simulated selection rule matches the actual interim rule, including stopping and tie-breaking, and that the interval is evaluated against the correct arm-specific effect.

The fixed-arm interval is a natural comparator: it describes coverage when an arm is specified independently of the data used for its estimate. It may be appropriate for independent validation estimates, as in the first example. Applying that same interval to a selected estimate is a different procedure and need not retain the fixed-arm coverage. Selection-adjusted intervals target inference accounting for the selection event; simultaneous intervals instead address joint coverage across a collection of arms. These are different targets, and their coverage statements should not be interchanged.

Conditioning is not the same as changing the estimand. \(P(C\mid S=k)\) concerns coverage of \(\theta_k\) in trials selecting arm \(k\); it is not coverage for a new parameter defined by the selected estimate.
Marginal validity does not imply arm-specific validity. The weighted average can meet a target while one or more conditional rates fall below it.
Marginal undercoverage can occur too. The equal-effect, three-arm example has both conditional and marginal coverage below the fixed-arm nominal rate because the selected maximum tends to be high.
Model dependence matters. Normality, known standard errors, independence, and the exact selection rule supported the displayed calculations. Correlation, estimated standard errors, interim stopping, or model misspecification can change conditional rates.

A common mistake is to argue, “The interval has 95% fixed-arm coverage, therefore it has 95% coverage whenever that arm is selected.” That conclusion treats selection as irrelevant without checking whether the interval data are independent of the selection event. The correction is to specify the conditioning event, derive or estimate \(P(C\mid S=k)\) under the joint model, and report marginal coverage separately when it is also of interest. The introductory material on confidence intervals for a selected treatment effect motivates the interval target; the additional issue here is how its coverage varies across selection outcomes.

Check Your Understanding

Answer each question, then compare with the solutions below.

If conditional coverage is 0.88 when arm A is selected with probability 0.25 and 0.96 when arm B is selected with probability 0.75, what is marginal coverage?
In the three-arm equal-effect example, why is the conditional coverage the same for each arm?
What key assumption supports the approximately 95% conditional coverage in the independent-validation example?
For three independent \(N(0,1)\) estimates, what event corresponds to coverage of the interval around the selected maximum?
Does 95% marginal coverage by itself establish 95% coverage conditional on every selection path? Explain.

Solutions. (1) \(0.25(0.88)+0.75(0.96)=0.94\), so marginal coverage is 94%. (2) The arm distributions and selection rule are symmetric, so relabelling the arms leaves the conditional coverage unchanged. (3) The validation error is independent of the screening data that determine selection, so conditioning on the selected arm does not alter its standardized validation-error distribution. (4) The maximum must lie between \(-1.96\) and \(1.96\). (5) No. Marginal coverage is a probability-weighted average, and different selection paths can have different conditional rates.