Selective Confidence Intervals by Test Inversion

Why invert tests after selection?

In a multi-arm, multi-stage trial, an arm may be selected because its interim evidence looked promising. An estimate from that arm is therefore not, in general, distributed like an estimate from an arm specified in advance. Even if the treatment-effect estimator has a normal distribution before selection, observing that an arm passed a boundary or ranked first changes its distribution. An interval built from the unselected distribution can consequently have poor coverage conditional on the selection that actually occurred.

A direct way to address this is to test each proposed value of the selected treatment effect using a null distribution conditioned on the observed selection event. Inverting those tests gives a selective confidence set. The target here is one selected treatment effect; simultaneous coverage for multiple selected arms is a separate problem.

The event \(E\) must represent the rule that generated the reported selection. In a simple illustration it may be “the estimate exceeded a threshold.” In a MAMS analysis it may include the interim comparisons, the arm retained at each stage, and the stopping decision. Conditioning on the wrong event does not correct the selection that took place. The conditional distribution is defined only when \(P_{\theta}(E)>0\) for the parameter values being considered.

Conditional distributions and test inversion

Let \(T\) be a statistic for the selected arm, and suppose first that all nuisance parameters are known. For a proposed value \(\theta_0\), a one-sided upper-tail conditional p-value is

It measures how often a statistic at least as large as the observed one would occur if the proposed effect were true, among repetitions in which the selection event also occurred. A two-sided procedure can be formed from two conditional tails. For a continuous statistic with conditional cumulative distribution function \(F_{\theta}(t\mid E)\), the equal-tailed \(100(1-\alpha)\%\) confidence set is

Equivalently, values of \(\theta\) are retained when neither conditional tail is too small. If the set is an interval, its endpoints solve the two tail equations at \(\alpha/2\) and \(1-\alpha/2\). In more complicated models the set need not be connected; an interval spanning separated pieces is a conservative summary, not the exact inverted set.

A derivation in a truncated-normal model

Suppose an estimate \(Y\) has distribution \(Y\sim N(\theta,1)\), with known standard error 1, and the arm is reported only if \(Y>c\). For an observed \(y>c\), the conditional density is the original normal density restricted to the selection region and divided by the probability of selection:

Here \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution function. Integrating this density from \(c\) to \(y\) gives

The denominator is essential: it accounts for the fact that only values above the selection threshold could have been reported. For a continuous conditional distribution, the probability integral transform says that \(F_{\theta}(Y\mid E)\) is uniform on \([0,1]\), conditional on \(E\), when \(\theta\) is the true parameter. Thus, the probability that this quantity falls outside \([\alpha/2,1-\alpha/2]\), conditional on selection, is \(\alpha\). This establishes the conditional coverage of the equal-tailed set in this model.

Conditional tests with more than one arm

Selection in MAMS trials is often based on comparisons among arms rather than a single fixed threshold. For example, an arm may be selected because its interim statistic is the largest. Even in a one-stage normal illustration, the selected statistic has a different distribution from an unselected one.

In a general MAMS analysis, let \(X\) collect the stagewise estimates or test statistics, and let \(E\) encode the observed selection path. If \(X\) has density \(f_{\theta,\eta}(x)\), where \(\eta\) denotes nuisance parameters, then the conditional density on the event is

The denominator is the probability, under that parameter configuration, of the selection event. Shared controls and repeated stagewise observations can make the components of \(X\) correlated, so the joint distribution—not a product of marginal distributions—must be used when the design creates dependence. Inverting tests based on an incorrect independence assumption will generally not deliver the claimed conditional coverage.

Nuisance parameters, computation, and diagnostics

Conditional distributions may depend on effects of other arms, correlations, variances, or other nuisance parameters. Treating these quantities as known when they are not can make a nominal selective interval fail to have its advertised coverage. If nuisance parameters can be eliminated by conditioning on an appropriate statistic, that can yield a nuisance-free test, but such a statistic is not always available.

One conservative option is to take the supremum of each conditional tail probability over the nuisance-parameter values allowed under the null. For example, use \(p_U(\theta_0)=\sup_{\eta}P_{\theta_0,\eta}(T\geq t_{\mathrm{obs}}\mid E)\) for an upper-tail test. Because this is at least as large as the tail probability at the true nuisance value, it cannot make rejection easier than the correctly specified conditional test. For a two-sided equal-tailed construction, take the supremum of each tail separately and retain \(\theta_0\) only when both are greater than \(\alpha/2\). This can be conservative, sometimes substantially so.

A practical calculation should be checked against the design and the probability model. In particular:

• Verify that \(E\) includes the actual stagewise selection and stopping information relevant to the reported estimate.
• Check that the model includes the covariance induced by shared controls or repeated measurements, when present.
• Confirm that \(P_{\theta,\eta}(E)\) is positive and that numerical integration accurately evaluates both the conditional numerator and denominator.
• Plot or otherwise inspect the inverted tail probabilities over a range of \(\theta\). Unexpected non-monotonicity can produce multiple pieces of a confidence set; it should not be hidden by reporting only one pair of roots.
• Examine sensitivity to nuisance-parameter values and to plausible changes in the selection rule. Small selection probabilities can make conditional calculations especially sensitive to model and numerical error.

How this interval differs from alternatives

The introductory material on confidence intervals for a selected treatment effect distinguishes selection-adjusted targets and coverage ideas. Test inversion specifies a concrete construction: for each candidate effect, calibrate a test under the conditional distribution and retain values not rejected. A marginal interval instead aims for coverage averaged over selection and non-selection outcomes; that target need not provide nominal coverage among trials in which an arm was selected. The choice depends on the inferential question.

An independent validation sample can avoid conditioning the estimate on the data that selected the arm, provided the validation data and analysis satisfy the required independence and model assumptions. It may, however, use less information or be less precise. Conditional inversion uses the selection data directly, but requires the selection mechanism and joint sampling model to be represented correctly. Neither approach repairs a misspecified outcome model or an inaccurately recorded selection rule.

A common mistake is to say, “The selected estimate is 2.4 with standard error 1, so a 95% interval is \(2.4\pm1.96=[0.44,4.36]\).” The arithmetic is \(2.4-1.96=0.44\) and \(2.4+1.96=4.36\), but the interval uses the unselected normal distribution. It does not account for the threshold that made this arm eligible to be reported. Under the hypothetical threshold model above, the equal-tailed conditional interval is instead approximately \([-0.588,4.353]\). The ordinary interval may be useful for a different coverage target, but it should not be described as a conditional selective interval.