Confidence Intervals for a Selected Treatment Effect

Why Selection Changes an Interval

In a multi-arm, multi-stage trial, an arm may be chosen for further study because its interim estimate looks favorable, its test statistic is large, or it meets a prespecified efficacy rule. The estimate for that arm is then not just an estimate for a fixed treatment: it is the estimate associated with a data-dependent choice. This distinction matters for confidence intervals because selected estimates tend to be unusually favorable, even when the underlying treatment effects are not.

Let $\theta_k$ be the treatment effect for arm $k$, with positive values representing benefit, and let $\hat\theta_k$ be its estimated effect. Write $J$ for the arm selected by the trial rule. A confidence interval for $\theta_J$ should reflect how the estimate was obtained and what data were used to select $J$. An ordinary interval treats the selected arm as if it had been specified before observing the data. A selection-adjusted interval accounts for selection in its construction, or uses data independent of the selection decision.

Definition: An ordinary interval for a selected treatment effect applies a standard fixed-arm interval formula to the estimate $\hat\theta_J$, without adjusting for the selection rule. A selection-adjusted interval changes the data or uncertainty calculation to account for the selection process, or bases inference on data independent of selection. The term alone does not guarantee a particular coverage property: that depends on the method, assumptions, and coverage target.

A common working model for planning or illustrating these intervals is that the vector of arm-effect estimates is approximately multivariate normal, with mean vector $(\theta_1,\ldots,\theta_K)$ and a known or consistently estimated covariance matrix. Dependence matters in MAMS trials, especially when arms share a control group or are compared at multiple stages. Any method must use standard errors and covariances that match the data actually included in the estimate.

Ordinary Intervals and the Selection Effect

For a prespecified arm with a normally distributed estimate and known standard error $SE_k$, an ordinary two-sided $100(1-\alpha)\%$ interval is

\hat\theta_k \mathbin{\pm} z_{1-\alpha/2}SE_k,

where $z_{1-\alpha/2}$ is the corresponding standard-normal quantile. With an estimated standard error, a suitable $t$ quantile may be used under the relevant model assumptions. Neither substitution, by itself, adjusts for choosing the arm based on the same data.

Worked Example: An Ordinary Interval for the Largest Estimate

Suppose a hypothetical three-arm trial compares each treatment with a common control. At an interim analysis, estimated benefits are 1.40, 1.00, and 0.80 points, each with standard error 0.50 points. The prespecified rule continues the arm with the largest estimated benefit, so arm A is selected. For illustration, assume the estimates are normally distributed and the standard errors are known.

If arm A had been fixed in advance, its ordinary 95% interval would be

1.40 \mathbin{\pm} 1.96(0.50) =1.40\mathbin{\pm}0.98 =[0.42,\ 2.38]\text{ points}.

This arithmetic is correct for the fixed-arm formula. But arm A was selected because its estimate was the largest of the three. The interval does not account for that selection, so its usual fixed-arm interpretation is not justified for the selected effect merely because the formula is familiar. In particular, the calculation does not establish that the interval has 95% coverage conditional on arm A being selected.

The mechanism can be seen with a simple threshold rule. Suppose $Y\sim N(\theta,\sigma^2)$, and an arm is selected when $Y>c$. Define $Z=(Y-\theta)/\sigma$ and $a=(c-\theta)/\sigma$. Conditional on selection, $Z>a$, and direct integration gives

E(Z\mid Z>a) =\frac{\int_a^\infty z\phi(z)\,dz}{1-\Phi(a)} =\frac{\phi(a)}{1-\Phi(a)} =\lambda(a)>0.

The equality follows because $\phi'(z)=-z\phi(z)$, so $\int_a^\infty z\phi(z)\,dz=\phi(a)$. Thus $E(Y\mid Y>c)=\theta+\sigma\lambda(a)$: among estimates that pass the threshold, the observed values average above the true effect. Choosing the largest among several estimates can create the same upward-selection tendency. This argument motivates adjustment; it does not, by itself, specify a confidence interval.

Intervals Using Independent Validation Data

One direct approach is to make the selection using screening data and estimate the selected effect using separate validation data. The essential condition is independence: conditional on the screening data and the selected arm, the validation estimate must have the stated sampling distribution. A clean data split, or an independently collected validation stage, can provide this separation. If validation data reuse participants, controls, or outcomes from screening, independence cannot simply be assumed.

Result: Suppose the selected arm $J$ is determined by screening data $S$. For each arm $k$, suppose an independent validation estimate satisfies $\hat\theta^{(V)}_k\sim N(\theta_k,SE_{V,k}^2)$, with known $SE_{V,k}$, and the validation data are independent of $S$. Then $$ \hat\theta^{(V)}_J\mathbin{\pm}z_{1-\alpha/2}SE_{V,J} $$ has coverage $1-\alpha$ for $\theta_J$, conditional on the screening data, under this model.

To justify the result, fix the observed screening data. They determine a particular arm $J=j$. By independence, the validation estimate for that arm still has its stated normal distribution with mean $\theta_j$ and standard error $SE_{V,j}$. Standardization therefore gives

P\left( \left|\frac{\hat\theta^{(V)}_j-\theta_j}{SE_{V,j}}\right| \le z_{1-\alpha/2} \ \middle|\ S \right)=1-\alpha.

This argument fails if selection used the validation outcomes, or if the validation estimate's distribution changes after conditioning on the screening data. It also assumes the effect scale, variance model, and analysis are correctly specified. Independent validation is conceptually simple, but it can be less precise than using all available observations and may require additional participants or stages.

Worked Example: A 95% Validation-Stage Interval

In a hypothetical trial, screening data select arm B. An independent validation stage estimates B's benefit over control as 0.90 points, with a known standard error of 0.40 points. Assume normally distributed validation errors, no overlap with screening outcomes, and an analysis specified before the validation data are examined.

The validation-based 95% interval is

0.90\mathbin{\pm}1.96(0.40) =0.90\mathbin{\pm}0.784 =[0.116,\ 1.684]\text{ points}.

Under the stated assumptions, this interval has 95% coverage for arm B's effect conditional on the screening data that selected B. It is not an interval for the screening estimate, and the calculation does not use that estimate as if it were independent validation evidence.

Intervals Around Bias-Corrected Estimates

When the selected estimate itself is used, a common strategy is to correct it for estimated selection bias and then quantify uncertainty in the corrected estimate. For example, earlier material on plug-in selection corrections and truncation corrections motivates subtracting an estimated bias. Write the corrected estimate as $\tilde\theta_J=\hat\theta_J-\hat b_J$, where $\hat b_J$ estimates the selection bias for the selected arm. This does not automatically produce an exact confidence interval. Both the correction and its uncertainty must be accounted for.

If $\hat\theta_J$ and $\hat b_J$ are jointly approximately normal, then

\operatorname{Var}(\tilde\theta_J) =\operatorname{Var}(\hat\theta_J) +\operatorname{Var}(\hat b_J) -2\operatorname{Cov}(\hat\theta_J,\hat b_J).

An approximate Wald interval is $\tilde\theta_J\mathbin{\pm}z_{1-\alpha/2}\widehat{SE}(\tilde\theta_J)$, using an estimated standard error that reflects this variance. If the correction is a nonlinear function of data or estimated nuisance parameters, a delta-method calculation can be used when its regularity and approximation assumptions are reasonable. Omitting the variance of the correction, or ignoring covariance when the correction uses the same data, can make the interval too narrow or otherwise miscalibrated.

Worked Example: A Plug-In Correction With Its Uncertainty

Suppose a hypothetical selected-arm analysis gives $\hat\theta_J=1.25$ points, with standard error 0.25 points. A prespecified model-based correction estimates the selection bias as $\hat b_J=0.28$ points, with standard error 0.10 points. For this illustration, assume the correction estimate is independent of $\hat\theta_J$, and that the corrected estimate is approximately normal.

The corrected estimate is

\tilde\theta_J=1.25-0.28=0.97\text{ points}.

Under the independence assumption, the estimated variance is the sum of the two variances:

\widehat{\operatorname{Var}}(\tilde\theta_J) =0.25^2+0.10^2 =0.0625+0.0100 =0.0725, \qquad \widehat{SE}(\tilde\theta_J)=\sqrt{0.0725}=0.2693\text{ points}.

The resulting approximate 95% interval is

0.97\mathbin{\pm}1.96(0.2693) =0.97\mathbin{\pm}0.5278 =[0.442,\ 1.498]\text{ points}.

This interval incorporates the stated uncertainty in the correction. Its validity remains approximate and depends on the assumed normal approximation, the correctness of the bias correction, and the independence assumption. If the correction were estimated from the same data, the covariance term would need to be included rather than set to zero by convenience.

Choosing and Checking an Interval

No single interval construction is best for every MAMS selection rule. Independent validation gives a particularly transparent route to inference but uses only the validation information for the interval. A bias-corrected Wald interval can use more data, but it relies on a credible correction model and a sound estimate of the corrected estimator's variance. Other approaches condition on the selection event or use the full selection path; their assumptions and coverage targets must be stated explicitly. A correction to the point estimate alone is not a substitute for an interval method.

Approach	What it uses	Main requirement or limitation
Ordinary interval	Selected estimate and its usual standard error	Does not account for selection using the same data
Independent validation interval	Selected arm and independent validation estimate	Needs valid separation between selection and estimation data
Bias-corrected approximate interval	Selected estimate, estimated bias, and correction uncertainty	Depends on a suitable correction model and variance calculation
Selection-event-based methods	Observed data and the specified selection rule	Requires modeling the selection event and specifying the coverage target

Before reporting an interval, check the following:

Reconstruct the selection rule. Confirm which interim estimates, test statistics, thresholds, and stopping decisions determined the reported arm.
Match the data to the method. Check whether the estimate includes screening data, later-stage data, or both, and whether those data also drove selection.
Check dependence. Shared controls and repeated stages can induce covariance across arm estimates. A standard error calculated as if the arms were independent may be wrong.
Audit the correction. Confirm that the correction corresponds to the actual selection rule and that uncertainty in its parameters has not been ignored.
State the target. Explain whether the interval is intended to cover the selected arm's effect under repeated use of the design, under a specified selection event, or under another clearly defined scheme.

A frequent mistake is to say, “The selected arm's ordinary 95% interval is valid because each arm had a 95% interval.” The fixed-arm statement concerns an arm chosen independently of the estimates used to form its interval. If the largest estimate is chosen first and that same estimate is then centered in the interval, the fixed-arm argument does not establish 95% coverage for the selected effect. The correction is to use a method whose assumptions match the selection process, and to report its actual coverage target rather than assuming that a familiar formula retains its usual interpretation.

Key takeaway: Selection affects interval estimation as well as point estimation. Use an ordinary interval only when its fixed-arm assumptions are appropriate; otherwise use independent estimation data or a selection-aware procedure that accounts for both the selection mechanism and the uncertainty in any adjustment.

Check Your Understanding

Answer each question, then compare with the brief solutions.

A trial selects the arm with the largest interim estimate, then reports that estimate plus or minus 1.96 standard errors. What feature of the design makes the usual fixed-arm interpretation questionable?
What independence condition allows an interval based on validation data to retain its nominal coverage after screening selects an arm?
A corrected estimate is $\hat\theta-\hat b$. Write its variance in terms of the variance of each component and their covariance.
In the validation example, what is the interval's interpretation under the stated model and data-separation assumptions?

Solutions. (1) The same interim data determine which estimate is reported and enter the interval, so the selected estimate is not treated as a prespecified-arm estimate. (2) The validation estimate must be independent of the screening data that determine the selected arm, with the stated validation sampling distribution still valid after selection. (3) $\operatorname{Var}(\hat\theta-\hat b)=\operatorname{Var}(\hat\theta)+\operatorname{Var}(\hat b)-2\operatorname{Cov}(\hat\theta,\hat b)$. (4) It is a 95% interval for the selected arm's effect conditional on the screening data, provided the normal model, standard error, and independence assumptions hold.