Inference After Adaptive Subgroup Selection

Why Subgroup Selection Changes the Estimation Problem

An interim analysis may show different apparent treatment effects across biomarker-defined subgroups. A trial can respond by selecting one subgroup for continued study. This is a useful adaptation, but it changes the interpretation of the data: the subgroup was selected partly because its interim result looked favorable. An estimate that reuses that result without accounting for selection can therefore overstate the treatment effect.

A clear way to separate the issues is to distinguish which subgroup was selected from how uncertain its treatment effect is. This tutorial develops inference using later-stage observations that are independent of the interim selection. The result is a direct, transparent interval for the selected subgroup’s treatment effect, conditional on that subgroup having been selected. The price is that the estimate does not borrow the selected interim effect estimate.

Definition: Let \(g\in\{1,\ldots,K\}\) index prespecified subgroups. The subgroup-specific treatment effect \(\Delta_g\) is the mean outcome under treatment minus the mean outcome under control for patients in subgroup \(g\), in the target trial population and on the stated outcome scale. Let \(S\) denote the subgroup selected using interim data. Inference after adaptive subgroup selection concerns \(\Delta_S\), the effect for the subgroup actually selected, while recognizing that the identity of \(S\) is data-dependent.

The approach below assumes a two-stage design. Interim data determine the subgroup to continue, and new patients in that subgroup contribute later-stage data. The primary interval uses those later-stage data. This is not the only possible inferential approach, but it gives a particularly simple validity argument when its independence conditions hold.

Model and Conditions for a Valid Later-Stage Interval

Suppose the outcome is continuous. For each subgroup \(g\), let the later-stage treatment and control sample means be \(\bar{x}_{T,g}^{(2)}\) and \(\bar{x}_{C,g}^{(2)}\), based on \(n_{T,g}^{(2)}\) and \(n_{C,g}^{(2)}\) patients. For the selected subgroup \(S\), estimate its effect by

\hat{\Delta}_{S}^{(2)} = \bar{x}_{T,S}^{(2)}-\bar{x}_{C,S}^{(2)}.

For a simple exact calculation, assume outcomes in each treatment group are normally distributed with a common, known standard deviation \(\sigma\). Treatment is randomized within the selected subgroup, patients contribute independent outcomes, and there is no interference between patients. The later-stage sample sizes are fixed by design or otherwise handled in the analysis. Most importantly, the later-stage outcomes and allocation are independent of the interim data used to choose \(S\), conditional on the design and selected subgroup.

Under these assumptions, for a fixed subgroup \(g\),

\hat{\Delta}_{g}^{(2)} \sim N\left(\Delta_g,\ \sigma^2\left(\frac{1}{n_{T,g}^{(2)}}+\frac{1}{n_{C,g}^{(2)}}\right)\right).

The standard error is therefore

SE_g = \sigma\sqrt{\frac{1}{n_{T,g}^{(2)}}+\frac{1}{n_{C,g}^{(2)}}}.

A two-sided confidence interval with nominal confidence level \(1-\alpha\) is \(\hat{\Delta}_{S}^{(2)}\pm z_{1-\alpha/2}SE_S\), where \(z_{1-\alpha/2}\) is the corresponding standard normal quantile. The outcome units of the interval are the same as the units of the outcome.

Why Conditioning on the Selection Does Not Break This Interval

The central justification is independence. Let \(A_g\) be the event that the interim rule selects subgroup \(g\). Since the selection rule uses interim information, \(A_g\) is determined before the later-stage outcomes are observed. If later-stage data are independent of the interim information, then, for every subgroup that can be selected, the later-stage estimator retains its sampling distribution after conditioning on selection.

P_{\Delta_g}\left( \frac{\hat{\Delta}_{g}^{(2)}-\Delta_g}{SE_g} \leq z \ \middle|\ A_g \right) = P_{\Delta_g}\left( \frac{\hat{\Delta}_{g}^{(2)}-\Delta_g}{SE_g} \leq z \right) = \Phi(z).

Consequently, the interval has conditional coverage \(1-\alpha\) for each subgroup \(g\) when it is selected:

P\left( \Delta_g\in \left[ \hat{\Delta}_{g}^{(2)}-z_{1-\alpha/2}SE_g,\ \hat{\Delta}_{g}^{(2)}+z_{1-\alpha/2}SE_g \right] \ \middle|\ S=g \right) = 1-\alpha.

This is a conditional statement: among repetitions in which subgroup \(g\) is selected, the interval procedure covers \(\Delta_g\) at its nominal rate, under the model and design assumptions. It is not a statement that the selected subgroup is certainly the biologically best subgroup, nor that the fixed but unknown effect has a probability of \(1-\alpha\) of lying in the observed interval.

Worked Example: A Conditional Interval for the Selected Subgroup

Suppose an interim analysis selects subgroup A from three prespecified subgroups. In a hypothetical independent later stage, 36 patients receive treatment and 36 receive control in subgroup A. The treatment-group mean is 7.2 points, the control-group mean is 2.4 points, and the known common standard deviation is 12 points. The later-stage allocation and outcomes were not used to select A.

The estimated treatment effect is \(7.2-2.4=4.8\) points. The standard error is

\(SE_A=12\sqrt{1/36+1/36}=12\sqrt{1/18}=2.828\) points.

For a 95% interval, use \(z_{0.975}=1.960\). The margin of error is \(1.960(2.828)=5.544\) points, using the unrounded standard error for the multiplication. Thus the interval is \(4.8\pm5.544\), or approximately \([-0.744,\ 10.344]\) points.

The data are compatible with effects ranging from a small decrease to a substantial increase in the mean outcome. Under the stated model and independent-stage design, the interval has 95% coverage for subgroup A’s treatment effect conditional on A being selected. Its width also shows that selection alone does not make the effect estimate precise.

Selection Uncertainty and Effect Uncertainty Are Different

A confidence interval for \(\Delta_S\) describes uncertainty about the effect in the subgroup that was selected. It does not describe uncertainty about whether that subgroup is truly the most responsive subgroup. These are distinct questions. The first is addressed by the interval; the second concerns the selection rule and the interim data.

For example, a selection probability can be explored under an explicit model for interim estimates. Such a calculation is not automatically a confidence statement about which subgroup has the largest true effect. It depends on the assumed effects, standard errors, and selection rule.

Worked Example: How Unstable Can a Two-Subgroup Selection Be?

Suppose two subgroups have independent interim effect estimates with standard error 2 points each. The observed estimates are 6 points for A and 5 points for B, so the rule selects A. For a simple plug-in calculation, treat the observed values as the means of the interim-estimate distributions. Under this working model, the difference between the two estimates has mean \(6-5=1\) point and standard deviation \(\sqrt{2^2+2^2}=2.828\) points.

The probability that A’s interim estimate exceeds B’s under this plug-in model is

\(P(\hat{\Delta}_{A}^{(1)}>\hat{\Delta}_{B}^{(1)})=\Phi(1/2.828)=\Phi(0.354)=0.638\), approximately.

Thus, even with A ahead in the observed interim results, the ranking is not especially stable under this illustrative model. The value 0.638 is a model-based plug-in probability, not a guarantee that A has the larger true effect. The later-stage interval for A still quantifies uncertainty about \(\Delta_A\); it does not resolve the ranking question by itself.

Why Naively Pooling the Selected Interim Estimate Can Mislead

The interim estimate of a selected subgroup is generally not an ordinary unbiased estimate conditional on selection. When several noisy subgroup estimates compete, the selected estimate tends to be high even if the underlying effects are equal. This is a form of selection bias, sometimes described as a winner’s curse.

Worked Example: Selection Bias When Two Null Effects Compete

Consider two subgroups with the same true effect of 0 points. For illustration, suppose their independent interim estimates, after division by their common standard error, are \(Z_1,Z_2\), where each has distribution \(N(0,1)\). The selected subgroup is the one with the larger interim estimate. The expected selected standardized estimate is \(E[\max(Z_1,Z_2)]\).

Use the identity \(\max(Z_1,Z_2)=(Z_1+Z_2+|Z_1-Z_2|)/2\). Since each \(Z_g\) has mean 0 and \(Z_1-Z_2\sim N(0,2)\),

\(E[\max(Z_1,Z_2)]=\tfrac12 E|Z_1-Z_2|=\tfrac12\sqrt{2}\sqrt{2/\pi}=1/\sqrt{\pi}=0.5642\).

So the selected interim estimate is, on average, 0.564 standard errors above the common true effect in this two-subgroup illustration. If the later-stage estimate is independent and has the same standard error, simply averaging it with the selected interim estimate gives an expected upward bias of \(0.564/2=0.282\) standard errors. Independence of the later-stage data does not remove the bias already present in the selected interim estimate.

This calculation explains why the interval above uses later-stage data alone. It does not imply that all reuse of interim data is invalid; it shows that ordinary pooling and an ordinary standard error are not justified merely because both stages measured the same outcome.

Practical Checks, Alternatives, and Limitations

Before reporting a later-stage interval, verify that the design actually supports the independence argument. Check that subgroup membership was defined before examining outcomes, that the selection rule and selected subgroup were recorded, and that later-stage participants and outcomes were not used in the interim decision. Also verify that treatment allocation in the selected subgroup supports the stated contrast and that the assumed variance model is credible.

The most important failure modes are:

Reusing selection data as if selection had not occurred. An ordinary pooled interval can be too optimistic or centered too high when the interim estimate was selected for being favorable.
Changing the target after seeing results. The subgroup effect must have a clear definition. If subgroup membership or the outcome scale is modified after interim results, the stated estimand and inference may no longer match the analysis.
Ignoring dependence. If later-stage outcomes influence selection, or if patients contribute data to both stages, the conditional independence argument does not apply as written.
Overstating what the interval answers. The interval concerns the treatment effect in the selected subgroup, not the probability that this subgroup is truly the best one or the effect in the full population.

There are two broad alternatives. One is to use only independent later-stage data, as developed here: the conditional coverage argument is simple, but precision may be lower because interim observations are not included. Another is to incorporate interim data through a method that explicitly accounts for the selection rule, such as selection-adjusted test inversion or a model-based estimator with appropriate adjustment. Those methods can use more information, but their validity depends on correctly specifying the selection mechanism and the relevant joint sampling model. A naive Wald interval that pools all observations without such adjustment is not a valid substitute.

The normal model with known \(\sigma\) makes the calculations transparent. In an actual trial, variance may be estimated, outcomes may be nonnormal, or allocation may be more complex. The standard error and interval procedure must then reflect those features; the independence principle remains useful, but the exact normal calculation above may not. The later-stage-only approach also does not eliminate concerns about multiplicity in broader claims across many subgroups or outcomes. It supports the selected-subgroup effect under its stated design and estimand, rather than every possible confirmatory claim.

Key takeaway: When subgroup selection is based only on interim data and the later-stage data are independent, the later-stage treatment-effect interval can retain its nominal coverage conditional on selection. The selected interim estimate, however, is generally biased upward if treated as though it had not been selected.

Check Your Understanding

Work through the questions, then compare your reasoning with the solutions.

A selected subgroup has later-stage treatment and control means of 5.0 and 1.0 points, with 25 patients per group and known \(\sigma=10\) points. Find the standard error and 95% interval.
Why can the later-stage interval have nominal coverage conditional on selecting subgroup \(g\), even when the interim estimate in that subgroup was selected for being favorable?
What does a 95% interval for the selected subgroup’s treatment effect not tell you about subgroup selection?
Name one condition whose failure would invalidate the simple conditional-coverage argument in this tutorial.

Solutions. (1) The estimate is \(5.0-1.0=4.0\) points. \(SE=10\sqrt{1/25+1/25}=10\sqrt{0.08}=2.828\) points. The margin is \(1.960(2.828)=5.544\) points, so the interval is approximately \([-1.544,\ 9.544]\) points. (2) Selection is determined from interim data, while the later-stage estimator is independent of those data; conditioning on the selection event therefore does not change the later-stage estimator’s sampling distribution. (3) It does not give the probability that the selected subgroup has the largest true effect, or that it is the biologically best subgroup. (4) For example, the argument fails as stated if later-stage outcomes are used to choose the subgroup, or if the same patients’ outcomes contribute to both the interim decision and the later-stage estimator without accounting for that dependence.