Reporting Treatment Estimates After Adaptive Selection

Why Reporting Must Include the Selection Process

An estimate reported after adaptive arm selection is not fully interpretable on its own. Readers need to know which treatment was selected, how the selection rule used the accumulating data, what treatment effect the estimate targets, and how both the point estimate and its uncertainty account for selection. Otherwise, a favorable estimate can be mistaken for an ordinary estimate from a trial in which the treatment was specified in advance.

For example, a trial may select the active arm with the largest interim benefit and then report that arm’s final estimate. Even if every arm has the same true effect, selecting the largest interim estimate tends to select a positive random error as well as any genuine benefit. The issue is not resolved simply by reporting a standard error or a confidence interval: those quantities must also correspond to an analysis that accounts for the adaptive selection.

For notation, let \(\theta_k\) denote the treatment effect for active arm \(k\) relative to the common control, with the effect scale and direction stated in context. Let \(J\) be the selected arm, determined by a prespecified rule \(g\) applied to interim and, if relevant, later trial data. The reported contrast is then for arm \(J\), not an arm that can be treated as if it had been named before the data were observed.

Conditional and Marginal Interpretations

A conditional analysis describes uncertainty given a specified selection event or path—for example, given that arm B was retained at the interim analysis and selected at the end. A marginal analysis describes uncertainty across repeated trials in which the adaptive rule is applied and the selected arm or path may differ. These are distinct inferential targets. A conditional interval should not be presented as if it covered across all possible trial paths, and a marginal interval should not be described as conditional on the particular path observed.

The phrase “conditional on selection” needs enough detail to identify the conditioning event. It might mean conditioning only on the final arm being selected, or on the entire sequence of interim decisions. Those events need not lead to the same distribution. A report should name the event actually used in the adjustment and interval calibration.

The underlying treatment effect may be a fixed arm-specific contrast, such as \(\theta_B\). The fact that the arm index \(J\) is random does not make \(\theta_B\) random; rather, which fixed parameter is reported can vary across repeated trials. This distinction helps prevent ambiguity between estimating the effect of the arm selected in this trial and describing average performance of the selection strategy over repeated trials.

Why Naive Selected Estimates Tend to Be Optimistic

A short calculation illustrates the source of selection bias. Suppose two arms have the same true effect \(\theta\), and their interim estimates are \(X_1=\theta+\epsilon_1\) and \(X_2=\theta+\epsilon_2\), where the errors are independent and each has distribution \(N(0,\sigma^2)\). If the arm with the larger estimate is selected, its interim estimate is \(\theta+\max(\epsilon_1,\epsilon_2)\).

The maximum can be written as \((\epsilon_1+\epsilon_2+|\epsilon_1-\epsilon_2|)/2\). The first two errors have mean zero. Since \(\epsilon_1-\epsilon_2\sim N(0,2\sigma^2)\), its expected absolute value is \(2\sigma/\sqrt{\pi}\). Therefore:

Thus, even when the true effects are equal, the selected interim estimate exceeds the common effect on average. Real MAMS designs can involve multiple stages, shared controls, different allocation ratios, and more complex rules; this simple calculation is an illustration, not a correction formula for those settings. It explains why a report should say how the actual selection process was handled.

What a Complete Report Needs to Say

A concise report can still be technically complete. It should give enough information for a reader to identify the estimand, reconstruct the decision context, and understand the source and meaning of uncertainty. In particular, report:

• Selection rule: which interim or final data were used, the decision criterion, and any arm-dropping or stopping decisions relevant to the reported arm.
• Observed selection: the arm selected and, where useful, the realized path—for example, which arms were dropped at interim.
• Effect scale and direction: such as a treatment-minus-control mean difference in points or a treatment-to-control risk ratio. State which direction favors treatment.
• Point estimate and adjustment: give the unadjusted estimate if it helps, but clearly distinguish it from the reported selection-adjusted estimate. Name the adjustment method and the main model or resampling assumptions.
• Uncertainty interval: state the confidence level, method, and whether its coverage is conditional on a stated selection event or marginal over paths.
• Diagnostics and sensitivity: summarize checks that bear on the selection rule, outcome model, or adjustment method and indicate whether conclusions materially change under reasonable alternatives.

A numerical interval alone does not communicate its calibration. For example, “95% interval” is incomplete unless it is clear whether the interval is an ordinary unadjusted interval, a selection-adjusted conditional interval, or a marginal interval. The usual repeated-sampling interpretation applies only under the assumptions and calibration method stated for that interval; it is not a probability statement about the fixed treatment effect after the data have been observed.

Keep the Estimate and Interval on the Same Target

Adjustment and interval construction are related but separate tasks. A correction to a point estimate does not, by itself, provide a valid standard error or confidence interval. Conversely, a selection-aware interval must be centered and interpreted in relation to the estimand for which its calibration was established. If a conditional point estimate is paired with a marginal interval, or an adjusted estimate with an ordinary unadjusted interval, the displayed pair may not support the intended conclusion.

The choice between conditional and marginal inference should follow the scientific question, not which interval is narrower. A conditional analysis can focus on the specific selected arm and observed path, but it depends on the conditioning event and may be less stable when that event is rare. Marginal inference reflects the full adaptive strategy and variation across paths, but may be less directly about the exact path seen in this trial. Neither description is universally preferable.

Reporting on a Ratio Scale

For ratio measures, it is often useful to perform estimation and interval construction on the log scale, then transform both the estimate and interval limits back to the ratio scale. The report should state the event, numerator, denominator, and direction of benefit. A ratio of 1 represents equal risks; whether values below or above 1 favor treatment depends on the event being analyzed.

Diagnostics, Alternatives, and Limitations

A well-written result does not replace diagnostics. The report should indicate whether the implemented rule matches the protocol and whether the adjustment method was evaluated under conditions resembling the design. Useful checks include comparing the programmed and actual selection pathways, examining model fit and shared-control dependence, assessing the frequency of the observed path in simulation, and evaluating coverage separately for conditional and marginal targets when both are relevant. Large changes under plausible selection-rule or outcome-model alternatives should be disclosed.

The main alternatives are not interchangeable labels. An unadjusted analysis is simple and may describe the observed data, but after data-driven selection its estimate and conventional interval may not have their usual repeated-sampling properties for the selected target. Selection-aware bootstrap or model-based methods can incorporate the adaptive rule, but their conclusions depend on how well the resampling or model represents the design and outcome distribution. Randomization-based approaches rely on the randomization structure and define inference through that design. The report should name the method actually used rather than saying only “corrected for selection.”

Adjustment cannot repair an inaccurately recorded selection rule, unmodeled outcome features, or an unclear estimand. Conditional intervals may be difficult to calibrate for rare paths; marginal results can obscure what is learned about the particular arm selected. Estimates can also be sensitive to the outcome model or to the precise decision boundary. These are reasons to report sensitivity and diagnostics, not reasons to omit the primary estimate.

A Common Reporting Mistake

• Wrong reasoning: “The selected arm’s final estimate is 4.3 points, and its ordinary 95% interval is 1.8 to 6.7, so selection is already handled because the estimate uses final-stage data.”
• Correction: Using later data does not automatically remove the consequences of selecting an arm with earlier data. State whether the final analysis accounts for the full selection process, identify the method and its assumptions, and label the interval’s conditional or marginal coverage target. If only an ordinary interval was calculated, report it as such rather than implying selection-aware inference.