Reporting Point Estimates and Intervals After Adaptation

Why Reporting Must Include the Adaptation

After an adaptive trial, the final estimate and interval are products of both the observed outcomes and the decisions made during the trial. For example, interim results may determine which treatment continues, how many participants are enrolled, or when follow-up ends. Reporting only the final sample size and a confidence interval can conceal the process that generated them. Readers need enough information to understand what effect was estimated and what repeated-trial guarantee, if any, the interval is intended to provide.

The central reporting task is to connect five items: the adaptation rule, the estimand, the point estimator, the interval construction, and the coverage target. These are related but not interchangeable. In particular, an adjusted point estimate does not by itself establish that an interval has correct coverage, and a nominal 95% interval does not say whether its guarantee applies conditionally on the observed adaptation path.

Definition: Let \(A\) denote the adaptation path actually taken, including the interim information used and the resulting design decisions. Let \(\theta\) denote the stated estimand. An adaptation-aware report specifies the rule producing \(A\), the estimator \(\hat{\theta}_{\mathrm{adj}}\) used for \(\theta\), the interval procedure \(C(D,A)\) based on the trial data \(D\) and the adaptation path, and the probability statement used to describe its coverage.

What a Complete Report Specifies

The adaptation rule should be described as a reproducible mapping from interim data to decisions. Report when the interim analysis occurred, which data were available, which quantities were reviewed, the decision thresholds or algorithm, and what actions could follow. Then state the realized path: for example, which dose was selected, whether the sample size changed, and what final sample size resulted. If the rule was modified, disclose the modification and when it occurred. “The study was adapted based on interim results” is not precise enough to reconstruct the design.

Define the estimand in clinical and statistical terms, rather than treating the outcome of the adaptation as the estimand. State the target population, treatment conditions being compared, outcome variable, time point, and summary measure. If treatment selection makes the target effect indexed by the selected treatment, write that explicitly: the target is then \(\theta_J\), where \(J\) is the treatment selected by the rule. If the target is instead a prespecified treatment contrast regardless of the path, say so. These choices affect interpretation.

Next identify the estimator. Give the raw estimate, if useful, and the adjusted estimate that supports the reported inference. Explain what “adjusted” means in this analysis: for example, a specified correction for selection bias, or an estimator calculated using a procedure calibrated for the complete adaptation. State the model and assumptions behind that procedure, and do not imply that all adaptations require the same correction. An adjustment should be defined by its method, not merely by its label.

For the interval, report its level, construction, and whether its calculation used the adaptation process. A reader should be able to tell whether it is a conventional interval applied after the trial or an interval calibrated using the full design. Include the endpoints in the units of the outcome and effect measure. If the method relies on a model, disclose the relevant model assumptions and the diagnostics or sensitivity analyses used to assess them.

Worked Example: Reporting an Estimate After Treatment Selection

Hypothetical trial. An interim rule compares two experimental doses using interim response data and continues the dose with the larger interim response rate. Dose B is selected. The target is the response-rate difference, in percentage points, between the selected dose and control in the trial’s target population at the prespecified assessment time. Thus the estimand is \(\theta_J\), with \(J\) the dose selected by the rule.

Suppose the final unadjusted difference for Dose B is 7.2 percentage points. A prespecified, model-based calibration estimates that the selection procedure contributes 1.3 percentage points of upward bias in this setting. The adjusted point estimate is:

\hat{\theta}_{\mathrm{adj}} = 7.2 - 1.3 = 5.9 \text{ percentage points}.

Suppose the specified selection-aware interval procedure gives \(SE=2.4\) percentage points and uses a calibrated multiplier of 2.05. Its interval is:

5.9 \pm 2.05(2.4) = 5.9 \pm 4.92 = (0.98,\ 10.82) \text{ percentage points}.

A report should say that the interval procedure was calibrated for the treatment-selection rule, and state what model and calibration were used. If its stated target is 95% unconditional coverage for the selected effect, explain that this means coverage across repeated trials in which the interim data, selected dose, and final data are all generated under the design. It is not automatically a 95% conditional guarantee for the particular path in which Dose B was selected.

Coverage Is Part of the Result

A confidence interval’s coverage statement describes repeated sampling under a specified design and assumptions. With adaptation, the phrase “95% confidence interval” is incomplete unless the probability statement is clear. Unconditional coverage averages over possible interim data and adaptation paths. Conditional coverage instead fixes a path, or conditions on specified interim information, and evaluates coverage within that restriction. One guarantee does not imply the other.

The distinction follows from the law of total probability. If \(A\) indexes the possible adaptation paths, then:

P_{\theta}\{\theta \in C(D,A)\} = \sum_a P_{\theta}\{\theta \in C(D,A)\mid A=a\}P_{\theta}(A=a).

Thus unconditional coverage is a weighted average of path-specific conditional coverages. The average can equal 0.95 even when some paths have coverage below 0.95 and others above it. Conversely, a procedure with 95% coverage conditional on every path also has 95% unconditional coverage, provided the conditioning events and probabilities are well-defined. When the adaptation path is continuous, the sum is replaced by an integral.

Reporting rule: State the intended guarantee in probability notation or equivalent words. For unconditional coverage, report \(P_{\theta}\{\theta \in C(D,A)\}\) and identify that the probability averages over the complete adaptive design. For conditional coverage, state what is conditioned on, such as the selected treatment or interim data, and do not claim a broader guarantee than the method establishes.

Worked Example: A Conditional Guarantee Is Not an Unconditional Description

Hypothetical trial. A prespecified procedure has 96% coverage on trials that select Treatment A and 92% coverage on trials that select Treatment B. Assume the selection probabilities under the stated model are 0.60 and 0.40. Its unconditional coverage is:

$$ 0.60(0.96)+0.40(0.92)=0.576+0.368=0.944. $$

The overall coverage is 94.4%, not 95%, even though the procedure has coverage near 95% on each path. If the design’s target was at least 95% unconditional coverage, these results do not meet that target. If the intended claim were instead a path-specific guarantee, the two conditional coverages would need to be reported separately. Rounding 94.4% to “about 95%” would not change the underlying guarantee.

Justifying the Reported Guarantee

The reporting claim must follow from the procedure’s justification. If a method is justified analytically under a model, name that model and the assumptions needed for the result. If calibration is based on simulation, describe the simulated design: the adaptation rule, outcome model, parameter settings, analysis procedure, and number of repetitions. Report whether calibration targeted unconditional or conditional coverage. Results under a limited set of simulated scenarios do not establish coverage under every possible data-generating process.

A useful check is to compare the rule used in the actual trial with the rule used to justify the interval. If the interim threshold, timing, endpoint, or decision options changed, the calibration may no longer apply. Similarly, an interval calibrated for one estimand cannot be casually presented as covering a different estimand. The conclusion is only as strong as the match among design, estimand, estimator, interval, and assumptions.

Worked Example: Reporting an Estimate After Sample-Size Adaptation

Hypothetical trial. The estimand is the mean difference in a continuous outcome between Treatment and control at 12 weeks in the target population. After an interim review, the sample size is increased under a prespecified rule. The final, unadjusted mean difference is 3.6 outcome units. A design-specific calibration estimates a 0.2-unit bias for the estimator under the stated model, giving:

\hat{\theta}_{\mathrm{adj}} = 3.6 - 0.2 = 3.4 \text{ outcome units}.

Suppose the interval procedure calibrated for the sample-size rule reports \(SE=1.5\) units and multiplier 2.00. The reported interval is:

3.4 \pm 2.00(1.5) = 3.4 \pm 3.0 = (0.4,\ 6.4) \text{ outcome units}.

A complete report would give the interim rule, initial and final sample sizes, calibration method, and the coverage target. For example, if calibration established only 95% unconditional coverage under a normal-outcome model with the specified variance structure, the report should state those qualifications. It should not describe the interval as having 95% conditional coverage for the realized sample-size decision unless that stronger result was established.

Diagnostics, Alternatives, and Limitations

Diagnostics for post-adaptation reporting are largely checks for mismatches and unsupported claims. Confirm that the recorded adaptation path matches the protocol and analysis code; that the analysis used the correct interim information and decision thresholds; and that the estimand, estimator, and interval refer to the same population, outcome, time point, and treatment contrast. For simulation-calibrated methods, examine coverage across plausible parameter values and adaptation paths, not only at a single convenient scenario. Report instability or poor performance rather than hiding it behind a nominal confidence level.

The main alternatives answer different questions. A conventional interval that ignores adaptation may be simple to calculate, but its nominal level need not be valid after selection or data-dependent stopping. A selection-adjusted or otherwise design-calibrated interval accounts for the specified adaptation, but depends on assumptions and on faithfully representing the rule. A Bayesian credible interval summarizes posterior uncertainty under a prior and likelihood; its posterior probability statement is not the same as a frequentist coverage guarantee. These approaches should not be described as interchangeable versions of “the 95% interval.”

Approach	What to report	Main qualification
Conventional post-trial interval	Estimator, standard error or interval formula, and model	Nominal coverage may not account for adaptation
Adaptation-aware calibrated interval	Full rule, calibration procedure, estimand, and coverage target	Validity depends on assumptions and design matching
Bayesian credible interval	Likelihood, prior, posterior summary, and target parameter	Posterior probability is not a frequentist coverage statement

No single reporting format can compensate for an unclear estimand, incomplete records of interim decisions, or a method that does not match the design actually run. An adjusted estimate may also be less familiar than the raw estimate. Reporting both, with the adjustment defined and the adjusted value identified as the primary inferential estimate, helps readers see what changed without conflating the two.

Common Mistakes

Reporting only the final sample size. That does not reveal which data or rule produced it. State the decision rule and the realized path.
Calling an estimate “bias-adjusted” without defining the adjustment. Give the correction or procedure and the assumptions used to obtain it.
Writing “95% coverage” without naming the probability target. Specify whether coverage is unconditional or conditional, and what the conditioning event is.
Claiming the interval covers the selected effect without accounting for selection. Explain how selection enters the interval construction and which selected-effect guarantee was established.
Describing a Bayesian credible interval as a frequentist confidence interval. State the inferential framework and interpret its probability statement correctly.

The practical standard is straightforward: a reader should be able to reconstruct the adaptation, identify the precise effect being estimated, distinguish the raw and adjusted estimates, and understand what repeated-trial claim the interval supports. If any one of those links is missing, the numerical result is harder to interpret and its uncertainty may be overstated.

Check Your Understanding

Answer each question, then compare with the short solutions below.

A trial reports a 95% interval after selecting the larger of two treatment effects at interim. Name two details needed to interpret the coverage claim.
An interval has 95% unconditional coverage but 90% coverage on one adaptation path. Is that necessarily a contradiction? Explain.
A final estimate is 4.8 units and the prespecified estimated bias correction is 0.6 units upward. What adjusted estimate should be reported?
A report gives an adjusted estimate but does not state the target population or outcome time point. What essential reporting element is missing?

Solutions. (1) State the selection rule and whether coverage is unconditional or conditional on a path; also identify the selected-effect estimand and the method accounting for selection. (2) No. Unconditional coverage is the probability-weighted average over paths, so one path can have lower coverage while the overall average is 95%. (3) \(4.8-0.6=4.2\) units. (4) The estimand is incompletely defined; the population and outcome time point are needed to determine which effect the estimate represents.