Sensitivity Analysis for Unrecorded Adaptation

Why Unrecorded Adaptation Requires Sensitivity Analysis

An adaptive trial’s final data may not reveal exactly what happened at an interim review. For example, the review may have changed which stage’s data were retained, which population was targeted, or which subgroup was carried forward. Those choices can affect both the treatment-effect estimate and the coverage of its interval. If the decision history cannot be reconstructed with confidence, a single analysis can give a misleading impression of certainty.

Sensitivity analysis addresses this uncertainty by specifying a set of plausible decision histories and recalculating the relevant estimates and intervals under each one. It does not identify which history actually occurred. Its purpose is to show how strongly the conclusions depend on choices that remain undocumented, and to make the assumptions behind each result visible.

The comparison is meaningful only when each scenario is explicit about its assumptions. These include the target treatment effect, which observations contribute, whether the stage-specific estimates are independent, the variance model, and any rule that selected a population or determined whether the trial continued. A scenario that changes the target population is not merely a different calculation of the same estimand; it may answer a different scientific question.

Recalculating Estimates Across Histories

Consider a common-effect model in which two stage-specific estimates \(Y_1\) and \(Y_2\) are independent and normally distributed around the same treatment effect \(\theta\), with known standard errors \(s_1\) and \(s_2\). Under this model, the inverse-variance weighted estimate uses more weight for the more precise stage. The assumptions of independence and a common target effect must be credible; if the stages overlap, or estimate different population effects, this calculation is not justified as written.

The variance expression follows because the estimator is a weighted sum of independent observations. If \(w_i=1/s_i^2\), then its variance is \[ \operatorname{Var}\left(\frac{w_1Y_1+w_2Y_2}{w_1+w_2}\right) =\frac{w_1^2s_1^2+w_2^2s_2^2}{(w_1+w_2)^2} =\frac{w_1+w_2}{(w_1+w_2)^2} =\frac{1}{w_1+w_2}. \] Thus, under the stated normal model, a two-sided 95% interval is the estimate plus or minus \(1.96SE\). This interval’s nominal coverage relies on the estimator’s normal distribution and on using the right analysis for the actual decision history; selection can invalidate the nominal interpretation.

For more than two histories, a useful report lists each scenario’s assumptions, estimate, standard error, and interval in a table. A range across scenarios can summarize the estimate’s sensitivity, but it should not be confused with statistical uncertainty within any one scenario. If the target effect differs across scenarios, report that change prominently rather than presenting the resulting estimates as directly interchangeable.

When the History Changes the Target Population

Interim decisions can also change which population is being studied. Suppose the final analysis might have targeted all eligible patients or, after an undocumented enrichment decision, only a biomarker-positive subgroup. The subgroup treatment effect and the all-patient treatment effect are different estimands in general. A sensitivity analysis should make that distinction explicit. If subgroup-specific estimates are available, one can also reconstruct the all-patient effect using prespecified population weights, provided those weights and the subgroup definitions are appropriate for the target population.

Selection Rules and Coverage Sensitivity

Coverage is a repeated-sampling property: for a fixed parameter value and a specified procedure, it is the probability that the procedure’s interval contains that value over repeated trials. When an interim rule selects which results are reported or analyzed, the coverage of an interval that ignores that selection can differ from its nominal level. If the rule itself is undocumented, coverage sensitivity requires evaluating clearly stated plausible rules, not guessing at a single adjustment.

A simple normal example shows why the selection rule matters. Let \(Z\sim N(0,1)\) under the null value \(\theta=0\), and suppose a reported estimate is \(Z\) standard-error units from zero. A nominal two-sided 95% interval contains zero exactly when \(-1.96\leq Z\leq1.96\). If only positive results exceeding a threshold \(d\) are selected, the coverage conditional on selection is the probability of lying between \(d\) and \(1.96\), divided by the probability of selection, for \(0<d<1.96\).

Here \(\Phi\) is the standard normal cumulative distribution function. This calculation is conditional on a specified selection event. It is not a universal coverage correction: a real trial can use a different statistic, threshold, or analysis after the interim decision.

A Practical Sensitivity Workflow

A disciplined analysis separates what is observed from what must be assumed. The following sequence makes the comparison auditable and helps prevent scenarios from silently changing the scientific question.

Diagnostics begin with the provenance of the data: check interim-review minutes, data snapshots, analysis code, randomization records, and the version history of the protocol and statistical analysis plan. Then check whether stage estimates are independent, whether the variance model matches the design, and whether subgroup definitions and weights are documented. Large changes when a modestly different history is substituted indicate that the conclusion depends materially on the missing information.

If plausible histories imply different selection rules, coverage should be assessed under each rule with its selection event represented correctly. An interval’s nominal level alone is not evidence that it achieves that level after an undocumented choice. Where the selection mechanism cannot be specified or the records are inadequate, the limitation is not repaired by reporting more decimal places.

What Sensitivity Analysis Can and Cannot Establish

Scenario analysis is useful when it exposes which conclusions are robust and which hinge on an unverified decision. It complements, rather than replaces, a justified primary analysis. A method designed to remain valid under a stated misspecification addresses a different question: here the central issue is uncertainty about which adaptation actually occurred. Likewise, multiplicity adjustments or simultaneous intervals may address a known set of possible selections, but they cannot by themselves establish which undocumented path governed a particular trial.

Do not average estimates across histories unless a defensible probability model for those histories has been specified. Without such a model, the scenario range is not a posterior distribution, a confidence interval, or a formal bound on the treatment effect. Also avoid labeling an interval “robust” solely because it is wide: its coverage still depends on the model and decision histories used to construct it.

The best remedy for substantial uncertainty is often better documentation: preserve the interim decision, data cut, analysis version, and rationale. When that is no longer possible, sensitivity analysis gives a transparent account of what can be concluded under plausible histories—and, just as importantly, what cannot be resolved from the available evidence.