Simultaneous Inference Across Adaptation Paths

Why Inference Must Cover the Paths You Could Have Taken

An adaptive trial can stop early, continue with a revised sample size, or select a treatment or population for further study. The final estimate therefore depends on the path taken through the design. If investigators report the estimate from the path that looks most favorable, an interval calibrated as though that path had been specified in advance may miss its target too often. The relevant protection is not merely coverage for one prespecified route: it is coverage across the possible routes from which the reported result could have been selected.

Here, a path means a terminal history of design decisions that can lead to a reported estimate. For example, “stop at the interim analysis,” “continue and select treatment A,” and “continue and select treatment B” may be three terminal paths. Count paths that can produce a report, not every intermediate decision node. If different paths concern different treatment effects or populations, each path must have its own clearly defined estimand.

The probability statement includes the randomness in outcomes and in the adaptation decisions. It is an unconditional joint guarantee over the trial procedure, not a claim that coverage is guaranteed separately within each realized path. The parameters \(\theta_r\) may all be the same, or may differ when the path determines the population or treatment effect being estimated.

Bonferroni Protection Across Paths

A direct construction allocates a total error budget across the paths. Suppose each candidate interval \(I_r\) is defined on the trial’s probability space and satisfies \(P(\theta_r\notin I_r)\leq\alpha_r\), with \(\sum_{r\in\mathcal{R}}\alpha_r\leq\alpha\). The union bound then gives simultaneous coverage. This argument does not require the pathwise estimates to be independent.

Taking complements shows that all intervals cover with probability at least \(1-\alpha\). Since \(S\) must be one of the paths in \(\mathcal{R}\), \(P(\theta_S\in I_S)\geq P(\theta_r\in I_r\text{ for every }r)\geq1-\alpha\). The chosen interval is protected even if the selection rule favors the largest observed effect, the smallest p-value, or another favorable statistic.

For equal allocation across \(K\) paths, set \(\alpha_r=\alpha/K\). If the standardized error for every path has a standard normal distribution, a two-sided interval is

where \(z_q\) is the \(q\) quantile of the standard normal distribution. This construction requires the stated normal error model and valid standard errors for the pathwise estimates. Other valid pathwise pivotal distributions can be used, but the critical values must match those distributions.

Using the Joint Distribution of Pathwise Errors

Bonferroni protection can be conservative, particularly when estimates from different paths are strongly correlated. If the joint distribution of the standardized errors is known under the assumed design and analysis model, a maximum-statistic calibration can use that dependence directly. Define \(W_r=(\hat{\theta}_r-\theta_r)/SE_r\) and \(M=\max_{r\in\mathcal{R}}|W_r|\). Choose \(c\) so that \(P(M\leq c)\geq1-\alpha\). Then the intervals \(\hat{\theta}_r\pm cSE_r\) have simultaneous coverage of at least \(1-\alpha\).

The justification follows from the definition of \(M\): if \(M\leq c\), then \(|W_r|\leq c\) for every path. That inequality is equivalent to \(\theta_r\in[\hat{\theta}_r-cSE_r,\hat{\theta}_r+cSE_r]\) for every \(r\). Thus the event used to calibrate \(c\) is exactly the joint coverage event for these intervals. The maximum distribution must reflect the dependence and the full path structure; a guessed correlation or a simulation that omits adaptation decisions does not provide this calibration.

Defining the Family and Checking the Model

The coverage argument is only as sound as the family and the pathwise intervals supplied to it. Before calculating critical values, specify the terminal paths the protocol can produce, the estimand for each reported result, and the rule that selects the final path. Include paths that can lead to a reported result even if they seem unlikely. When several paths share a target parameter, state that explicitly; when their target populations or treatments differ, do not describe their intervals as covering one common effect.

A candidate interval must also be valid for the design that generated it. A fixed-sample normal interval does not automatically have the required marginal coverage when its estimate, sample size, or stopping time is affected by interim data. The relevant model must justify each pathwise error distribution while accounting for the adaptation procedure. If estimates for unselected paths are not directly observed, their role in a simultaneous calibration must still be defined through a valid model for the candidate pathwise statistics. A collection of nominal intervals computed only for the observed path is not, by itself, a simultaneous procedure.

When the joint distribution is available and credible, maximum-statistic calibration can avoid some conservatism by reflecting dependence. When it is unavailable or difficult to verify, Bonferroni is transparent and needs only valid marginal bounds. Both approaches require the correct path family and a defensible model. A path-conditional selective method is another possible approach, but it requires a distribution that accounts for the selection event; it is not obtained merely by applying an ordinary interval after selection.

Diagnostics and Limitations

Several checks help identify a procedure that is not protecting the intended report:

• Audit the path list. Compare the analysis family with every stopping, continuation, and selection outcome that can lead to a reported estimate. An omitted route is not protected by a guarantee over the listed routes.
• Check the target and scale path by path. Confirm that each interval targets the stated \(\theta_r\) and that its \(SE_r\) and error distribution remain valid under the relevant adaptation.
• Verify the calibration. For a maximum-statistic method, check that the joint model includes the dependence among candidate errors and the actual adaptation rule. For Bonferroni, verify each marginal error bound and that the allocated error probabilities sum to no more than \(\alpha\).
• Report what the guarantee covers. State the family of paths and whether the stated coverage is simultaneous and unconditional. Do not present it as a guarantee conditional on the observed path unless that was separately established.

Simultaneous protection generally costs precision: the intervals must allow for the possibility that any member of the family could have been selected. A large or poorly specified path family may therefore yield uninformative intervals. This method also cannot repair a biased estimator, an incorrectly defined estimand, or an invalid variance model. It controls selection across the specified paths only under the assumptions that justify the pathwise intervals and their calibration.