Unconditional Intervals Averaged Over Adaptation

Why average over adaptation paths?

An adaptive trial may follow different paths: it may select one treatment, continue with a revised design, or stop according to a prespecified rule. The path is part of the experiment, not an incidental label attached after analysis. Because selection and stopping rules can depend on interim outcomes, the distribution of the reported estimate is generally affected by the rule that produced it.

A confidence interval calibrated separately within each path can provide a conditional coverage guarantee, but may be wider than necessary if the scientific target is coverage over all repetitions of the trial. An unconditional interval instead assesses coverage across repetitions in which different paths occur, weighting each path by its probability under the parameter value being considered. This is a marginal guarantee for the full design; it does not promise the nominal coverage within every path.

Definition: Let $A$ denote the adaptation path, taking values in a set of mutually exclusive, exhaustive paths $\mathcal{A}$. A confidence procedure $C(D)$, based on all data $D$ observed under the design, has unconditional coverage at least $1-\alpha$ for a parameter $\theta$ if $$ P_\theta\{\theta\in C(D)\}\geq 1-\alpha. $$ Its coverage decomposes over paths as $$ P_\theta\{\theta\in C(D)\} =\sum_{a\in\mathcal{A}} P_\theta(A=a)\, P_\theta\{\theta\in C(D)\mid A=a\}. $$

The probabilities in this decomposition refer to repeated execution of the entire design at the same parameter value. If a path is unlikely, its conditional coverage contributes less to the average; if it is common, its contribution is larger. When path probabilities depend on $\theta$, the weights must be evaluated at the parameter value being tested, rather than treated as fixed.

Calibrating with the full design

A general construction starts with a test of each candidate value $\theta_0$. Specify a test statistic $T(D;\theta_0)$, with larger values representing stronger evidence against $H_0:\theta=\theta_0$. Under the null, its unconditional tail probability is

p_{\mathrm{uncond}}(\theta_0;d) =\sum_{a\in\mathcal{A}}P_{\theta_0}(A=a) P_{\theta_0}\{T(D^*;\theta_0)\geq T(d;\theta_0)\mid A=a\},

where $d$ is the observed data and $D^*$ is data from a repetition of the complete trial under $\theta_0$. The conditional tail probability in each term must reflect the rule that leads to path $a$; it is not the tail probability from an unadapted experiment. The sum is simply the law of total probability applied to the full design.

Reject $H_0$ when $p_{\mathrm{uncond}}(\theta_0;d)\leq\alpha$. Inverting this family of tests gives the confidence set

C(d)=\{\theta_0:p_{\mathrm{uncond}}(\theta_0;d)>\alpha\}.

If each test has size at most $\alpha$ under its null, then the inverted set has coverage at least $1-\alpha$. Exact continuous tests may attain size $\alpha$; discrete tests often have size below $\alpha$, making coverage conservative. In practice, the set need not be a simple interval: if the test’s acceptance set is disconnected, the inverted confidence set may also be disconnected. Report its actual shape rather than silently replacing it with an interval that lacks the same guarantee.

Why test inversion controls coverage

The justification is a direct connection between exclusion of the true value and rejection of its null test. For a fixed true value $\theta$, the confidence set fails to cover $\theta$ exactly when the test of $H_0:\theta=\theta$ rejects.

Use the full-trial null distribution.
Under $H_0:\theta=\theta_0$, generate or derive the distribution of $T$ by applying the prespecified adaptation rule to the interim and later data.

Control the test size.
Choose the rejection rule so that $P_{\theta_0}\{p_{\mathrm{uncond}}(\theta_0;D)\leq\alpha\}\leq\alpha$. For an exact continuous p-value this probability is $\alpha$; conservative calibration gives a value no larger than $\alpha$.

Invert the tests.
By definition, $\theta\notin C(D)$ precisely when the test of $H_0:\theta=\theta$ rejects. Therefore $P_\theta\{\theta\notin C(D)\}\leq\alpha$, and hence $P_\theta\{\theta\in C(D)\}\geq1-\alpha$.

This argument is unconditional: it controls the chance of exclusion over the repeated trial as a whole. It does not establish $P_\theta\{\theta\in C(D)\mid A=a\}\geq1-\alpha$ for every path $a$. That distinction is the central design choice, not a technical detail.

Worked Example: Decomposing coverage across two paths

Suppose a hypothetical adaptive trial has two exhaustive paths: select treatment A and continue (path 1), or select treatment B and continue (path 2). At a particular parameter value, assume the prespecified design has path probabilities 0.30 and 0.70. A candidate interval procedure has conditional coverages 0.90 and 0.96 on those paths, respectively. Calculate its unconditional coverage.

The paths are mutually exclusive and exhaustive, so their probabilities sum to $0.30+0.70=1.00$. Apply the coverage decomposition:

P_\theta\{\theta\in C(D)\} =(0.30)(0.90)+(0.70)(0.96) =0.270+0.672 =0.942.

The procedure has 94.2% unconditional coverage at this parameter value, below a 95% target, even though path 2 has 96% conditional coverage. Path 1’s 90% conditional coverage pulls down the weighted average. To claim a 95% guarantee, a procedure must be calibrated so the weighted coverage is at least 0.95 at every parameter value in the stated model, not merely for one selected value.

Path probabilities and conditional coverage in practice

The decomposition is useful both for construction and for checking a proposed interval. Suppose selection depends on a normally distributed interim statistic $X$, with $X\sim N(\theta,\sigma^2)$, and path 1 occurs when $X>c$. Then

P_\theta(A=1)=1-\Phi\left(\frac{c-\theta}{\sigma}\right), \qquad P_\theta(A=2)=\Phi\left(\frac{c-\theta}{\sigma}\right),

where $\Phi$ is the standard normal cumulative distribution function. These are probabilities under the model before conditioning on the observed path. Conditional outcome distributions within each path are affected by the selection rule: for example, on path 1 the observed interim statistic is restricted to $X>c$. Ignoring that restriction, or multiplying a conditional coverage by a path probability computed under a different parameter value, generally gives the wrong unconditional coverage.

In a real design, path probabilities and conditional tail probabilities may be obtained analytically, by numerical integration, or by simulation. Simulation must reproduce the complete adaptation rule for each replicate, including the same thresholds, selection decisions, stopping decisions, and data-collection plan. For test inversion, repeat this calculation for candidate values $\theta_0$; the weights and null distribution can change with $\theta_0$.

Worked Example: Weighting selection paths under a normal model

Consider a hypothetical interim statistic $X\sim N(\theta,0.4^2)$. The design selects treatment A if $X>0.8$, and treatment B otherwise. Evaluate unconditional coverage at $\theta=0.5$ for a candidate interval whose conditional coverages, obtained under the correct path-specific distributions, are 0.88 on path A and 0.97 on path B.

First calculate the probability of selecting A:

P_{0.5}(A)=P_{0.5}(X>0.8) =1-\Phi\left(\frac{0.8-0.5}{0.4}\right) =1-\Phi(0.75) \approx 1-0.7734 =0.2266.

Thus $P_{0.5}(B)=1-0.2266=0.7734$. Weight the path-specific coverages by these probabilities:

P_{0.5}\{0.5\in C(D)\} =(0.2266)(0.88)+(0.7734)(0.97) =0.1994+0.7502 \approx0.9496.

The resulting coverage is approximately 94.96%, close to 95% at this particular value of $\theta$. This calculation does not establish a 95% guarantee for all $\theta$: the selection probability and possibly both conditional coverages can change as $\theta$ changes. Nor does the average erase the fact that coverage on path A is only 88%.

Worked Example: A common cutoff calibrated to average coverage

For a simplified hypothetical calibration, suppose a signed test statistic $T$, under each tested null value, has conditional distributions $N(0,1)$ on path 1 and $N(0,1.5^2)$ on path 2. Assume the path probabilities are 0.40 and 0.60 under that null. Use a common two-sided cutoff $k$, rejecting when $|T|>k$, and find the cutoff giving 95% unconditional coverage for the inverted procedure.

Under path 1, the conditional probability of retaining the null is $2\Phi(k)-1$. Under path 2 it is $2\Phi(k/1.5)-1$. The unconditional probability of retaining it is therefore

0.40\{2\Phi(k)-1\}+0.60\{2\Phi(k/1.5)-1\}.

Set this expression to 0.95 and solve numerically. At $k=2.645$, $2\Phi(2.645)-1\approx0.9918$ and $2\Phi(2.645/1.5)-1\approx0.9222$, giving

(0.40)(0.9918)+(0.60)(0.9222) =0.3967+0.5533 \approx0.9500.

Thus a common cutoff of approximately 2.645 attains 95% average coverage in this stylized model. It does not give 95% coverage on each path: the conditional coverages are approximately 99.18% and 92.22%. The example illustrates the calculation, not a universal cutoff. In an actual adaptive trial, the null distributions and path probabilities must be derived from that trial’s design, and if weights depend on $\theta_0$, calibration must be repeated for each tested value.

Diagnostics, alternatives, and limitations

A useful diagnostic is to report overall coverage alongside path-specific coverage and path probabilities. A satisfactory average can conceal material undercoverage for a clinically important path, especially if that path is uncommon. Also examine coverage across a grid of plausible parameter values, near adaptation thresholds, and at values where path probabilities change rapidly. Simulation estimates of coverage have Monte Carlo error; report enough simulation detail to distinguish a genuine shortfall from simulation noise, and use conservative calibration when numerical uncertainty could otherwise push coverage below the target.

Approach	Coverage target	Main trade-off
Unconditional calibration	Across repetitions of the complete adaptive design	Can be narrower overall, but need not protect every path separately
Conditional calibration	Within a specified adaptation event	Answers a path-specific question; may require wider intervals and depends on the chosen conditioning event
Naive interval that ignores adaptation	Usually based on an unadapted sampling distribution	May be convenient, but its nominal coverage is not guaranteed after outcome-dependent selection or stopping

The principal alternative is a conditional confidence interval: it calibrates inference given the observed adaptation event and is appropriate when conditional coverage is the required target. Unconditional calibration instead averages over paths. Neither target dominates in every scientific setting; the choice should reflect whether the trial-level repeated-sampling guarantee or a guarantee after learning the realized path is required.

Unconditional guarantees rely on a correctly specified model and a correctly implemented adaptation rule. Misspecified outcome distributions, unrecorded deviations from the rule, nuisance-parameter estimation, or inaccurate numerical integration can invalidate nominal calibration. Coverage is also a repeated-sampling property, not the probability that a fixed parameter lies in the particular interval already observed. Finally, average coverage does not guarantee good precision or pathwise coverage, and it does not by itself address simultaneous inference for multiple parameters.

Common mistake

Wrong reasoning: “The interval has 95% unconditional coverage, so it must cover the true value 95% of the time among trials that selected treatment A.”
Correction: Unconditional coverage is the weighted sum over all paths. A path can have coverage below 95% while the overall average is 95% or higher. If coverage after selection is the target, calibrate conditionally on that selection event.
Wrong reasoning: “The path probabilities are just the observed proportions of paths in this trial.”
Correction: Calibration uses probabilities under the assumed model and candidate null value, derived from the design rule. A single realized path does not estimate those probabilities reliably.

Key takeaway: Unconditional intervals calibrate the full adaptive experiment. Their coverage is a probability-weighted average over paths, so the path weights, path-specific outcome distributions, and scientific coverage target must all be explicit.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A design has two paths with probabilities 0.20 and 0.80. Conditional coverages are 0.85 and 0.98. What is the unconditional coverage?
Why must the path probabilities be evaluated at the candidate value $\theta_0$ when they depend on the parameter?
In a test-inversion procedure, what event corresponds to the confidence set failing to contain the true parameter?
Can a 95% unconditional interval have 90% coverage on one path? Explain.
What is one important diagnostic to report in addition to the overall coverage?

Solutions. (1) The weighted coverage is $(0.20)(0.85)+(0.80)(0.98)=0.17+0.784=0.954$, or 95.4%. (2) The path probabilities describe how often the design reaches each path under the tested null; if they vary with $\theta$, using weights from another value gives the wrong null distribution. (3) Failure to cover the true value is exactly rejection of the test of $H_0$ at that true value. (4) Yes. The overall coverage is a weighted average, so another path with sufficiently high coverage can raise the average to 95%; this does not provide a 95% guarantee on the lower-coverage path. (5) Report path-specific coverage and the probability of each path, so poor performance on a clinically important path is not hidden by the average.