Expected Length Versus Coverage of Adaptive Intervals

Why Coverage and Length Must Be Compared Together

An adaptive trial can follow different routes, such as using a smaller or larger final sample depending on an earlier decision. A confidence interval may perform differently on each route: its coverage can vary, and its expected length can vary. Reporting only one overall coverage figure can conceal a route on which the interval misses its target often. Requiring the target on every route can protect against that weakness, but may increase the average interval length.

The comparison depends on what is guaranteed. Unconditional coverage averages over the random adaptation route and the final data. Route-specific coverage evaluates coverage among trials that followed a particular route. These are different properties. Neither coverage nor length alone summarizes the trade-off: a very wide interval can have high coverage, while an interval that is short on average can have inadequate coverage on an important route.

The expected length is not the length seen in one trial. It is a repeated-trial average under the specified design and model. Likewise, unconditional coverage describes repeated trials before knowing which route will occur. Route-specific coverage describes repeated trials restricted to a route. This tutorial compares those criteria in a model where each route’s precision is known and the final estimator has a normal distribution conditional on that route.

A Model for Comparing the Criteria

Suppose the target parameter is \(\theta\), and the final estimate is \(\widehat\theta\). On route \(r\), assume \(\widehat\theta\mid R=r\sim N(\theta,s_r^2)\), where \(s_r\) is a known standard error for that route. Assume also that every route has positive probability, the route probabilities are specified, and the conditional normal model is correctly calibrated. One way these assumptions can hold is for an independent pilot to determine the route, followed by final data whose estimator has the stated normal distribution on each route.

For a central interval with conditional coverage \(c_r\), define \(z_{(1+c_r)/2}\) as the standard normal quantile with cumulative probability \((1+c_r)/2\). Then the route-specific interval and its length are

Under the stated normal model, the standardized error \((\widehat\theta-\theta)/s_r\), conditional on route \(r\), has a standard normal distribution. The probability that it lies between the two symmetric endpoints is therefore \(c_r\). If \(\pi_r=P(R=r)\), the law of total probability gives

The first expression averages coverage; the second averages length. These identities are a direct justification for comparing procedures route by route before summarizing them overall. They do not justify using the normal intervals if the conditional distribution is not normal or the standard error is not correctly specified.

The Precision Cost of a Route-by-Route Requirement

Now require at least 95% coverage on each route. In this model, using the central 95% interval on both routes gives exact route-specific coverage \(0.95\): its critical value is \(z_{0.975}=\Phi^{-1}(0.975)\approx1.959964\). Its unconditional coverage is also 95%, because it averages two 95% values. The lengths are \(2(1.959964)(0.5)=1.959964\) units on route \(A\), and \(2(1.959964)(2)=7.839856\) units on route \(B\). Hence expected length is \(0.5(1.959964)+0.5(7.839856)=4.899910\) units.

The difference from the procedure in the worked example is \(4.899910-4.678711=0.221199\) units, or about \(4.51\%\) of the route-by-route procedure’s expected length. In this particular comparison, the procedure requiring 95% on both routes is longer on average. Its benefit is a clear guarantee for route \(B\); the cost is not a universal constant, but follows here from how coverage was allocated across routes.

The calculation also shows why stronger coverage requirements do not logically imply that every possible procedure must be longer. If a procedure already has at least 95% coverage on every route, it meets both criteria without any increase. If route-specific intervals have very different lengths, an unconditional procedure might allocate coverage in a way that is inefficient, and another procedure could satisfy a stronger requirement at similar or even lower expected length. The comparison must specify both procedures, the route probabilities, and the precision on each route.

Coverage Allocation When Routes Have Different Probabilities

The cost also depends on how likely each route is. Consider two routes with probabilities \(0.8\) and \(0.2\), and the same standard errors \(0.5\) and \(2\) units, respectively. Assign 96% coverage to the more frequent, more precise route and 91% to the other. The unconditional coverage is \(0.8(0.96)+0.2(0.91)=0.768+0.182=0.950\).

Assumptions, Diagnostics, and Limitations

The calculations rely on more than the stated overall coverage target. Before using them, verify that the parameter and estimator are defined consistently across routes; the route-specific standard errors are appropriate; and the conditional distribution of the estimator is adequately represented by the normal model. The route probabilities must also correspond to the adaptive design being evaluated. If the route is chosen using data that are also used in the final estimator, conditional normality and the stated standard error do not follow automatically. Selection can change the estimator’s conditional distribution, so a simple normal interval may not have its claimed route-specific coverage.

Useful diagnostics include simulating the full adaptation rule under parameter values relevant to the trial, then tabulating coverage and length separately by route as well as overall. Compare empirical route frequencies with the probabilities used in the expected-length calculation. Check whether nominal route-specific coverage is close to its target and whether interval length changes in the direction predicted by the route-specific standard errors. Large discrepancies can indicate a miscalibrated conditional distribution, incorrect route probabilities, or an implementation error. A satisfactory overall coverage estimate can hide a route-specific shortfall.

Expected length is only one precision summary. Two procedures with the same expected length can have different route-specific lengths, tail behavior, or probabilities of producing very wide intervals. Moreover, the numerical comparison depends on the chosen coverage allocations; the examples do not establish a universally best allocation. Alternatives include imposing a minimum coverage on every route, reporting both overall and route-specific operating characteristics, or using a common critical value across routes. Each choice answers a different design or reporting question.