Inverting Stopping-Aware Tests for Confidence Intervals

Why Inversion Must Respect the Stopping Rule

A confidence interval reported after an early stop is based on data selected by a monitoring design. Its coverage therefore depends not only on the statistic observed at the terminal look, but also on how the trial could have stopped at every look. Inverting stopping-aware tests gives a direct way to preserve that design in the inferential procedure: test each candidate parameter value using a test whose rejection probability accounts for all possible terminal stages, then retain the values not rejected.

The goal here is the construction principle, not a particular interval formula for one special design. The key requirement is that the test of each candidate value has a controlled type I error under the actual stopping rule. A procedure can achieve this by exact calculation under the stopped sampling law or, more simply but often less efficiently, by allocating error across looks and applying a union bound.

Model and Definition

For a concrete model, suppose the trial observes independent increments \(X_1,X_2,\ldots,X_K\), with \(X_j\sim N(\theta,1)\), where \(\theta\) is the parameter of interest and \(K\) is the maximum number of looks. Let \(S_k=\sum_{j=1}^k X_j\). Thus \(S_k\sim N(k\theta,k)\). The design specifies in advance when to stop: for each \(k\), let \(A_k\) be the set of data histories that lead to stopping at look \(k\). These terminal-stage regions are disjoint and their union contains every possible trial outcome. The regions are determined by the monitoring rule, not chosen after seeing the result.

This normal model is illustrative. In another application, the joint sampling distribution and stopping regions must be specified correctly for that application. In particular, a standard error estimated from the data cannot automatically be treated as known; the test calibration must reflect its sampling distribution if it is estimated.

Definition: For a candidate value \(\theta_0\), a stopping-aware level-\(\alpha\) test of \(H_0:\theta=\theta_0\) has rejection probability no greater than \(\alpha\) under that value, calculated over the trial's possible stopping stages. Inverting such tests gives the confidence set \(C=\{\theta_0:\text{the test of }H_0:\theta=\theta_0\text{ does not reject}\}\). If every candidate-value test has rejection probability at most \(\alpha\), then \(P_\theta(\theta\in C)\geq 1-\alpha\) for every \(\theta\) covered by the model.

The last statement follows from the definition of inversion. When the true value is \(\theta\), the event that the confidence set excludes \(\theta\) is precisely the event that the test of \(H_0:\theta=\theta\) rejects. Thus controlling that test's rejection probability controls the probability of noncoverage. Equality is not guaranteed: a conservative test can produce coverage greater than \(1-\alpha\).

Calculating Error Across Terminal Stages

Let \(R_k(\theta_0)\) denote the data histories at look \(k\) that lead the test of \(\theta_0\) to reject. Only histories in \(A_k\) can be observed as terminal data at that look. Under the null, the overall rejection probability is therefore the sum of the probabilities of rejection at each terminal stage:

P_{\theta_0}(\text{reject}) = \sum_{k=1}^{K}P_{\theta_0}\{A_k\cap R_k(\theta_0)\}.

The events in this sum are disjoint because a trial stops at only one look. In the independent-increment normal model, each term can be calculated by integrating the joint density of the first \(k\) increments over the terminal rejection region:

P_{\theta_0}\{A_k\cap R_k(\theta_0)\} = \int_{A_k\cap R_k(\theta_0)} \prod_{j=1}^{k}\phi(x_j-\theta_0)\,dx_1\cdots dx_k,

where \(\phi\) is the standard normal density. The stopping event is part of the integration region. Omitting it, or treating each possible look as though it were an independent fixed-sample experiment, generally gives the wrong rejection probability. Exact evaluation may require numerical integration or simulation from the specified model; every candidate \(\theta_0\) must be tested using the same prespecified design.

A Conservative Construction by Error Allocation

A useful construction avoids evaluating the full joint integral. At look \(k\), define the score for testing \(\theta_0\) as \(Z_k(\theta_0)=(S_k-k\theta_0)/\sqrt{k}\). Under the normal model and \(H_0\), each \(Z_k(\theta_0)\) has a standard normal marginal distribution. The scores across looks are not independent, but independence is unnecessary for the following bound.

Choose nonnegative stagewise error allocations \(\alpha_1,\ldots,\alpha_K\) with \(\sum_{k=1}^K\alpha_k\leq\alpha\). At look \(k\), reject the candidate value if \(|Z_k(\theta_0)|>c_k\), where \(c_k=\Phi^{-1}(1-\alpha_k/2)\) and \(\Phi\) is the standard normal cumulative distribution function. A rejection at the observed terminal look must be one of the events that some planned look's score crosses its threshold. Therefore:

P_{\theta_0}(\text{reject}) \leq \sum_{k=1}^{K}P_{\theta_0}\{|Z_k(\theta_0)|>c_k\} = \sum_{k=1}^{K}\alpha_k \leq\alpha.

This is the union bound. It remains valid when the stopping rule selects the terminal look based on earlier data, because it bounds the rejection event by the union of all planned-look crossing events. Correlation among the looks can make the bound conservative, but does not invalidate it. Inverting the test at the observed terminal look \(k\) retains the candidate values satisfying \(|S_k-k\theta_0|\leq c_k\sqrt{k}\), an interval in this model.

Worked Example: Inverting a Two-Look Test

Suppose a hypothetical trial has at most two looks. Its prespecified first-look rule stops for efficacy if \(S_1\geq2\), stops for futility if \(S_1\leq-1\), and otherwise continues. The observed first-look value is \(S_1=1.100\), so the trial continues; at look 2, \(S_2=3.253\). The increments are assumed to follow the stated independent normal model with known unit variance. Construct a two-sided 95% confidence set using equal Bonferroni allocation across the two looks.

Set \(\alpha_1=\alpha_2=0.025\), so each look has tail probability \(\alpha_k/2=0.0125\). The common critical value is \(c=\Phi^{-1}(0.9875)\approx2.2414\). At the observed terminal look, inversion requires \(|S_2-2\theta_0|/\sqrt{2}\leq c\). Solving the two inequalities for \(\theta_0\) gives:

\frac{S_2-c\sqrt{2}}{2} \leq\theta_0\leq \frac{S_2+c\sqrt{2}}{2}.

Substituting the unrounded observed score and critical value gives \((3.253-2.2414027\sqrt{2})/2\approx0.04159\) and \((3.253+2.2414027\sqrt{2})/2\approx3.21141\). The reported interval is approximately \(0.0416\) to \(3.211\) in the units of \(\theta\). Its coverage is at least 95% under the model and this test construction. The first-look continuation is consistent with the stated stopping boundaries, and the error bound includes both possible looks whether or not the trial reaches the second.

Worked Examples and Practical Use

Worked Example: A Stopping-Aware p-Value Bound

For the same hypothetical two-look data, test \(H_0:\theta=0\) against a two-sided alternative. The observed standardized scores are \(Z_1(0)=1.100\) and \(Z_2(0)=3.253/\sqrt{2}\approx2.300\). A conservative p-value for the maximum absolute score is the smaller of 1 and two times the single-look two-sided tail probability at the observed maximum:

p_{\mathrm{bound}} = \min\left\{1,\;2\left[2\{1-\Phi(2.300)\}\right]\right\} \approx \min\{1,\;2(0.02145)\} =0.04290.

The first factor of 2 accounts for the two planned looks; the two-sided normal tail probability is approximately 0.02145. This bound is below 0.05, so the conservative two-look test rejects \(H_0:\theta=0\) at the 5% level. That conclusion agrees with the inverted interval, whose lower endpoint is positive. This calculation uses the planned number of looks, not just the look at which the trial stopped.

Worked Example: Unequal Error Allocation at a Terminal Look

Consider a hypothetical three-look design using the same normal model and two-sided testing. Allocate \(\alpha_1=0.010\), \(\alpha_2=0.015\), and \(\alpha_3=0.025\); the allocations sum to 0.050. Suppose the trial stops at look 2 with \(S_2=3.000\). The second-look critical value is \(c_2=\Phi^{-1}(1-0.015/2)=\Phi^{-1}(0.9925)\approx2.4324\). Inversion at this terminal look gives:

\frac{3.000-2.4324\sqrt{2}}{2} \leq\theta_0\leq \frac{3.000+2.4324\sqrt{2}}{2},

Since \(2.4324\sqrt{2}\approx3.440\), the interval is approximately \(-0.220\) to \(3.220\). The allocations at looks 1 and 3 still matter: they are included in the overall error bound even though this trial stopped at look 2. The interval need not use the same critical value at every terminal stage, because the error budget was allocated unequally in advance.

Exact Calibration, Diagnostics, and Limitations

The union-bound construction is straightforward, but it can be wider than necessary. An alternative is to choose the rejection regions and calibrate their critical values so that the sum of the terminal-region probabilities is exactly \(\alpha\), or no greater than \(\alpha\), under each candidate null value. This uses the joint stopped sampling law, including correlations between looks and the stopping regions. Exact calibration can use the available error budget more efficiently; it requires a correctly specified model and careful computation. The essential difference is that the exact approach evaluates terminal rejection probabilities directly, whereas the union-bound approach controls them through marginal probabilities.

A fixed-sample interval applied after an early stop is a different approach. Its nominal coverage is justified under its own sampling plan, not automatically under a plan that selects the sample size using interim data. Likewise, a conditional coverage statement given a particular stopping stage is not interchangeable with unconditional coverage over all stages. The construction here targets the stated unconditional error guarantee unless the tests are explicitly calibrated to a different target.

Useful checks before reporting an inverted set include:

Check the design regions. Confirm that the efficacy, futility, and continuation rules used in the calculation match the prespecified monitoring rule and that terminal-stage events partition the possible outcomes.
Check the sampling law. Verify the assumed distribution, variance, dependence across looks, and treatment of any estimated standard error. A normal marginal score at each look does not by itself supply the correct joint stopped law.
Check error accounting. For a conservative construction, verify that the stagewise allocations sum to no more than \(\alpha\). For exact calibration, verify the sum of the terminal rejection probabilities over all stages.
Check inversion numerically. Evaluate the test at parameter values near each reported endpoint and just outside them. A numerical root-finding or integration error can move an endpoint or conceal a disconnected confidence set.

An interval results when, as in the normal score construction, the accepted candidate values form a continuous range. For more complicated rejection rules, the confidence set need not be an interval; it is the full set of nonrejected values. Reporting only its smallest and largest elements can incorrectly include values that the test rejects. Further, the stated guarantee is only as reliable as the model and the stopping rule supplied to the calculation. Misspecified boundaries, unplanned looks, or an incorrect variance model can undermine coverage.

Common Mistake

Common mistake: “The trial stopped at look 2, so I can use a 5% fixed-sample test at look 2 and ignore look 1.” The correction is to calibrate the test for the actual monitoring procedure. A valid conservative method allocates error across all planned looks and bounds the probability of crossing at any look; an exact method sums probabilities over all terminal-stage rejection regions. Stopping early does not erase the other possible paths that were part of the design.

A Practical Inversion Workflow

Specify the model and stopping rule.
State the sampling distribution under each candidate parameter value, the maximum number of looks, and the terminal-stage regions determined by the design.

Define the candidate-value test.
Specify the rejection rule for \(H_0:\theta=\theta_0\), including the treatment of every possible terminal stage.

Control its rejection probability.
Use exact stopped-law probabilities or a valid conservative bound such as stagewise error allocation.

Invert and verify.
Retain every \(\theta_0\) not rejected by the test and check endpoints or set boundaries against the test calculations.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Why are the terminal-stage rejection probabilities added rather than treated as probabilities from independent trials?
In a four-look design, stagewise two-sided error allocations are 0.005, 0.010, 0.015, and 0.020. Does the union-bound construction control overall type I error at 0.05?
At a terminal look \(k\), what inequality defines the accepted candidate values for a two-sided normal score test with critical value \(c_k\)?
Why can an exact stopped-law calculation be narrower than a Bonferroni-based interval?

Solutions. (1) The events correspond to mutually exclusive terminal stages: a trial stops at one stage only, so their probabilities sum to the overall rejection probability. They are not separate independent trials. (2) Yes. The allocations sum to \(0.005+0.010+0.015+0.020=0.050\), so the union bound gives an overall rejection probability no greater than 0.05. (3) The accepted values satisfy \(|S_k-k\theta_0|/\sqrt{k}\leq c_k\); solving this inequality gives \((S_k-c_k\sqrt{k})/k\leq\theta_0\leq(S_k+c_k\sqrt{k})/k\). (4) Exact calibration accounts for dependence across looks and the stopping regions, avoiding some of the conservatism of bounding the rejection probability by a union of marginal tail events.