Test-Inversion Confidence Intervals

Why Invert a Test?

After an adaptive trial, the distribution of an estimator may depend on how the trial changed in response to interim data. A confidence interval built as if the final sample size or analysis plan had been fixed in advance can therefore have incorrect coverage. Test inversion offers a direct route to an interval: for each candidate value of the treatment effect, test that value using a procedure calibrated for the full adaptive design, then retain the values the test does not reject.

The central requirement is calibration. It is not enough to compute an ordinary fixed-design test statistic after adaptation; its null distribution must be valid under the design that actually generated the data, including the prespecified adaptation rule. The resulting interval inherits the error-rate guarantee of those tests.

Definition: Let $X$ denote all observed trial data, including information that determines adaptations, and let $\theta$ be the parameter of interest. Suppose that for every candidate value $\theta_0$, a test of $H_0:\theta=\theta_0$ rejects on a set $R_{\theta_0}$. The test-inversion confidence set at level $1-\alpha$ is $$ C(X)=\{\theta_0:\ X\notin R_{\theta_0}\}. $$ If each test has size at most $\alpha$ under the full trial design, then $P_\theta\{\theta\in C(X)\}\geq 1-\alpha$ for every $\theta$.

How the Coverage Guarantee Follows

The defining event for coverage is exactly the event that the test at the true parameter does not reject. This equivalence is the key justification; no separate interval approximation is needed.

\{\theta\in C(X)\}=\{X\notin R_\theta\} \quad\Longrightarrow\quad P_\theta\{\theta\in C(X)\} =1-P_\theta(X\in R_\theta)\geq 1-\alpha.

The inequality uses the level condition $P_\theta(X\in R_\theta)\leq\alpha$, evaluated under the complete design. If the tests have exactly size $\alpha$, the coverage is at least $1-\alpha$, and it is exactly $1-\alpha$ when the test has no conservatism or discreteness effects at the true value. If a test is conservative, its inverted set can have coverage greater than the nominal level.

The phrase “under the full design” matters. The probability includes the random interim outcomes, the adaptations they trigger, and the final outcomes collected under those adaptations. It does not automatically assert $1-\alpha$ coverage after conditioning on a particular interim event. Conditioning can change the distribution of the data used by the test; unconditional calibration alone does not promise conditional calibration.

A Two-Stage Model for an Adaptive Trial

To make the inversion concrete, suppose the outcome is normally distributed with known standard deviation $\sigma$, and the treatment effect $\theta$ is a difference in mean outcome, measured in outcome units. Stage 1 has a fixed sample size $n_1$. After observing its mean $\bar{x}_1$, the trial selects a stage-2 sample size $n_2$ according to a prespecified rule. Stage-2 outcomes are newly observed and independent of stage-1 outcomes, conditional on the selected sample size. There is no early stopping, and the treatment effect is assumed constant across stages.

For the illustration, use a fixed-weight combination statistic with equal weights. For any candidate value $\theta_0$, define

Z(\theta_0)=\frac{1}{\sqrt{2}}\left[ \frac{\sqrt{n_1}(\bar{x}_1-\theta_0)}{\sigma} + \frac{\sqrt{n_2}(\bar{x}_2-\theta_0)}{\sigma} \right].

Here $\bar{x}_2$ is the observed stage-2 mean at the sample size selected by the adaptation rule. Under $H_0:\theta=\theta_0$, each standardized stage statistic has a standard normal distribution. The second statistic remains standard normal conditional on the stage-1 data and selected sample size, because stage-2 observations are independent normal observations with the stated mean and variance. Its conditional distribution does not depend on the stage-1 data. Thus the two stage statistics are independent standard normal variables under the null, and their equal-weight combination $Z(\theta_0)$ is standard normal.

This argument relies on the fixed weights, known variance, independent stage outcomes, and the stated normal model. It is not a general claim that any statistic calculated after adaptation is standard normal. In particular, choosing weights based on the interim statistic would require a new calibration argument.

A two-sided level-$\alpha$ test rejects $H_0:\theta=\theta_0$ when $|Z(\theta_0)|>z_{1-\alpha/2}$, where $z_q$ is the $q$ quantile of the standard normal distribution. Inversion retains precisely those $\theta_0$ satisfying $|Z(\theta_0)|\leq z_{1-\alpha/2}$. Solving this inequality yields an interval:

C(X)= \left[ \frac{\sqrt{n_1}\bar{x}_1+\sqrt{n_2}\bar{x}_2 -z_{1-\alpha/2}\sigma\sqrt{2}}{\sqrt{n_1}+\sqrt{n_2}}, \quad \frac{\sqrt{n_1}\bar{x}_1+\sqrt{n_2}\bar{x}_2 +z_{1-\alpha/2}\sigma\sqrt{2}}{\sqrt{n_1}+\sqrt{n_2}} \right].

The algebra works because $Z(\theta_0)$ is a decreasing linear function of $\theta_0$: the two-sided acceptance condition bounds that function above and below, producing lower and upper limits. The interval’s center is a weighted combination of the stage means, with weights proportional to the square roots of their sample sizes. Its coverage guarantee, however, comes from the calibrated test—not from treating that center as an ordinary estimator with a fixed-design standard error.

Worked Example: Inverting a Two-Sided Test

Suppose a hypothetical adaptive trial measures a clinical score in points, with known $\sigma=4$ points. It has $n_1=16$ participants at stage 1 and, under the prespecified interim rule, selects $n_2=16$. The observed stage means are $\bar{x}_1=1.0$ point and $\bar{x}_2=2.0$ points. Find the 95% test-inversion interval.

The standard normal critical value is $z_{0.975}=1.96$. The inverted interval has center $$ \frac{\sqrt{16}(1.0)+\sqrt{16}(2.0)}{\sqrt{16}+\sqrt{16}} =\frac{4+8}{8}=1.50\text{ points}. $$ Its half-width is $$ \frac{1.96(4)\sqrt{2}}{4+4} =\frac{11.087}{8}=1.386\text{ points}. $$ Therefore the 95% interval is $$ [1.50-1.386,\ 1.50+1.386]=[0.114,\ 2.886]\text{ points}. $$

For example, the candidate value $\theta_0=0$ gives $$ Z(0)=\frac{4(1.0)+4(2.0)}{4\sqrt{2}}=2.121. $$ Since $2.121>1.96$, the test rejects zero, which is consistent with zero being outside the inverted interval. The 95% coverage claim is across repetitions of this adaptive design under the model, not a 95% probability that this particular interval contains the fixed treatment effect.

Examples With an Adapted Sample Size

The sample size selected after stage 1 appears in the observed interval. That does not invalidate inversion: validity comes from the distribution of the test statistic under the adaptation rule, not from pretending that $n_2$ was fixed beforehand.

Worked Example: A Larger Second Stage Is Selected

In a hypothetical trial, $\sigma=8$ mmHg and $n_1=16$. The protocol selects $n_2=36$ if the stage-1 mean reduction is at least 3 mmHg, and otherwise selects $n_2=16$. The observed stage-1 mean is 4 mmHg, so $n_2=36$; the stage-2 mean is 2.5 mmHg. Using the calibrated equal-weight test, find the 95% interval.

The observed means and sample sizes give $$ Z(\theta_0)= \frac{4(4-\theta_0)+6(2.5-\theta_0)}{8\sqrt{2}} =\frac{31-10\theta_0}{11.314}. $$ The nonrejection condition is $-1.96\leq Z(\theta_0)\leq1.96$. The center is the value that makes $Z(\theta_0)=0$: $$ \frac{31}{10}=3.10\text{ mmHg}. $$ The half-width is $$ \frac{1.96(11.314)}{10}=2.217\text{ mmHg}. $$ Thus the 95% interval is $[3.10-2.217,\ 3.10+2.217]=[0.883,\ 5.317]$ mmHg. This interval uses the selected $n_2=36$, while its test calibration accounts for the rule that selected that sample size.

Worked Example: Inverting a One-Sided Test

A hypothetical trial has $\sigma=5$ points, $n_1=9$, and an adaptation rule that selects $n_2=25$ after the observed interim data. The stage means are $\bar{x}_1=1.2$ points and $\bar{x}_2=2.0$ points. To obtain a 95% one-sided lower confidence bound, invert the upper-tail test that rejects $H_0:\theta=\theta_0$ when $Z(\theta_0)>1.645$.

For these data, $$ Z(\theta_0)= \frac{3(1.2-\theta_0)+5(2.0-\theta_0)}{5\sqrt{2}} =\frac{13.6-8\theta_0}{7.071}. $$ The acceptance condition is $Z(\theta_0)\leq1.645$. Since the statistic decreases as $\theta_0$ increases, solving gives $$ 13.6-8\theta_0\leq1.645(7.071)=11.632, \qquad \theta_0\geq\frac{13.6-11.632}{8}=0.246. $$ The inverted set is $[0.246,\infty)$ points. Under the stated model and calibrated upper-tail tests, this is a 95% one-sided lower confidence set: its lower endpoint is a bound, not a two-sided interval endpoint.

Calibration, Diagnostics, and Alternatives

In practice, the null distribution used for each test may be obtained analytically, by exact enumeration, or by simulation of the prespecified trial design under the candidate null value. For simulation calibration, the simulated trials must reproduce the adaptation rule, stopping decisions if any, and final analysis statistic. Calibration should be checked across relevant values of $\theta_0$, especially when the distribution is discrete, the adaptation rule has thresholds, or nuisance parameters are estimated. The simulated rejection frequency should not exceed the target $\alpha$ beyond Monte Carlo error.

Useful checks include verifying that the statistic’s null distribution matches the one used to choose its critical value; confirming that all adaptation decisions are reproduced; checking whether the acceptance set is bounded and connected; and confirming that its endpoints satisfy the test boundary equations. If the acceptance region is not a single interval, inversion can produce multiple disjoint pieces. Reporting only the smallest interval that contains them changes the set and does not automatically preserve the original coverage guarantee.

Approach	How it forms an interval or set	Main caution
Test inversion	Retains parameter values not rejected by tests calibrated to the adaptive design.	Requires valid tests across candidate parameter values; the set may be awkward or disconnected.
Naive Wald interval	Centers an estimate and adds or subtracts a normal critical value times an estimated standard error.	A fixed-design standard error and normal approximation may not account for selection or adaptation.
Likelihood-based interval	Uses the likelihood to compare candidate parameter values with an estimate.	Its usual approximation needs justification under the adaptive sampling and selection process.

Test inversion is particularly attractive when a defensible adaptive-design test is available. It does not itself make an invalid test valid, remove bias from a point estimate, or guarantee a short interval. The test may be conservative, and a valid interval may consequently be wider than a naive Wald interval. That width reflects the inferential procedure’s calibration; it is not a reason to replace the procedure with an uncalibrated one.

A Common Mistake

A frequent error is to say, “The final sample size is known now, so I can use the ordinary fixed-sample interval for that sample size.” This treats the selected sample size as if it had been set independently of the data. When the interim outcome affects the sample size, that reasoning can ignore the selection induced by the adaptation. The correction is to calibrate the test under the rule that generated the sample size and then invert that test. In the normal example above, the fixed-weight combination test has a justified null distribution under the specified assumptions; an arbitrary statistic with weights or critical values chosen after looking at interim results does not inherit that justification automatically.

Key takeaway: A test-inversion confidence set contains exactly the parameter values whose calibrated tests do not reject. Its coverage guarantee follows directly from the tests’ size control, provided that size control applies to the full adaptive design and the parameter value of interest.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If every test in a family has size at most 0.05 under the full design, what coverage guarantee follows for the inverted set?
In the equal-weight normal combination test, why does the stage-2 statistic remain standard normal under the null after the sample size is selected from stage-1 data?
For a one-sided test that rejects when $Z(\theta_0)>c$, what inequality defines the inverted acceptance set?
Why does unconditional calibration not, by itself, guarantee 95% coverage conditional on a particular adaptation event?

Solutions. (1) For every $\theta$, $P_\theta\{\theta\in C(X)\}=1-P_\theta(\text{reject }H_0:\theta=\theta)\geq0.95$. (2) Conditional on the stage-1 data and selected sample size, independent stage-2 normal observations have the stated null mean and variance, so their standardized mean is standard normal; this conditional distribution is unchanged across stage-1 outcomes. (3) Retain the candidate values satisfying $Z(\theta_0)\leq c$. (4) Conditioning changes the distribution of the data to a selected subset of trial outcomes, and an unconditional error-rate bound does not establish an error-rate bound within that subset.