Why Invert a Test?
A confidence interval for an average treatment effect can be built by asking, in turn, which candidate values of that effect are compatible with the observed data under a test calibrated for the randomization design. This approach is called test inversion. It is especially useful when the standard error used by the test depends on the candidate effect, because then the interval endpoints need not follow from a single estimate plus or minus a fixed margin.
The target here is the finite-population average treatment effect, not an effect in a superpopulation. Each unit has two fixed potential outcomes, while the adaptive assignment mechanism determines which one is observed. The central challenge is that the null hypothesis specifies an average effect but does not specify every unit’s treatment effect. Consequently, tests of this weak null are generally not finite-sample exact randomization tests.
Write \(Z_i=1\) if unit \(i\) is assigned treatment and \(Z_i=0\) otherwise. The observed outcome is \(Y_i=Z_iY_i(1)+(1-Z_i)Y_i(0)\). In a sequential adaptive design, let \(\pi_i\) denote the conditional probability of assigning treatment to unit \(i\), given the history available immediately before that assignment. A common design-based estimator is the inverse-probability-weighted estimator
The probabilities must be those actually used by the design, and the history used to calculate \(\pi_i\) must precede unit \(i\)'s assignment. Conditional on that history, the expected contribution inside braces is \(Y_i(1)-Y_i(0)\). Iterated expectation therefore gives \(E(\hat{\tau}_{HT})=\tau\), when the probabilities are known and strictly between zero and one. This unbiasedness does not by itself ensure a useful standard error or accurate small-sample inference.
From Candidate Tests to a Confidence Set
For each proposed value \(\tau_0\), calculate a studentized statistic using the estimator and a design-appropriate estimated standard error. Denote that standard error by \(\widehat{SE}(\tau_0)\) when it can depend on the candidate. A two-sided large-sample test rejects at level \(\alpha\) when the absolute statistic exceeds the standard normal critical value.
Inverting this family of tests gives the confidence set
A candidate belongs to the set exactly when its test is not rejected. If \(\widehat{SE}\) is constant in \(\tau_0\), this is the familiar symmetric interval \(\hat{\tau}\pm z_{1-\alpha/2}\widehat{SE}\). If it varies with \(\tau_0\), solve the inequality using the stated standard-error formula, or evaluate it over a sufficiently fine grid and check the boundaries. Do not assume the result must be symmetric or even a single interval without examining the acceptance set.
Worked Example: Inverting a Test With a Fixed Standard Error
Suppose a hypothetical adaptive trial has \(\hat{\tau}=4.2\) points and a design-appropriate estimated standard error of \(1.5\) points that is constant across candidate values. Use a two-sided 5% test, with \(z_{0.975}=1.96\).
For a candidate \(\tau_0\), the test statistic is \(T(\tau_0)=(4.2-\tau_0)/1.5\). A candidate is retained when
Thus \(4.2-2.94\le\tau_0\le4.2+2.94\), giving the 95% confidence interval \([1.26,7.14]\) points. Under the asymptotic conditions discussed below, the procedure is designed to cover the finite-population average effect in repeated randomizations about 95% of the time. It does not mean there is a 95% probability that this fixed effect lies in this particular realized interval.
Candidate-Dependent Standard Errors
Some weak-null procedures use a variance estimate that depends on \(\tau_0\). In that case inversion can produce a quadratic or another nonlinear inequality. The algebra must retain only candidate values for which the estimated variance is nonnegative and the test is defined. The following hypothetical calculation illustrates the endpoint arithmetic; its variance function is specific to the example, not a universal formula for adaptive trials.
Worked Example: Solving a Candidate-Dependent Inequality
Suppose a hypothetical analysis reports \(\hat{\tau}=10\) points and, for \(x=\tau_0-10\), uses the candidate-dependent estimated variance \(\widehat{SE}(\tau_0)^2=5.02823+2.16597x\) square points. The variance is positive over the accepted range below. At the 5% two-sided level, inversion requires \(x^2\le1.96^2(5.02823+2.16597x)\).
Since \(1.96^2=3.8416\), rearranging gives, to the displayed precision,
The roots of this quadratic are approximately \(-1.8915\) and \(10.2123\). Its leading coefficient is positive, so the inequality holds between the roots. Therefore \(x\in[-1.8915,10.2123]\), and adding 10 points gives the confidence set \([8.109,20.212]\) points, rounded to three decimal places. The endpoint calculation depends on the displayed variance formula and its rounding; an actual analysis should retain full precision until the final report.
Why Coverage Is Asymptotic
Test inversion transfers the calibration of the candidate test to the confidence set, but it cannot make a poorly calibrated test valid. A typical justification is that under the true candidate \(\tau_0=\tau\), the studentized statistic converges in distribution to a standard normal random variable. This requires conditions on the design, the potential outcomes, and the variance estimator.
Each \(\pi_i\) is determined from information available before assignment \(i\), and the recorded probability matches the probability actually used by the design.
Assignment probabilities stay away from zero and one, or the design otherwise ensures that no few inverse-weighted observations dominate the total. Potential outcomes must also satisfy suitable boundedness or moment conditions.
The randomization variance grows at an appropriate rate and no single contribution dominates it, so a design-based central limit theorem applies to the centered estimator.
The estimated standard error, evaluated at the true candidate, consistently estimates the relevant design-based standard deviation. The studentized statistic then has an asymptotic standard normal distribution.
Under these conditions, if \(T(\tau)\) converges in distribution to \(N(0,1)\), then
This is an asymptotic coverage statement, not a finite-sample identity. In a finite sample, the distribution of the statistic can be skewed or discrete, the variance estimate can be unstable, and a normal critical value need not give exactly the nominal rejection probability.
Worked Example: A Horvitz–Thompson Estimate Under Adaptive Assignment
Consider four hypothetical units assigned sequentially. Their assignment probabilities are calculated from the prior history; the table lists those probabilities, assignments, and observed outcomes. No unobserved potential outcome is needed to calculate the displayed estimator.
| Unit | \(\pi_i\) | \(Z_i\) | Observed \(Y_i\) | HT contribution |
|---|---|---|---|---|
| 1 | 0.50 | 1 | 8 | \(8/0.50=16\) |
| 2 | 0.40 | 0 | 4 | \(-4/0.60=-6.667\) |
| 3 | 0.60 | 1 | 6 | \(6/0.60=10\) |
| 4 | 0.50 | 0 | 3 | \(-3/0.50=-6\) |
The estimated average effect is
This arithmetic illustrates how known assignment probabilities enter the estimator. It does not establish that a normal approximation is accurate for only four units, nor does the table alone provide a valid standard error. For inference, the analyst needs a variance estimator justified for the specific adaptive design and conditions adequate for its approximation.
Diagnostics, Alternatives, and Limitations
Inspect the assignment-probability history, the range of inverse weights, and the contributions to the estimated variance. Probabilities close to zero or one can produce large weights and make the normal approximation unreliable. Also check that the variance estimator is nonnegative and that its formula matches the assignment mechanism; a variance formula borrowed from complete randomization does not automatically remain valid under adaptive assignment. When the standard error depends on \(\tau_0\), inspect the entire accepted set rather than reporting only one root pair by habit.
A sharp null, such as \(Y_i(1)=Y_i(0)\) for every unit, specifies every missing potential outcome under the null. Under a known randomization mechanism, this can permit an exact randomization test by evaluating the statistic over assignments allowed by the design. The weak null \(\tau=\tau_0\) does not supply those unit-level effects, so exact enumeration under a sharp null is not an exact test of the weak null. A studentized weak-null test instead relies on a variance estimate and usually an asymptotic approximation.
When the standard error is candidate-invariant, the inverted test yields a simple Wald-type interval. Candidate-dependent studentization requires solving a more general inequality and can yield a set that is not a single interval. If an interval is required, the smallest interval containing the accepted set is a conservative enlargement, but it may include candidate values rejected by the original test. Neither approach removes the need to justify the design-based standard error. The introductory material on adaptive randomization inference provides context for why the assignment mechanism must be reflected in the analysis; here, the specific focus is the weak-null test and its inversion.
Common Mistakes
- Treating a weak null as a sharp null. The claim that the average effect equals \(\tau_0\) does not imply that every unit’s effect equals \(\tau_0\), so it does not determine all missing potential outcomes.
- Calling an asymptotic interval exact. A normal critical value and consistent standard error support approximate coverage under conditions, not guaranteed finite-sample coverage.
- Using the wrong assignment probabilities. In an adaptive trial, the probability must be the conditional probability used at that assignment, based only on the preceding history.
- Assuming the standard error is constant. If its formula depends on the candidate, the test inversion may require solving a nonlinear inequality and checking the resulting set.
- Rounding roots too early. Keep full precision through solving and endpoint transformations; small rounding errors can change a reported endpoint.
Check Your Understanding
Answer each question, then compare with the solutions below.
- What is the difference between the weak null \(H_0:\tau=\tau_0\) and the sharp null \(Y_i(1)-Y_i(0)=\tau_0\) for every unit?
- A test uses \(\hat{\tau}=2.5\) points, a constant standard error of \(0.8\) points, and \(z_{0.975}=1.96\). Find the inverted 95% interval.
- Why can inverse-probability weighting become unstable when adaptive assignment probabilities approach zero or one?
- If a candidate-dependent test accepts two separated ranges of candidate values, what set does direct inversion produce, and what is one consequence of reporting its interval hull?
Solutions. (1) The weak null constrains only the average of the unit effects; the sharp null specifies each unit’s effect and therefore determines missing potential outcomes under the null. (2) The margin is \(1.96(0.8)=1.568\) points, so the interval is \([2.5-1.568,2.5+1.568]=[0.932,4.068]\) points. (3) The inverse weights \(1/\pi_i\) or \(1/(1-\pi_i)\) can become very large, allowing a small number of observations to dominate the estimate and its variance. (4) Direct inversion gives a disconnected confidence set. Its interval hull contains that set and therefore does not lose coverage, but may include candidate values that the test itself rejects.