Exact Intervals by Inverting Randomization Tests

From a Sharp Null to a Confidence Set

A randomization test answers a question about a specified null hypothesis. If that null is a sharp statement about every unit’s treatment effect, the observed outcomes reveal all the potential outcomes needed to calculate the test’s randomization distribution. Repeating that test over possible effect values turns a collection of hypothesis tests into a confidence set.

This inversion is useful when an experiment is analyzed under a constant additive treatment effect assumption. It also requires care: the resulting finite-sample coverage guarantee is for that sharp constant-effect model. It does not automatically provide the same guarantee for the average treatment effect when individual effects may vary.

Definition: For unit $i$, let $Y_i(1)$ and $Y_i(0)$ be the potential outcomes under treatment and control, and let $Z_i$ be the assignment indicator. The observed outcome is $Y_i=Z_iY_i(1)+(1-Z_i)Y_i(0)$. The sharp null of a constant additive effect $\tau$ is $H_0(\tau):Y_i(1)-Y_i(0)=\tau$ for every unit $i$. Under this null, $Y_i(1)=Y_i(0)+\tau$, so the missing potential outcome for every unit can be imputed from the observed one.

Suppose $n$ units are assigned by complete randomization, with exactly $n_1$ assigned to treatment and $n_0=n-n_1$ assigned to control. The assignment set $\Omega$ contains all assignments with these group sizes; under complete randomization each is equally likely. Assume consistency and no interference, so each unit’s observed outcome is the potential outcome corresponding to its assignment.

For a candidate value $\tau$, define an adjusted outcome $R_i(\tau)=Y_i-\tau Z_i$. If $H_0(\tau)$ is true, then $R_i(\tau)=Y_i(0)$ for every unit, whether that unit was assigned treatment or control. Thus the adjusted outcomes are fixed across possible assignments under the candidate sharp null. For each assignment $w\in\Omega$, calculate a test statistic from these adjusted outcomes.

Definition: Let $\bar R_1(w;\tau)$ and $\bar R_0(w;\tau)$ be the means of $R_i(\tau)$ among units assigned treatment and control, respectively, under assignment $w$. Use the two-sided difference-in-means statistic $T(w;\tau)=\bar R_1(w;\tau)-\bar R_0(w;\tau)$. The conventional randomization p-value is the proportion of assignments whose absolute statistic is at least as large as the observed one: $$ p(\tau)=\frac{1}{|\Omega|}\sum_{w\in\Omega} \mathbf{1}\left\{|T(w;\tau)|\geq |T(Z;\tau)|\right\}. $$

The observed adjusted difference is the observed difference in means minus the candidate effect: $T(Z;\tau)=\bar Y_1-\bar Y_0-\tau$. The other assignments’ statistics must also be recalculated for each candidate $\tau$, because the adjusted outcomes depend on $\tau$. Inverting the test at significance level $\alpha$ gives

\mathcal{C}_{1-\alpha}=\{\tau:p(\tau)>\alpha\}.

The strict inequality corresponds to rejecting when $p(\tau)\leq\alpha$. A candidate is retained if its sharp null is not rejected by that rule. “Retained” does not mean the null has been proved true; it means the data do not reject it at the chosen level.

Worked Examples of Testing and Inversion

Worked Example: Testing One Candidate Effect

Consider four hypothetical units, with units 1 and 2 assigned treatment. Their observed outcomes are 14 and 16; the control outcomes for units 3 and 4 are 10 and 12. There are $\binom{4}{2}=6$ equally likely assignments. Test the candidate constant effect $\tau=3$ using the two-sided difference in means.

The observed difference in means is $(14+16)/2-(10+12)/2=4$. Under the candidate, the adjusted outcomes are $R(3)=(11,13,10,12)$: subtract 3 from each treated unit’s observed outcome and leave the controls unchanged. Their total is 46, so for any assigned pair the difference in adjusted means is the pair’s sum minus $46/2=23$.

The six possible treated pairs give the following statistics: pair 12, $24-23=1$; pair 13, $21-23=-2$; pair 14, $23-23=0$; pair 23, $23-23=0$; pair 24, $25-23=2$; and pair 34, $22-23=-1$. The observed assignment is pair 12, whose absolute statistic is 1. Four of the six absolute statistics are at least 1, so $$ p(3)=\frac{4}{6}=0.667. $$ At significance level $\alpha=0.05$, this candidate sharp null is not rejected. The result is a test decision about the constant-effect value 3, not a probability that this value is true.

Worked Example: Inverting the Tests to Obtain an Interval

Use the same hypothetical experiment and invert the tests at $\alpha=1/3$. For a general candidate $\tau$, the adjusted outcomes are $(14-\tau,16-\tau,10,12)$, with total $52-2\tau$. Half the total is $26-\tau$. Subtracting this quantity from each possible treated-pair sum gives the six statistics:

Pair 12: $4-\tau$; pair 13: $-2$; pair 14: $0$; pair 23: $0$; pair 24: $2$; and pair 34: $\tau-4$. The observed absolute statistic is $q=|4-\tau|$, and the complementary pair 34 has the same absolute statistic.

If $q>2$, only pairs 12 and 34 have absolute statistics at least $q$, giving $p(\tau)=2/6=1/3$. Those candidates are rejected because the rule rejects when $p(\tau)\leq\alpha$. If $0<q\leq2$, pairs 12, 13, 24, and 34 qualify, giving $p(\tau)=4/6>1/3$. When $q=0$, all six qualify, so $p(\tau)=1$. Therefore the retained values satisfy $$ |4-\tau|\leq2, $$ which is equivalent to $2\leq\tau\leq6$. The inverted confidence set is the interval $[2,6]$, in the units of the outcome. Its confidence level is at least $1-\alpha=2/3$ under the constant additive effect model; discreteness means coverage need not equal exactly $2/3$.

Why the Coverage Guarantee Is Finite-Sample

Under the true constant-effect model with value $\tau_0$, the adjusted outcomes $R_i(\tau_0)$ equal the fixed control potential outcomes $Y_i(0)$, regardless of the realized assignment. The randomization distribution used by the test therefore matches the actual assignment mechanism: the observed assignment is one uniformly selected member of $\Omega$, applied to a fixed set of adjusted outcomes.

For a test statistic whose randomization distribution is enumerated over all assignments, the observed statistic’s rank among the possible statistics gives a super-uniform p-value when ties are included. In particular, under the true sharp null, the chance that $p(\tau_0)\leq\alpha$ is at most $\alpha$. Taking the complement yields the coverage statement.

Fix the true candidate.
Under $H_0(\tau_0)$, every adjusted outcome $R_i(\tau_0)$ is the fixed value $Y_i(0)$, for every assignment.

Use the actual assignment distribution.
Complete randomization selects each $w\in\Omega$ with probability $1/|\Omega|$, exactly the distribution used to calculate the p-value.

Translate the test into coverage.
Because the inclusive-tail p-value is super-uniform under the sharp null, $P\{p(\tau_0)\leq\alpha\}\leq\alpha$. Thus $P\{\tau_0\in\mathcal{C}_{1-\alpha}\}=P\{p(\tau_0)>\alpha\}\geq1-\alpha$.

This is a finite-sample randomization argument; it does not rely on a large-sample normal approximation. The conservatism comes from the discreteness of the assignment distribution and from counting ties in the tail. A randomized test that breaks ties appropriately can attain a specified rejection probability under the sharp null, but the usual nonrandomized p-value and its inverted set generally guarantee coverage of at least the nominal level rather than exact equality.

Inverting a test produces a set of candidate values. In many applications that set is an interval, as in the example, but inversion does not in general guarantee that every accepted set is connected. If a reported interval is formed by taking the smallest interval containing a disconnected accepted set, that interval includes the entire accepted set and is therefore at least as conservative as the original set. The inversion rule, rather than an assumption that the answer must be one interval, determines which values are retained.

The Limit: Heterogeneous Effects

When effects vary, the finite-population average treatment effect is $\tau_{\mathrm{ATE}}=n^{-1}\sum_i\{Y_i(1)-Y_i(0)\}$, but the null $\tau_{\mathrm{ATE}}=\tau$ is generally not sharp. Knowing that the average equals $\tau$ does not reveal every missing potential outcome. Subtracting $\tau$ from treated outcomes then does not reconstruct a fixed set of control outcomes under that average-effect null. Consequently, the exact randomization calibration just established does not automatically apply to tests of the weak, average-effect null.

Worked Example: Average-Effect Coverage Can Fall Below Nominal

Consider a hypothetical finite population of four units, with $n_1=n_0=2$. Let all control potential outcomes be zero and let the individual treatment effects be $(0,0,0,4)$. The finite-population average treatment effect is $4/4=1$, although effects are not constant. Invert the two-sided sharp-null tests at $\alpha=0.34$, and ask how often the resulting set contains the true average effect $\tau=1$ over the six possible assignments.

For any realized assignment, test $\tau=1$ by subtracting 1 from the observed outcome of each treated unit. If the treated pair is 12, 13, or 23, both treated units have observed outcome zero. The adjusted outcomes are therefore a permutation of $(-1,-1,0,0)$. Across the six possible reassigned pairs the statistics are $-1,0,0,0,0,1$. The observed absolute statistic is 1, so $p(1)=2/6=0.333$. Since $0.333\leq0.34$, the test rejects $\tau=1$ for each of these three assignments.

If the treated pair includes unit 4, the adjusted outcomes are a permutation of $(-1,0,0,3)$. For the observed pair, the absolute statistic is 1; enumerating all six pairs gives absolute statistics $2,2,1,1,2,2$, so $p(1)=6/6=1$. The test does not reject for those three assignments. Thus the inverted set contains the true average effect for only $3/6=0.5$ of assignments, below the nominal coverage $1-\alpha=0.66$.

This calculation does not contradict the sharp-null coverage result: the individual effects are heterogeneous, so the constant-effect null at $\tau=1$ is false even though the average effect equals 1. The inversion procedure is exact for its sharp constant-effect hypotheses, not automatically for an unrestricted average effect.

Common Mistakes and Practical Interpretation

Holding the randomization distribution fixed while changing $\tau$. The imputed adjusted outcomes $Y_i-\tau Z_i$ change with the candidate. Recalculate the statistic for every assignment at each candidate value.
Calling a retained candidate “proved.” A value with $p(\tau)>\alpha$ has not been rejected by this test. It is not thereby established as the true effect.
Equating the sharp null with an average-effect null. The assertion that every unit’s effect equals $\tau$ is stronger than the assertion that the effects average to $\tau$. Only the former permits the sharp-null imputation used here.
Expecting coverage to equal the nominal level exactly. With a finite set of assignments and inclusive ties, the usual test is often conservative. Its confidence set has coverage at least the nominal level under the constant-effect model.
Assuming an interval is automatically valid for any estimand. The guarantee follows from the null tested and the randomization design. A set obtained this way does not inherit that guarantee for a different, heterogeneous-effect estimand.

Key takeaway: Invert a sharp-null randomization test by imputing outcomes under each candidate constant effect, enumerating the design’s assignments, and retaining candidates whose p-values exceed $\alpha$. This yields finite-sample coverage of at least $1-\alpha$ when the constant additive effect model holds; it does not by itself yield that coverage for an unrestricted average treatment effect.

Check Your Understanding

Answer each question using the definitions and randomization logic developed above.

Why can the missing potential outcomes be imputed under the sharp null of a constant additive effect, but not generally under a null about only the average effect?
In a two-sided inversion using the rule $p(\tau)>\alpha$, what happens to a candidate whose p-value equals $\alpha$?
Why must the randomization statistics be recalculated for each candidate effect?
What does the finite-sample coverage guarantee say under a true constant additive effect, and why can coverage exceed the nominal level?
Can an interval obtained by this inversion be interpreted as a guaranteed confidence interval for a heterogeneous average treatment effect? Explain.