Inverting Sharp-Null Tests For Constant-Effect Intervals in Response-Adaptive Trials

From a Sharp Null to a Confidence Set

A randomization test can test a sharp null because that null specifies every participant’s missing potential outcome. Inverting such tests turns a collection of candidate-effect tests into a confidence set for a common treatment effect. In a response-adaptive trial, the important extra step is to reproduce the assignment rule under each candidate: later assignment probabilities may depend on responses that would differ across simulated assignment paths.

Let \(Z_i=1\) denote assignment to treatment and \(Z_i=0\) assignment to control for participant \(i\). Write \(Y_i(1)\) and \(Y_i(0)\) for that participant’s potential outcomes, and \(Y_i=Z_iY_i(1)+(1-Z_i)Y_i(0)\) for the observed outcome. The outcome scale and the direction of subtraction must be specified; here effects are measured as treatment minus control, in the outcome’s units.

Definition: For a candidate value \(\tau\), the sharp constant-additive-effect null is \(H_{0,\tau}:Y_i(1)-Y_i(0)=\tau\) for every participant \(i\). Under this null, \(\tau\) is the same additive treatment effect for every participant. It is stronger than a null stating only that an average treatment effect equals \(\tau\).

The test also requires a well-defined assignment mechanism: the randomization probabilities at each decision must be known or reproducible from the protocol and the observed history available at that decision. Potential outcomes are treated as fixed for randomization inference; assignments are the source of randomness. The usual potential-outcomes conditions are also needed: each participant has well-defined outcomes under either assignment, the observed outcome equals the potential outcome corresponding to the assignment received, and one participant’s assignment does not change another participant’s potential outcome. The outcome scale must support the proposed additive-effect model. If the design has random stopping, that stopping rule must be included in the mechanism being reconstructed.

Imputation and the Adaptive Reference Distribution

For a fixed candidate \(\tau\), the observed outcome reveals both potential outcomes under \(H_{0,\tau}\). If participant \(i\) received treatment, then \(Y_i(1)=Y_i\) and \(Y_i(0)=Y_i-\tau\). If participant \(i\) received control, then \(Y_i(0)=Y_i\) and \(Y_i(1)=Y_i+\tau\). Equivalently, the imputed outcome for a simulated assignment \(z^*\) is

Y_i^*(z_i^*;\tau)=Y_i-\tau Z_i+\tau z_i^*.

This formula uses the actual assignment \(Z_i\) and observed outcome \(Y_i\) to recover the candidate potential outcomes, then selects the outcome corresponding to simulated assignment \(z_i^*\). It is not enough to reassign the observed outcomes unchanged. Doing so would ignore the candidate effect and would generally test a different null.

Let \(T(z,y)\) be a statistic chosen before examining the test result. It might be a treatment-control contrast or another statistic sensitive to the effect being tested. Under candidate \(\tau\), each possible complete assignment path \(z^*\) has a probability determined by the trial’s sequential assignment rule. If a decision depends on past responses, those responses must be the imputed responses generated along that path. The probability of a path is the product of its sequential conditional assignment probabilities. The two-sided randomization p-value is

p_\tau=\sum_{z^*}\Pr_\tau(Z=z^*)\, I\!\left(\left|T\!\left(z^*,Y^*(z^*;\tau)\right)\right| \geq \left|T(Z,Y)\right|\right).

Here \(I(\cdot)\) is 1 when the condition holds and 0 otherwise. The observed statistic is calculated from the observed assignments and outcomes. The comparison uses the absolute statistic shown above; a one-sided test or a different pre-specified extremeness ordering is also possible, but the inversion and its interpretation must match that choice. Ties are included in the tail probability.

The accepted confidence set at significance level \(\alpha\) is

C_{1-\alpha}=\{\tau:p_\tau>\alpha\}.

A candidate is retained when its test does not reject at level \(\alpha\). This is a set of effects, not automatically a single interval. The justification for its coverage is that, if the constant-effect model is true with value \(\tau_0\), then \(H_{0,\tau_0}\) is sharp and the imputation is correct. A valid randomization p-value under that null satisfies \(\Pr_{\tau_0}(p_{\tau_0}\leq\alpha)\leq\alpha\). Therefore

\Pr_{\tau_0}(\tau_0\in C_{1-\alpha}) =\Pr_{\tau_0}(p_{\tau_0}>\alpha)\geq 1-\alpha.

The inequality can be strict because assignment paths are discrete and the test includes ties. Thus the set has at least its nominal coverage under the assumed constant-effect model; it need not have exactly that coverage.

Worked Example: Imputing Outcomes and Calculating a Candidate p-Value

Consider four participants in a hypothetical response-adaptive design. Participants 1 and 2 are assigned treatment and control, respectively. After observing participant 2’s response, the design assigns exactly one of participants 3 and 4 to treatment. If participant 2’s response is at most 7.5, the observed orientation, treatment for participant 3 and control for participant 4, is selected with probability 0.02; the reverse orientation is selected with probability 0.98. Suppose the observed assignments are \((1,0,1,0)\), and the observed outcomes, in outcome units, are \((10,7,9,8)\). Use the difference in arm means as \(T\), and test the candidate \(\tau=0\).

At \(\tau=0\), the sharp null says each participant’s two potential outcomes are equal. The observed outcomes therefore supply both potential outcomes for each participant. The observed statistic is

T_{\mathrm{obs}}=\frac{10+9}{2}-\frac{7+8}{2}=9.5-7.5=2.

Participant 2’s observed response is 7, so the adaptive rule gives the observed orientation probability 0.02 and the reverse orientation probability 0.98. The observed orientation has statistic 2. Under the reverse orientation, participants 1 and 4 receive treatment and participants 2 and 3 receive control, giving

T_{\mathrm{reverse}}=\frac{10+8}{2}-\frac{7+9}{2}=9-8=1.

Only the observed orientation is at least as extreme as the observed statistic in absolute value: \(|2|\geq 2\), but \(|1|<2\). Hence \(p_0=0.02\). At the 0.05 significance level, the candidate \(\tau=0\) is rejected. The p-value is the probability, under this candidate null and the stated adaptive rule, of a statistic at least as extreme as the one observed; it is not the probability that the null is true.

Inverting the Tests

For a fixed observed data set, inversion repeats the test for candidate values of \(\tau\). The observed outcomes and assignments stay fixed, but the missing potential outcomes and the simulated outcomes on each path change with \(\tau\). If the assignment probabilities depend on simulated responses, those probabilities can change too. Each candidate therefore has its own reference distribution.

Worked Example: The Inverted Set Can Be Disconnected

Continue with the four-participant design and data above. For a general candidate \(\tau\), the observed outcomes imply \(Y_1(0)=10-\tau\), \(Y_2(1)=7+\tau\), \(Y_3(0)=9-\tau\), and \(Y_4(1)=8+\tau\). The observed orientation’s statistic remains 2, because it reproduces the observed assignments and outcomes. Under the reverse orientation, the treatment mean is \(\{10+(8+\tau)\}/2\), while the control mean is \(\{7+(9-\tau)\}/2\). Thus

T_{\mathrm{reverse}}(\tau) =\frac{18+\tau}{2}-\frac{16-\tau}{2} =1+\tau.

The observed response from participant 2 remains 7 under every candidate because participant 2 received control and that observed outcome is known. The path probabilities are consequently 0.02 for the observed orientation and 0.98 for the reverse orientation at every \(\tau\). The observed orientation always meets the two-sided tail condition. The reverse orientation meets it exactly when \(|1+\tau|\geq 2\). Therefore

p_\tau= \begin{cases} 1, & \tau\leq -3\ \text{or}\ \tau\geq 1,\\ 0.02, & -3<\tau<1. \end{cases}

With \(\alpha=0.05\), the inverted set is \((-\infty,-3]\cup[1,\infty)\), in outcome units. It is disconnected and unbounded. The inclusive tail rule matters at the endpoints: there, the reverse-orientation statistic has absolute value 2 and is counted as extreme, so the p-value is 1. Reporting the interval from \(-3\) to 1 as though it were the accepted set would be wrong; it consists of rejected values. More generally, taking the interval hull of a disconnected set includes values that the tests rejected.

Coverage, Diagnostics, and Alternatives

The coverage statement is conditional on the model and mechanism used to construct the tests. To see why discreteness can make coverage conservative, consider the same design when the true constant effect is \(\tau_0=0\). The observed orientation occurs with probability 0.02 and has p-value 0.02. The reverse orientation occurs with probability 0.98 and has statistic 1; its p-value is 1 because both paths have absolute statistics at least 1. At \(\alpha=0.05\), the null is rejected only on the observed orientation, so its rejection probability is 0.02 and the confidence set covers \(\tau_0\) with probability \(1-0.02=0.98\), exceeding 0.95.

Worked Example: Checking Coverage in the Two-Path Design

Under the true effect \(\tau_0=0\), suppose the potential outcomes are those implied by the displayed hypothetical data. The assignment rule selects the observed orientation with probability 0.02 and the reverse orientation with probability 0.98. As just calculated, their p-values when each is observed are 0.02 and 1, respectively. The probability of rejecting the true candidate at level 0.05 is therefore

\Pr_{\tau_0}(p_{\tau_0}\leq 0.05) =0.02\,I(0.02\leq 0.05)+0.98\,I(1\leq 0.05) =0.02.

The probability that the inverted set contains the true value is \(1-0.02=0.98\). This verifies the at-least-95% coverage claim for this example and shows why a discrete randomization test need not attain its nominal error rate exactly.

A practical calculation should be checked for errors in both the imputation and the reference distribution. For every candidate, verify that each simulated outcome is the potential outcome corresponding to that simulated assignment; confirm that sequential probabilities use the correct simulated history; and check that the path probabilities sum to 1. Also confirm that the statistic and its two-sided or one-sided ordering were specified consistently. If the design’s randomization rule cannot be reconstructed, the resulting p-values do not have the stated design-based guarantee.

The central alternative is a test of a weak, average-effect null. Such a null does not specify every missing potential outcome, so it cannot generally be tested by this sharp-null imputation alone. Studentized randomization tests and Horvitz–Thompson contrasts address different inferential targets and rely on different arguments; they are not substitutes that make a constant-effect confidence set valid for an average effect. Likewise, permuting assignments as though the trial used a fixed allocation can give the wrong reference distribution when the actual probabilities adapted to response history.

In practice, one may calculate p-values on a grid of candidate effects and refine the search near changes in the accepted set. A grid is a numerical aid, not proof that all values between two accepted points are accepted. If the exact path space is large and simulation is used, Monte Carlo error must also be accounted for before making a definitive boundary claim. The limits are substantive as well as computational: a constant additive effect may be implausible, the chosen outcome scale matters, and the resulting coverage guarantee applies under the assumed constant-effect model—not automatically to a heterogeneous effect or an average effect.

Common Mistakes

Holding outcomes fixed while changing assignments. This fails to impose the candidate sharp null. Impute the missing potential outcomes under each \(\tau\), then use the corresponding outcomes along each simulated path.
Replaying probabilities from the observed trial history for every path. A response-adaptive design may make different decisions after different simulated responses. Recompute the sequential probabilities from each path’s history.
Calling every accepted set an interval. Inversion yields a confidence set, which can be disconnected or unbounded. Report its actual components; an interval hull is not equivalent.
Interpreting coverage without its model. The guarantee is for the true common additive effect when the sharp constant-effect model and assignment mechanism are correct. It does not establish coverage for a weak average-effect target.

Key takeaway: Invert a sharp-null test by testing each candidate additive effect with its own imputed potential outcomes and response-adaptive reference distribution. The resulting set has at least nominal coverage for the common effect under the assumed constant-effect model, but need not be a single interval.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For a participant assigned to control with observed outcome 12, what are the imputed \(Y_i(0)\) and \(Y_i(1)\) under candidate \(\tau=3\)?
Why must a response-adaptive assignment probability sometimes be recomputed for each simulated path?
In the two-path example, what is the p-value at \(\tau=0\), and what is the p-value at \(\tau=1\)?
Why does an inverted set with 95% coverage not necessarily contain exactly 95% of repeated-study sets under the model?
Does a confidence set obtained by inverting constant-effect sharp-null tests automatically provide coverage for an average treatment effect? Explain.

Solutions. (1) The observed control outcome gives \(Y_i(0)=12\); the candidate effect gives \(Y_i(1)=12+3=15\). (2) A simulated assignment can produce different observed responses from those on the actual path, and the rule may use those responses to choose later probabilities. The path probability must follow the rule applied to that simulated history. (3) At \(\tau=0\), the reverse statistic is 1, so only the observed path is as extreme as 2 and \(p_0=0.02\). At \(\tau=1\), the reverse statistic is \(1+1=2\), which is tied with the observed statistic and included, so \(p_1=1\). (4) Discrete randomization distributions and inclusive treatment of ties can make a level-\(\alpha\) test reject with probability less than \(\alpha\), giving coverage greater than \(1-\alpha\). (5) No. The guarantee concerns the common additive effect under the sharp constant-effect model; an average-effect null does not determine every missing potential outcome.