Pairwise Contrasts In Adaptive Multiarm Trials

Why a Pairwise Test Needs Care

An omnibus randomization test asks whether the outcomes are compatible with a null covering all treatment arms. A pairwise test asks a narrower question: whether two specified arms, say $a$ and $b$, would give the same outcome to every trial participant. The narrower null can be useful when the scientific comparison was chosen in advance, but it supports fewer imputations. In particular, it does not say what would have happened to a participant assigned to a third arm $c$ under assignment to $a$ or $b$.

This distinction matters in a response-adaptive trial. Assignment probabilities can depend on the accumulated assignment and outcome history. Consequently, the reference distribution for a contrast is generally not the distribution obtained by uniformly permuting treatment labels or by pretending that allocations were fixed in advance. A valid test must respect both the pairwise null and the actual assignment mechanism.

Definition: Let $Y_i(z)$ denote the potential outcome for participant $i$ under arm $z$. The pairwise sharp null for arms $a$ and $b$ is $$ H_0^{ab}: Y_i(a)=Y_i(b)\quad\text{for every participant }i. $$ It makes no claim that $Y_i(a)=Y_i(c)$ for any other arm $c$. A sharp null specifies the missing pair-arm outcomes needed to compare $a$ with $b$, but only for participants whose assignments are varied between those two arms.

The Contrast and Its Reference Distribution

For an observed assignment vector $Z=(Z_1,\ldots,Z_N)$, where $Z_i$ is participant $i$'s assigned arm, a simple statistic is the difference in observed arm means:

T(Z,Y)=\bar{Y}_a-\bar{Y}_b,\qquad \bar{Y}_z=\frac{\sum_{i=1}^N I(Z_i=z)Y_i^{\mathrm{obs}}} {\sum_{i=1}^N I(Z_i=z)}.

Here $I(\cdot)$ is an indicator and $Y_i^{\mathrm{obs}}=Y_i(Z_i)$. The statistic is defined when both arms have at least one assigned participant. It is a descriptive contrast; under adaptive assignment it need not be an unbiased estimator of a population or finite-trial average effect. A randomization test can nevertheless use it: validity comes from the null and the reference distribution, not from unbiasedness of the statistic.

Write $H_{t-1}$ for the observed assignment and outcome history before decision $t$, and let $q_t(z\mid H_{t-1})$ be the design's probability of assigning the next participant to arm $z$. For a complete assignment path $z=(z_1,\ldots,z_N)$, its probability is obtained by multiplying the sequential probabilities along that path, updating the history at each step. When the algorithm uses additional recorded information, that information must also be included in the history.

A pairwise null does not permit us to replay every possible multiarm path: if a participant changes from $c$ to $a$, the outcome under $a$ is not supplied by $H_0^{ab}$. A direct exact approach is therefore to condition on an assignment restriction $C$, chosen without using outcomes, that holds assignments to arms other than $a$ and $b$ fixed. One may also condition on a prespecified assignment feature, such as the number assigned to each of the two arms, if the restricted support remains adequate. For every path in this restricted support, pair-arm outcomes can be imputed under the null; outcomes on fixed non-pair assignments remain the observed outcomes.

P(Z=z\mid C,H_0^{ab})= \frac{\prod_{t=1}^N q_t(z_t\mid H_{t-1}(z))} {\displaystyle\sum_{u\in\mathcal{Z}(C)} \prod_{t=1}^N q_t(u_t\mid H_{t-1}(u))}, \qquad z\in\mathcal{Z}(C).

In this expression, $\mathcal{Z}(C)$ is the set of paths satisfying the conditioning restriction. Each history $H_{t-1}(z)$ is reconstructed for that path using the observed outcomes for fixed non-pair assignments and the common imputed outcome for pair assignments under $H_0^{ab}$. The expression is the full sequential design probability, normalized over the restricted paths; it is not generally uniform.

Definition: A conditional pairwise randomization test compares the observed statistic with its distribution over $\mathcal{Z}(C)$, assigning each path its conditional probability under the full adaptive assignment mechanism. For a prespecified upper-tailed alternative, its exact p-value is $P(T(Z,Y)\geq T_{\mathrm{obs}}\mid C,H_0^{ab})$. For a two-sided test based on extremeness in absolute value, use $P(|T(Z,Y)|\geq |T_{\mathrm{obs}}|\mid C,H_0^{ab})$. Ties are included.

Conditioning is a way to make the pairwise null sufficient for imputation, not a license to discard parts of the design arbitrarily. The restriction $C$ and the test direction should be specified independently of the observed outcomes. If no nontrivial assignment paths remain, or if a required arm is absent on some paths, the selected statistic or conditioning scheme is not usable as stated.

Worked Examples

Worked Example: Stating the Null and Computing the Observed Contrast

Suppose a hypothetical adaptive trial compares three arms, A, B, and C. The prespecified comparison is A versus B. The observed outcomes (in symptom-score points, with lower scores better) are A: 6, 5, 8; B: 4, 7, 5; and C: 9, 6. Compute the observed contrast and state the pairwise sharp null.

The observed mean in A is $(6+5+8)/3=19/3=6.333$ points. The observed mean in B is $(4+7+5)/3=16/3=5.333$ points. Thus $$ T_{\mathrm{obs}}=\bar{Y}_A-\bar{Y}_B =\frac{19}{3}-\frac{16}{3}=1.000\text{ point}. $$

The null is $Y_i(A)=Y_i(B)$ for every participant $i$. It does not assert that C has no effect or that C's potential outcomes equal those under A or B. The observed contrast is positive, so this sample has a higher mean symptom score in A than in B. That descriptive direction alone is not evidence against the null; the adaptive reference distribution is needed to assess how unusual the contrast is.

Worked Example: An Adaptive Conditional Reference Distribution

Consider a hypothetical three-arm trial in which one participant is assigned to C and, under a prespecified assignment restriction, three subsequent participants receive either A or B. Their outcomes under the pairwise null are 4, 7, and 5 points, in that order. The design assigns A with probability 0.5 at the first pair decision; at the next decision it assigns A with probability 0.3 if the preceding outcome was below 6; at the final decision it assigns A with probability 0.6 if the preceding outcome was at least 6. Otherwise, it assigns B. The observed pair assignments are B, A, B. Test the one-sided alternative that A has a lower mean outcome than B, using the conditional distribution given exactly one A assignment.

Under the pairwise sharp null, the three outcomes remain 4, 7, and 5 whichever of A or B each participant receives. The observed A mean is 7 points, and the observed B mean is $(4+5)/2=4.5$ points, so $T_{\mathrm{obs}}=7-4.5=2.5$ points. This observed contrast points away from the stated lower-mean alternative.

The three paths satisfying the one-A restriction are ABB, BAB, and BBA. Their probabilities before conditioning are calculated sequentially:

ABB: $0.5\times0.7\times0.4=0.140$.
BAB: $0.5\times0.3\times0.4=0.060$.
BBA: $0.5\times0.7\times0.6=0.210$.

The total probability of the conditioning event is $0.140+0.060+0.210=0.410$. Therefore the conditional path probabilities are $0.140/0.410=0.3415$, $0.060/0.410=0.1463$, and $0.210/0.410=0.5122$, respectively.

For ABB, the contrast is $4-(7+5)/2=-2.000$ points. For BAB, it is $7-(4+5)/2=2.500$ points. For BBA, it is $5-(4+7)/2=-0.500$ points. For the lower-tailed alternative, the exact p-value is the probability of a contrast no greater than the observed 2.5 points: $$ p=P(T\leq 2.5\mid C,H_0^{AB})=0.3415+0.1463+0.5122=1.000. $$

Every supported contrast is at most the observed value, so the observed result is not unusual in the direction specified. The probabilities are unequal because the adaptive rule changes assignment probabilities with the history. Treating the three paths as equally likely would give a different reference distribution and would not implement this design.

Worked Example: Why Reassigning Participants to a Third Arm Is Not Justified

In another hypothetical trial, participant 1 received C and had an observed outcome of 8 points; participants 2 and 3 received A and B, respectively, with observed outcomes 3 and 6 points. The target is the sharp null for A versus B. Can a reference distribution be formed by swapping all three arm labels among the participants?

No. The null implies $Y_2(A)=Y_2(B)=3$ and $Y_3(A)=Y_3(B)=6$, because participants 2 and 3 were observed in one of the two tested arms. It says nothing about $Y_1(A)$ or $Y_1(B)$: participant 1 was observed under C. A permutation assigning participant 1 to A or B would require one of those unknown outcomes to compute the statistic. Nor does the null provide the outcome under C for participants 2 or 3 if a proposed path reassigns them to C.

A valid pairwise reference distribution must therefore retain participant 1's C assignment and vary only assignments for which the pairwise null supplies the needed imputation. If the adaptive mechanism makes no such alternative path possible under the chosen restriction, this dataset and design do not provide a useful exact conditional test by this method. A broader sharp null covering C would permit more imputations, but it would test a different, stronger hypothesis.

Justification, Diagnostics, and Alternatives

The exactness argument is short. Under $H_0^{ab}$, swapping a participant between A and B leaves that participant's outcome unchanged. If assignments to other arms are fixed, every outcome needed to evaluate each supported path is therefore known. The adaptive policy then assigns each path its probability by the sequential rules, with histories reconstructed under that path. Conditional on $C$, the observed path has the stated conditional distribution under the null. Ranking its statistic within that distribution gives a p-value whose probability of being at most a chosen level is no greater than that level; discreteness and ties can make the test conservative.

P_{H_0^{ab}}\!\left(p\leq\alpha\mid C\right)\leq\alpha.

This justification depends on the assignment algorithm being correctly represented, including outcome-dependent updates, randomization restrictions, and any conditioning event. Practical checks include:

Confirm that every path in the reference support satisfies the prespecified restriction and has its probability computed from the actual sequential rule.
Verify that each path's imputed outcomes are identified by the pairwise null or are observed because the non-pair assignment is held fixed.
Check that the statistic exists on every included path. If a path leaves A or B empty, use a statistic defined on that support or choose a justified restriction that ensures both counts are positive.
Inspect the number and probability mass of supported paths. A very small support yields coarse p-values; zero-probability paths must not be included.

A common mistake is to reason, “The null says A and B are equal, so all treatment labels can be permuted.” The correction is that equality between A and B says nothing about potential outcomes under C or any other arm. Another mistake is to use a uniform permutation because it is convenient. Under adaptive assignment, path probabilities generally differ; the design probabilities, not convenience, determine the reference distribution.

A difference in means is transparent, but other prespecified statistics can be used. If the design supplies suitable inclusion probabilities, a Horvitz–Thompson contrast can weight arm-specific outcomes to account for unequal assignment probabilities. Its validity as a randomization statistic still requires evaluating it over the correct assignment distribution; weighting does not make fixed-allocation permutations valid. Conversely, a studentized or covariate-adjusted statistic may improve sensitivity in some settings, but it must also be recomputed on each supported path using only information available under the null. An omnibus statistic addresses a broader all-arms question, whereas this test targets one chosen pair.

The conditional procedure has limitations. Conditioning can reduce the reference support and power, and some adaptive designs may leave too few paths for a meaningful test. The pairwise sharp null is also stronger than a null of zero average effect: it requires equality for every participant, not merely equality on average. A rejection is evidence against that sharp null for the specified pair under the stated randomization model; it does not by itself establish a particular average effect or justify conclusions about other arms. When the pair was selected after examining results, the nominal p-value does not account for that selection.

Key takeaway: A pairwise sharp null supports imputation only between the two named arms. Construct the reference distribution from the actual adaptive assignment probabilities over paths for which all required outcomes are known; do not substitute uniform label permutations or an all-arms reassignment.

Check Your Understanding

Answer each question, then compare with the solutions.

For arms A and B, write the pairwise sharp null and state one claim it does not make.
A supported assignment path has sequential probabilities 0.4, 0.7, and 0.5. Its unnormalized path probability is 0.140. If the total probability of paths satisfying the conditioning event is 0.350, what is its conditional probability?
Under a two-sided absolute-statistic test, the observed statistic is 1.8. The reference paths have statistics $-2.0, 1.8,$ and $0.5$, with probabilities $0.2, 0.3,$ and $0.5$. Find the exact p-value.
Why can a pairwise A-versus-B null fail to justify a path that reassigns a participant from C to A?

Solutions. 1. The null is $Y_i(A)=Y_i(B)$ for every participant $i$; it does not assert equality between A and C. 2. The conditional probability is $0.140/0.350=0.400$. 3. Include paths with absolute statistic at least 1.8: the probabilities for $-2.0$ and $1.8$ sum to $0.2+0.3=0.500$. 4. The null does not specify that participant's potential outcome under A when the observed outcome was under C, so the reassigned path's statistic cannot generally be imputed.

Pairwise Contrasts In Adaptive Multiarm Trials

What You'll Learn