Two Different Claims About Treatment Effects
A randomization test needs a null hypothesis that tells us what outcomes would have been observed under assignments other than the one that occurred. The crucial distinction is between a claim about every participant’s effect and a claim only about the average effect. The first can determine all missing potential outcomes; the second generally cannot.
Consider a fixed set of \(N\) trial participants and two treatment arms, labeled 1 and 0. For participant \(i\), let \(Y_i(1)\) and \(Y_i(0)\) be the outcomes that would be observed under treatment 1 and treatment 0, respectively, and let \(Z_i\) indicate the assigned arm. Assume the treatment versions are well-defined and one participant’s assignment does not change another participant’s potential outcomes. The observed outcome is
Thus, each participant contributes one observed potential outcome and one unobserved potential outcome. The finite-population average treatment effect is
This is an average over the trial participants, not necessarily an average over a broader population. The two null hypotheses below impose different restrictions on the \(2N\) potential outcomes.
The sharp null implies the weak null: if every difference \(Y_i(1)-Y_i(0)\) is zero, their average is zero. The converse does not hold. Positive and negative individual effects can cancel, leaving an average of zero even though treatment changes outcomes for many participants.
Why Sharpness Makes a Randomization Test Possible
In a randomization test, the outcomes are held fixed as assignments are varied according to the trial’s assignment mechanism. For a response-adaptive trial, that mechanism may assign each participant using probabilities conditional on the history observed so far. A path is a sequence of assignments; its probability must be evaluated using the design’s sequential assignment rules, not by assuming assignments were independent or that the final arm counts were fixed.
Under the sharp null, every participant’s observed outcome equals both potential outcomes. Therefore, for any candidate assignment path, the outcome that would be observed for participant \(i\) is already known: it is the same value as the one observed. The trial’s adaptive rule can be replayed using those outcomes to obtain the reference distribution of a chosen statistic.
For example, with a prespecified statistic \(T\), a two-sided Fisher randomization p-value sums the assignment probabilities of paths whose statistic is at least as extreme as the observed statistic. A small p-value is evidence against the sharp null, relative to this assignment mechanism. The test is exact, or conservative when discreteness prevents equality, when the sharp null holds and the assignment mechanism is correctly represented. This is a statement about the probability of the test’s rejection under that null, not the probability that the null is true.
Worked Example: A Zero Average With Unequal Individual Effects
Suppose a hypothetical trial has four participants. For illustration, imagine the complete potential-outcome table below is known to the analyst, although in an actual trial only one outcome per participant would be observed.
| Participant | \(Y_i(1)\) | \(Y_i(0)\) | Effect \(Y_i(1)-Y_i(0)\) |
|---|---|---|---|
| 1 | 14 | 10 | 4 |
| 2 | 4 | 8 | -4 |
| 3 | 13 | 10 | 3 |
| 4 | 4 | 7 | -3 |
The average treatment effect is
So the weak null holds for this finite set of participants. The sharp null does not hold: for instance, participant 1’s effect is 4 outcome units, not 0, and the other participants also have nonzero effects. The positive and negative effects cancel only when averaged.
If the observed assignments were \(Z=(1,0,1,0)\), the observed outcomes would be 14, 8, 13, and 7. An analyst who sees only those four observations cannot tell whether the unobserved outcomes are 10, 4, 10, and 4, as in the table, or something else. In particular, the observed outcomes alone do not establish that the weak null holds or that the sharp null holds.
Why a Weak Null Does Not Fill in Missing Outcomes
The weakness of the average-effect restriction can be seen by counting what it determines. The observed data provide \(N\) outcomes, one for each participant. The weak null adds the single equation
That equation constrains the sum of the \(N\) individual effects, but it does not specify each effect. For \(N>1\), many different sets of missing potential outcomes can satisfy the same average restriction. A randomization distribution that requires the outcome under each candidate assignment therefore cannot, in general, be obtained from observed outcomes plus the weak null.
This matters especially in an adaptive trial. An alternative assignment path may produce a different observed response history. If later assignment probabilities depend on that history, evaluating the probability of the path may itself require knowing outcomes that were not observed. The sharp null supplies the needed outcomes by making them invariant to assignment. A weak average-effect restriction does not.
Worked Example: The Same Observed Data, Different Missing Outcomes
Return to four participants assigned \(Z=(1,0,1,0)\), with observed outcomes \((14,8,13,7)\). One possible completion sets each missing potential outcome equal to the observed one. Then all four effects are zero, so both the sharp null and the weak null hold.
A different completion sets the missing control outcomes for participants 1 and 3 to 10 and 10, and the missing treatment outcomes for participants 2 and 4 to 12 and 12. The observed outcomes remain \((14,8,13,7)\), because those newly specified values belong to unassigned arms. The effects are now
so the average effect is \((4+4+3+5)/4=4\) outcome units. These two completions are both compatible with the same observed data, but they imply different effects. The weak-null restriction would rule out the second completion if it were known to hold; even then, it would not select a unique completion. For example, the first table in the previous example also matches these observed outcomes and has average effect zero while violating the sharp null.
This is an identification point, not just a calculation issue: observed outcomes do not reveal the missing potential outcomes. A test of a weak null must use an inferential method justified for that null, rather than treat one arbitrary completion as known.
Worked Example: Missing Outcomes Can Change an Adaptive Path Probability
Consider a hypothetical two-stage allocation rule. Participant 1 is assigned to arm 1 with probability 0.5. Participant 2 is then assigned to arm 1 with probability 0.8 if participant 1’s observed outcome is at least 5, and with probability 0.2 otherwise. Suppose participant 1 was assigned to arm 1 and had observed outcome 8.
Under the sharp null, participant 1’s outcome under arm 0 must also be 8. If we replay the path with \(Z_1=0\), the response remains 8, so the probability of assigning participant 2 to arm 1 is 0.8. The probability of the path \((Z_1,Z_2)=(0,1)\) is then
Under a weak null, participant 1 could instead have had an outcome below 5 under arm 0. For a concrete compatible potential-outcome pattern, let participant 1’s outcomes be \(Y_1(1)=8\) and \(Y_1(0)=2\), and let participant 2’s treatment effect be \(-6\). The average effect across these two participants is \((6-6)/2=0\), yet the sharp null is false. Under this pattern, the path \((0,1)\) has probability \(0.5\times 0.2=0.1\), not 0.4.
The example isolates the issue: the adaptive rule is known, but its path probabilities can depend on outcomes that are missing under a candidate assignment. The weak null does not supply those outcomes. In an actual analysis they are not available merely because the average effect is hypothesized to be zero.
What the Two Nulls Allow You to Conclude
A Fisher test of the sharp null addresses whether there is evidence against no effect for every participant. Rejecting it does not, by itself, establish that the average effect is nonzero: heterogeneous effects can average to zero. Conversely, failure to reject the sharp null is not proof that every participant’s effect is zero.
A test aimed at the weak null asks whether the average effect is zero, allowing individual effects to vary. Its reference distribution and error guarantees must be justified for that weaker hypothesis. The ordinary sharp-null imputation argument does not provide that justification. Other methods can target average effects, but their assumptions and calibration need to match the assignment design and the weak null. The distinction is particularly important under response adaptation, where unobserved outcomes can affect both a statistic and the probabilities of later assignments.
| Feature | Sharp null | Weak average-effect null |
|---|---|---|
| Restriction | Every participant’s effect is zero | The average of participant effects is zero |
| Effect heterogeneity allowed? | No | Yes; positive and negative effects may cancel |
| Missing outcomes under alternative assignments | Imputed as the observed outcomes | Not determined by the null |
| Ordinary Fisher reference distribution | Justified by the sharp-null imputation | Not generally an exact test of this null |
Common Mistakes and Diagnostics
- Equating zero average with no individual effects. The reasoning “the effects average to zero, so everyone has no effect” is false. The correction is to calculate or reason about individual effects: nonzero values can cancel in the average.
- Permuting observed outcomes as though they were fixed under every assignment. This is justified by the sharp null, not by a weak null. Under the weak null, ask which unobserved potential outcomes the proposed reference distribution needs and whether the null actually specifies them.
- Calling an adaptive replay exact without checking its inputs. Confirm that the assignment probabilities are evaluated from the correct histories and that the outcomes needed to construct those histories are justified under the null being tested.
- Interpreting rejection too broadly. Rejection of a sharp null is evidence against zero effects for all participants; it is not automatically a test conclusion about the sign or size of the average effect.
The practical diagnostic is simple: write down the null restriction, then ask whether it determines each participant’s outcome under every candidate assignment path. If not, the sharp-null randomization argument cannot be transferred unchanged to that hypothesis.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- For three participants, the individual effects are 5, \(-2\), and \(-3\). Which of the two null hypotheses holds?
- Explain why the sharp null implies the weak null, but the weak null does not imply the sharp null.
- A response-adaptive rule assigns the next participant using the previous participant’s observed response. Under which null can the previous participant’s response under the other arm be imputed directly from the observed response?
- Two candidate completions of missing potential outcomes agree with all observed data but have different average effects. What does this demonstrate about the observed data?
Solutions. (1) The effects sum to \(5-2-3=0\), so the weak null holds. The sharp null fails because none of the three effects is zero. (2) If every effect is zero, their average is zero. But an average of zero can result from nonzero effects that cancel, as \(5-2-3=0\) demonstrates. (3) The sharp null: it requires each participant’s two potential outcomes to be equal. The weak null specifies only the average and does not determine that participant’s missing response. (4) The observed data do not identify the missing potential outcomes or, without further assumptions, the average treatment effect. The two completions cannot both establish a unique average effect from the observed data alone.