Why Weight by the Assignment Probability?
In a response-adaptive trial, the chance of assigning treatment can change as the trial history changes. A participant assigned treatment with probability \(0.8\) and a participant assigned treatment with probability \(0.2\) do not represent the treated group in the same way. An unweighted difference between the observed treatment and control means can therefore give disproportionate influence to participants who were more likely to receive their observed assignment. The Horvitz–Thompson contrast corrects this imbalance by weighting each observed outcome by the inverse probability of the assignment that revealed it.
The key result is a statement about the randomization procedure, not a guarantee that one realized estimate will be close to the true effect. When the sequential probabilities are known and positive, the inverse-probability-weighted contrast is unbiased for the finite-population average treatment effect, even when the probabilities depend on earlier assignments and observed outcomes.
Finite-Population Target and Assignment Probabilities
Consider a fixed population of \(N\) trial participants, indexed in assignment order by \(i=1,\ldots,N\). For each participant, let \(Y_i(1)\) be the outcome that would be observed under treatment and \(Y_i(0)\) the outcome that would be observed under control. These potential outcomes are fixed for the design-based argument. Assume consistency and no interference: the observed outcome is the potential outcome corresponding to the participant’s assigned arm, and one participant’s assignment does not change another participant’s potential outcomes.
Let \(A_i=1\) indicate treatment assignment and \(A_i=0\) indicate control assignment. Write \(H_{i-1}\) for the observed trial history available immediately before participant \(i\) is assigned. The assignment mechanism supplies the sequential probability \(e_i(H_{i-1})=P(A_i=1\mid H_{i-1})\). This probability may vary across participants and histories. For compactness, write \(e_i=e_i(H_{i-1})\) for the probability at the history actually reached.
The \(e_i\) are conditional probabilities for the current assignment given the observed past. They are not generally the same as probabilities calculated without conditioning on the history. The estimator below uses the probability specified by the actual assignment algorithm at each step; it does not require assignments to be independent or to have a fixed treatment proportion.
The Sequential Horvitz–Thompson Contrast
The observed outcome is \(Y_i^{\mathrm{obs}}=A_iY_i(1)+(1-A_i)Y_i(0)\). A treated participant contributes their observed outcome divided by the probability of treatment; a control participant contributes the negative of their observed outcome divided by the probability of control. The estimator is
Equivalently, because only the outcome in the assigned arm is observed, the \(i\)th summand is \(Y_i(1)/e_i\) when \(A_i=1\), and \(-Y_i(0)/(1-e_i)\) when \(A_i=0\). Thus each participant contributes a weighted term whether assigned treatment or control. The estimator is the average of those signed contributions.
For this formula to be defined and to estimate effects for all participants, assume positivity at every history the design can reach: \(0<e_i(H_{i-1})<1\). Both assignments must have a positive chance for each participant. In practice, it is also important that the probabilities used in the calculation are the probabilities the algorithm actually used, including any constraints or changes made during the trial.
Why the Estimator Is Design-Unbiased
Design-based expectation averages over the random assignments generated by the trial algorithm while holding the finite population’s potential outcomes fixed. At each step, condition on the history just before assignment. Given that history, \(e_i\) is known, and the current assignment is a random draw with treatment probability \(e_i\). Therefore,
The argument applies at every history that can precede participant \(i\), including histories created by response-adaptive rules. Taking expectations over those histories preserves the equality: the expected \(i\)th contribution is \(Y_i(1)-Y_i(0)\). Adding the contributions and dividing by \(N\) gives the result.
This is unbiasedness over repeated trials using the same design and finite population. It does not say that the estimate from a single allocation equals the target, that its sampling variability is small, or that the probability the target lies in a particular interval has been established. Those questions require additional uncertainty analysis.
Worked Examples
Worked Example: Calculating an Estimate From One Adaptive Allocation
Consider a hypothetical finite population of four participants. The table gives each participant’s fixed potential outcomes, the assignment probability used at the history reached, the observed assignment, and the observed outcome. The potential outcomes are shown here to make the true finite-population effect checkable; in an actual trial, the unobserved potential outcomes would not be available.
| Participant | \(Y_i(1)\) | \(Y_i(0)\) | \(e_i\) | \(A_i\) | \(Y_i^{mathrm{obs}}\) |
|---|---|---|---|---|---|
| 1 | 8 | 6 | 0.50 | 1 | 8 |
| 2 | 7 | 5 | 0.60 | 0 | 5 |
| 3 | 10 | 7 | 0.75 | 1 | 10 |
| 4 | 9 | 4 | 0.40 | 0 | 4 |
All probabilities are between zero and one, and each observed outcome matches the potential outcome for the assigned arm. Substituting the observed data and probabilities gives
The known finite-population effect for this hypothetical example is \(\{(8-6)+(7-5)+(10-7)+(9-4)\}/4=3.000\) outcome units. The estimate \(2.542\) does not equal \(3.000\) in this allocation. That difference is compatible with design-unbiasedness: the theorem concerns the average over allocations generated by the design, not exact recovery in every allocation.
Worked Example: Verifying Unbiasedness Across Adaptive Paths
Take a hypothetical two-participant trial with fixed potential outcomes \(Y_1(1)=10\), \(Y_1(0)=6\), \(Y_2(1)=8\), and \(Y_2(0)=4\). Participant 1 has treatment probability \(0.5\). Participant 2 has treatment probability \(0.8\) if \(A_1=1\), and \(0.2\) if \(A_1=0\). Both probabilities are positive, so either assignment remains possible for participant 2 at either history. The true average effect is \(\{(10-6)+(8-4)\}/2=4\).
The four assignment paths and their probabilities follow by multiplying the conditional assignment probabilities. Applying the estimator on each path gives:
| Path \((A_1,A_2)\) | Path probability | HT estimate |
|---|---|---|
| (1, 1) | \(0.5(0.8)=0.40\) | \(\{10/0.5+8/0.8\}/2=15\) |
| (1, 0) | \(0.5(0.2)=0.10\) | \(\{10/0.5-4/0.2\}/2=0\) |
| (0, 1) | \(0.5(0.2)=0.10\) | \(\{-6/0.5+8/0.2\}/2=14\) |
| (0, 0) | \(0.5(0.8)=0.40\) | \(\{-6/0.5-4/0.8\}/2=-8.5\) |
The path probabilities sum to \(0.40+0.10+0.10+0.40=1\). The design expectation of the estimate is
The expected estimate equals the finite-population average effect. The path estimates vary widely, but their probability-weighted average is correct. The example also shows why adaptivity itself does not invalidate the unbiasedness argument: participant 2’s probability changes with the first assignment, while the inverse probability used on each path is the one specified for that path.
Worked Example: A Rare Assignment Can Produce an Extreme Estimate
In a hypothetical trial of three participants, suppose the assignment probabilities are \(e_1=0.05\), \(e_2=0.50\), and \(e_3=0.50\). The observed assignments and outcomes are: participant 1 is treated with outcome \(20\); participant 2 is controlled with outcome \(6\); participant 3 is treated with outcome \(9\). These probabilities are all positive, and assume the probabilities recorded are the ones actually used by the design. The estimate is
outcome units. If the fixed potential outcomes were \((20,18)\), \((7,6)\), and \((9,8)\) for participants 1, 2, and 3, respectively, the finite-population average effect would instead be \(\{2+1+1\}/3=1.333\) outcome units. The large realized estimate arises because the treated outcome for participant 1 is multiplied by \(1/0.05=20\). The example illustrates a limitation, not a contradiction: a rare assignment can receive a large inverse-probability weight, so unbiasedness alone does not ensure a stable estimate in a particular trial.
Checks, Alternatives, and Limitations
A practical analysis needs an auditable record of the probability used at every assignment. Check that each probability corresponds to the actual history and algorithm, that treatment and control probabilities are both positive, and that observed assignments and outcomes are coded consistently. Very small values of \(e_i\) or \(1-e_i\) are warning signs: their reciprocals can make a small number of observations dominate the contrast. Also inspect for missing outcomes or departures from the planned assignment mechanism. The stated result assumes the relevant outcomes are observed and the probabilities used in the estimator are correct.
The ordinary difference in observed arm means is often easier to report, but it does not in general give each participant the inverse-probability correction required here. Under unequal, history-dependent assignment, participants’ chances of appearing in each arm differ; an unweighted mean contrast is not generally design-unbiased for \(\tau_N\). A normalized, or Hájek-type, contrast divides weighted outcomes by the sum of weights within each arm rather than by the fixed population size \(N\). Normalization can reduce sensitivity to overall weight variation, but its random denominators mean it is not generally exactly design-unbiased in finite samples. Regression adjustment is another possible approach, but its properties depend on the adjustment and its assumptions; it is not a substitute for correctly reconstructing the randomization probabilities.
There is a trade-off. The Horvitz–Thompson contrast has a direct finite-population unbiasedness result, but extreme weights can produce high variability. Truncating or capping weights may reduce the influence of extreme contributions, yet generally changes the estimator and removes the exact unbiasedness guarantee. Report such a change as a different estimator, not as an innocuous numerical fix. Finally, unbiasedness supplies neither a variance estimate nor a confidence interval. It is a property of the estimator’s design expectation, not a complete uncertainty analysis.
Common Mistake
Check the Logic Before Reporting
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A participant is assigned control with observed outcome \(3\) and treatment probability \(e_i=0.25\). What is the participant’s signed Horvitz–Thompson contribution before division by \(N\)?
- In one sentence, state what randomness the design-based expectation averages over and what it holds fixed.
- Why does a sequential probability depending on earlier outcomes not, by itself, invalidate the unbiasedness derivation?
- What positivity condition is needed, and what practical problem can arise even when it holds?
- Why does replacing the fixed denominator \(N\) with the sum of observed inverse-probability weights generally change the exact unbiasedness claim?
Solutions. 1. The control probability is \(1-0.25=0.75\), so the signed contribution is \(-3/0.75=-4\). 2. It averages over assignments generated by the trial’s randomization mechanism while holding the finite population’s potential outcomes fixed. 3. Conditional on the history immediately before the current assignment, its actual treatment probability is known and the current assignment is randomized with that probability; the conditional expectation of the weighted contribution is still \(Y_i(1)-Y_i(0)\). 4. At every reachable history, \(0<e_i<1\); even then, probabilities close to zero or one can create very large weights and unstable realized estimates. 5. The sum of weights is random, so taking an expectation of a ratio does not generally give the ratio of expectations; the normalized estimator therefore need not retain exact finite-sample design-unbiasedness.