Why Adaptive Assignment Changes Estimation
In an adaptively randomized trial, the probability of receiving treatment can change as the trial proceeds. A participant assigned when treatment is unlikely to be allocated may therefore represent more of the trial cohort, for estimation, than a participant assigned when treatment is common. An unweighted average treats each observed outcome as equally representative, even when the assignment chances differ substantially. A design-weighted estimate uses the known probabilities to account for that difference.
The goal here is a treatment-effect estimate for a fixed cohort of \(n\) participants eligible for either treatment or control. The outcome \(Y_i\) is binary, with 1 denoting the event and 0 its absence. Let \(Y_i(1)\) and \(Y_i(0)\) denote participant \(i\)'s potential outcomes under treatment and control. Only one is observed, so \(Y_i=A_iY_i(1)+(1-A_i)Y_i(0)\), where \(A_i=1\) indicates treatment and \(A_i=0\) control.
This estimand averages the two potential outcomes over the same cohort. It is not necessarily the difference between the outcomes of the participants who happened to receive each assignment. In a sequential design, define \(H_i\) as the information available immediately before participant \(i\)'s randomization, and let \(p_i=P(A_i=1\mid H_i)\) be the treatment assignment probability specified by the randomization rule at that point.
Assumptions and the Inverse-Probability Estimate
The probabilities \(p_i\) must be the actual probabilities used by the assignment algorithm, conditional on the information available before that assignment. They may vary with \(i\) and with the observed history. The estimator also requires sequential randomization: conditional on \(H_i\), assignment is randomized with probability \(p_i\), and assignment does not use the participant's unobserved current potential outcomes. The participant must receive the assigned intervention as defined, and one participant's treatment must not change another participant's outcome. These are consistency and no-interference assumptions.
For this two-arm comparison, positivity requires \(0<p_i<1\) for every participant in the target cohort. If an arm has zero assignment probability at a point in the trial, the unobserved potential outcome for that participant cannot be recovered by weighting. Very small probabilities do not make the formula undefined, but can make estimates highly variable.
The denominator is the total cohort size \(n\), not the observed number assigned to that arm. Each observed outcome is expanded by the inverse of its chance of being observed under that assignment. The estimates target average potential outcomes for the entire fixed cohort.
The justification is a conditional expectation calculation. Given \(H_i\) and the potential outcomes, \(E(A_i\mid H_i)=p_i\). Therefore
The corresponding control term has conditional expectation \(Y_i(0)\), because \(E(1-A_i\mid H_i)=1-p_i\). Averaging over participants and using linearity of expectation gives \(E(\widehat{\mu}_1^{HT})=\mu_1\), \(E(\widehat{\mu}_0^{HT})=\mu_0\), and \(E(\widehat{\tau}^{HT})=\tau\), over the randomization process for the fixed cohort. This is a design-unbiasedness result; it does not say that one realized estimate must be close to its target.
Worked Example: A Small Binary-Outcome Trial
Consider eight hypothetical participants. The table gives each participant's treatment probability just before assignment, observed assignment, and observed binary outcome. Assume the probabilities shown were recorded correctly from the randomization algorithm and were strictly between 0 and 1.
| Participant | \(p_i\) | \(A_i\) | \(Y_i\) |
|---|---|---|---|
| 1 | 0.50 | 1 | 1 |
| 2 | 0.75 | 1 | 1 |
| 3 | 0.50 | 1 | 0 |
| 4 | 0.25 | 1 | 1 |
| 5 | 0.50 | 0 | 0 |
| 6 | 0.50 | 0 | 1 |
| 7 | 0.50 | 0 | 0 |
| 8 | 0.50 | 0 | 1 |
For treatment, the four observed contributions \(A_iY_i/p_i\) are \(2\), \(4/3\), \(0\), and \(4\). Thus
For control, the two observed events each have weight \(1/(1-0.50)=2\), and the other two contributions are zero. Hence
The estimated treatment risk exceeds the estimated control risk by about 41.7 percentage points for this hypothetical cohort. The calculation does not establish a causal effect without the stated design and outcome assumptions, nor does it quantify uncertainty.
What the Weights Do—and Do Not Do
Inverse-probability weighting compensates for unequal assignment chances; it does not create additional information. When a participant receives treatment despite a low treatment probability, that participant has a large treatment weight \(1/p_i\). A single event with a large weight can move an estimate sharply. Moreover, a Horvitz–Thompson risk estimate can fall below 0 or exceed 1 in a small sample. This does not contradict its design-unbiasedness: unbiasedness concerns an average over possible randomization paths, not the range of every realized estimate.
A common alternative is a normalized, or Hájek, estimate. It divides the weighted outcome total by the total weight observed in that arm:
These ratios often stay within the range from 0 to 1 and can be more stable in practice, but they are not generally exactly design-unbiased for the finite-cohort risks. Their denominators are random. The choice between Horvitz–Thompson and Hájek estimation is therefore a choice of estimator and properties, not a reason to ignore the assignment probabilities.
Worked Example: A Sparse High-Probability Allocation
In a hypothetical four-person cohort, two participants are assigned treatment with \(p_i=0.90\), and each has outcome 1. Two are assigned control with \(p_i=0.10\), and each has outcome 0. All assignment probabilities satisfy positivity.
The treatment contributions are \(1/0.90\) for each observed treated event, so
Both observed control outcomes are zero, giving \(\widehat{\mu}_0^{HT}=0\). The estimated risk difference is therefore \(5/9\approx 0.5556\), or about 55.6 percentage points. The observed treated participants all had the event, but the estimate is not 1: those two participants were each very likely to be assigned treatment and so each represents less of the full cohort than a treated participant assigned with a low probability would. This small example illustrates the arithmetic, not a claim that the estimate is precise.
Worked Example: Extreme Weights and a Risk Estimate Above One
Consider six hypothetical participants. The first three have \(p_i=0.80\), and the last three have \(p_i=0.20\). The realized assignments are treatment, control, control, control, control, treatment, respectively; observed outcomes are 1, 1, 0, 0, 0, 1. The assignment probabilities are positive for both arms at every point.
The treatment-weighted event total is \(1/0.80+1/0.20=1.25+5=6.25\). Dividing by all six participants gives \(\widehat{\mu}_1^{HT}=6.25/6=25/24\approx1.0417\). For control, only participant 2 has an event. That participant's control probability is \(1-0.80=0.20\), so the weighted control event total is \(1/0.20=5\), and \(\widehat{\mu}_0^{HT}=5/6\approx0.8333\). Thus
The treatment risk estimate is above 1 because the single treated event observed at probability 0.20 receives weight 5. It should not be silently clipped to 1: clipping changes the estimator and its properties. In this same example, the Hájek treatment estimate is \(6.25/6.25=1\). The control weight total is \(5+5+1.25+1.25=12.5\), so its Hájek estimate is \(5/12.5=0.4000\). The Hájek risk difference is \(0.6000\). The marked difference between the two estimates is a warning about how little information this realized sample provides, not evidence that one calculation is erroneous.
Diagnostics, Alternatives, and Limits
Before interpreting a weighted estimate, verify that each \(p_i\) is the probability actually used at that participant's assignment—not a later estimate of the probability, nor an observed fraction assigned to treatment. Check the time ordering and probability record against the randomization algorithm. Then inspect the smallest assignment probabilities, the largest inverse weights, and whether a few observations dominate each arm's weighted total. One descriptive diagnostic is the effective sample size for weights \(w_i\):
Calculate it separately for treatment weights \(A_i/p_i\) and control weights \((1-A_i)/(1-p_i)\). It is a diagnostic of weight concentration, not a replacement sample size or a complete uncertainty calculation. Extreme weights or a very small effective sample size signal that estimates may be unstable. If an arm's probability is zero for part of the target cohort, positivity fails and weighting alone cannot estimate that cohort's risk without additional assumptions.
Crude arm-specific event proportions are easy to calculate, but under changing assignment probabilities they do not generally give a design-unbiased estimate of the same fixed-cohort risks. They can be reasonable summaries in some settings, but should not automatically be interpreted as the finite-cohort effect. Outcome-regression or Bayesian outcome models provide alternatives that can use prognostic information, but their estimates depend on model assumptions; they do not make the assignment record irrelevant. A design-weighted point estimate also is not a posterior distribution or a complete uncertainty analysis.
For uncertainty assessment, the adaptive assignment process matters. Since future probabilities can depend on earlier trial information, treating the observed weighted contributions as if they were a simple independent sample can give inappropriate standard errors. Use an uncertainty method justified for the actual sequential design and estimand, and report the probability history and positivity checks. If the target cohort, eligibility rules, or adaptation changes during the trial, state clearly which participants the estimate represents.
Common Mistake
The key practical trade-off is between respecting the randomization design and controlling instability. Horvitz–Thompson estimation directly targets the finite-cohort risks without a random weight-sum denominator, but can be noisy and can yield risks outside the probability range. Hájek normalization often reduces that instability but is not exactly unbiased in finite samples. Neither choice repairs zero assignment probabilities, an incorrect probability record, or violations of the outcome assumptions.
Check Your Understanding
Answer each question, then compare with the solutions below.
- State the finite-cohort risk difference and the positivity condition needed for the two-arm weighted estimate.
- For a participant assigned treatment with \(p_i=0.25\) and observed outcome \(Y_i=1\), what is the treatment contribution to the weighted total?
- With \(n=4\), two treated participants each have \(p_i=0.50\) and outcome 1; the other two are controls with outcome 0 and \(p_i=0.50\). Calculate both Horvitz–Thompson risks and the risk difference.
- Why can a Horvitz–Thompson risk estimate exceed 1 without implying an arithmetic error?
- What is one key distinction between Horvitz–Thompson and Hájek risk estimates?
Solutions. (1) The estimand is \(\tau=n^{-1}\sum_i[Y_i(1)-Y_i(0)]=\mu_1-\mu_0\); positivity requires \(0<p_i<1\) for every participant in the target cohort. (2) The contribution is \(A_iY_i/p_i=1/0.25=4\). (3) The treatment total is \(2+2=4\), so \(\widehat{\mu}_1^{HT}=4/4=1\); the control event total is 0, so \(\widehat{\mu}_0^{HT}=0\), and the estimated difference is 1, or 100 percentage points. (4) A low assignment probability can give an observed event a large inverse-probability weight; design-unbiasedness applies across randomization paths, not to the range of every realized estimate. (5) Horvitz–Thompson divides the weighted event total by the fixed cohort size \(n\); Hájek divides by the random total weight in that arm and is not generally exactly design-unbiased in finite samples.