Unequal Assignment Probabilities and Horvitz-Thompson Estimation

Why Unequal Assignment Probabilities Need Weights

In a randomized experiment, some units may be more likely than others to receive treatment. This can arise by design, for example when assignment probabilities depend on a prespecified risk group. If we simply average the outcomes observed in each arm, units with higher assignment probabilities may contribute more often to that average. The result need not represent the finite population of interest.

Horvitz–Thompson estimation corrects this unequal representation by weighting each observed potential outcome inversely to its probability of being observed. The estimator is a design-based one: the finite population and its potential outcomes are held fixed, and the randomness comes from the assignment mechanism.

Definition: For a finite population of \(N\) units, let \(Y_i(1)\) and \(Y_i(0)\) be unit \(i\)'s fixed potential outcomes under treatment and control. Let \(Z_i=1\) if the unit is assigned treatment and \(Z_i=0\) otherwise, with known assignment probability \(\pi_i=P(Z_i=1)\). The observed outcome is \(Y_i=Z_iY_i(1)+(1-Z_i)Y_i(0)\). The finite-population potential-outcome means are \(\bar{Y}(1)=N^{-1}\sum_{i=1}^N Y_i(1)\) and \(\bar{Y}(0)=N^{-1}\sum_{i=1}^N Y_i(0)\); their difference \(\tau_{FP}=\bar{Y}(1)-\bar{Y}(0)\) is the finite-population average treatment effect.

The potential-outcome framework and the role of random assignment are the standard starting points for causal inference. Here, the essential additional condition is that every unit has a positive, known chance of assignment to each arm: \(0<\pi_i<1\). Without this condition, one or both potential outcomes for a unit can never be observed under the design.

The Horvitz–Thompson Estimators

The treatment potential outcome \(Y_i(1)\) is observed only when \(Z_i=1\). Its indicator \(Z_i\), with expectation \(\pi_i\), therefore provides an inverse-probability weighted estimate of that fixed value: \(Z_iY_i(1)/\pi_i\). If the unit is assigned treatment, the value is scaled by \(1/\pi_i\); otherwise, its contribution is zero. Averaging these contributions over all \(N\) units gives the Horvitz–Thompson estimator of the treatment mean. The control mean uses the probability \(1-\pi_i\) of control assignment.

\widehat{\bar{Y}}_{\mathrm{HT}}(1) =\frac{1}{N}\sum_{i=1}^N\frac{Z_iY_i}{\pi_i}, \qquad \widehat{\bar{Y}}_{\mathrm{HT}}(0) =\frac{1}{N}\sum_{i=1}^N\frac{(1-Z_i)Y_i}{1-\pi_i}.

In these expressions, \(Y_i\) is the observed outcome. Whenever \(Z_i=1\), it equals \(Y_i(1)\), and whenever \(Z_i=0\), it equals \(Y_i(0)\). The estimator of the finite-population average treatment effect is the difference between the two estimated means:

\widehat{\tau}_{\mathrm{HT}} =\widehat{\bar{Y}}_{\mathrm{HT}}(1)-\widehat{\bar{Y}}_{\mathrm{HT}}(0) =\frac{1}{N}\sum_{i=1}^N \left\{ \frac{Z_iY_i}{\pi_i} -\frac{(1-Z_i)Y_i}{1-\pi_i} \right\}.

Unlike the difference-in-means estimator under complete randomization, this expression uses each unit's own assignment probability. If all \(\pi_i\) are equal and the number assigned to treatment is fixed, the Horvitz–Thompson means reduce to the corresponding observed-arm means. With unequal probabilities, inverse-probability weights are what adjust contributions back toward the target population.

Worked Example: Estimating Both Potential-Outcome Means

Consider three hypothetical units with treatment probabilities \((0.20,0.50,0.80)\). The observed assignments and outcomes are: unit 1 is treated with outcome 12, unit 2 is assigned control with outcome 6, and unit 3 is treated with outcome 9. Calculate the Horvitz–Thompson means and their difference.

For the treatment mean, the observed treated outcomes are weighted by their treatment probabilities. Unit 2 contributes zero to this estimate because its treatment outcome is unobserved:

\widehat{\bar{Y}}_{\mathrm{HT}}(1) =\frac{1}{3}\left(\frac{12}{0.20}+\frac{0}{0.50}+\frac{9}{0.80}\right) =\frac{60+0+11.25}{3} =23.75.

For the control mean, unit 2 contributes its observed outcome, weighted by its probability of control assignment, \(1-0.50=0.50\). The other units contribute zero:

\widehat{\bar{Y}}_{\mathrm{HT}}(0) =\frac{1}{3}\left(\frac{0}{0.80}+\frac{6}{0.50}+\frac{0}{0.20}\right) =\frac{12}{3} =4.

Thus \(\widehat{\tau}_{\mathrm{HT}}=23.75-4=19.75\) outcome units. This estimate is not the raw difference between the two observed outcomes' group averages: it reweights the observed outcomes to estimate means for all three units. With such a small sample, the estimate may be far from the true finite-population effect; unbiasedness refers to its average across repetitions of the assignment procedure, not to accuracy in every realized assignment.

Why the Estimator Is Unbiased

Unbiasedness follows directly from the known assignment probabilities. Conditional on the fixed potential outcomes, \(E(Z_i)=\pi_i\) and \(E(1-Z_i)=1-\pi_i\). Therefore, the expected weighted contribution for unit \(i\) to the treatment mean is \(Y_i(1)\), and the expected contribution to the control mean is \(Y_i(0)\).

E\left[\frac{Z_iY_i}{\pi_i}\right] =\frac{E(Z_i)Y_i(1)}{\pi_i} =Y_i(1), \qquad E\left[\frac{(1-Z_i)Y_i}{1-\pi_i}\right] =\frac{E(1-Z_i)Y_i(0)}{1-\pi_i} =Y_i(0).

Taking averages across the \(N\) fixed units establishes the result for each mean and, by subtraction, for the treatment effect. This argument does not require the assignment indicators for different units to be independent; it only uses the correct marginal assignment probability for each unit.

Theorem: Under an assignment design with known probabilities \(0<\pi_i<1\), and with the observed outcome equal to the potential outcome under the assigned treatment, the Horvitz–Thompson estimators satisfy \(E\{\widehat{\bar{Y}}_{\mathrm{HT}}(1)\}=\bar{Y}(1)\), \(E\{\widehat{\bar{Y}}_{\mathrm{HT}}(0)\}=\bar{Y}(0)\), and \(E(\widehat{\tau}_{\mathrm{HT}})=\tau_{FP}\). Expectations are over the random assignment, holding the potential outcomes fixed.

Worked Example: Checking Unbiasedness Directly

Suppose a hypothetical population contains two units with treatment potential outcomes \(Y_1(1)=8\) and \(Y_2(1)=4\), and treatment probabilities \(\pi_1=0.20\) and \(\pi_2=0.80\). The Horvitz–Thompson estimate of the treatment mean is

\widehat{\bar{Y}}_{\mathrm{HT}}(1) =\frac{1}{2}\left(\frac{Z_1(8)}{0.20}+\frac{Z_2(4)}{0.80}\right).

Taking its design expectation gives

E\{\widehat{\bar{Y}}_{\mathrm{HT}}(1)\} =\frac{1}{2}\left(\frac{0.20(8)}{0.20}+\frac{0.80(4)}{0.80}\right) =\frac{8+4}{2} =6.

The finite-population treatment mean is also \((8+4)/2=6\). A unit assigned with low probability receives a larger weight when observed, but that larger contribution occurs less often. Those two features balance in expectation.

Variance and the Cost of Unequal Probabilities

Inverse-probability weighting preserves unbiasedness, but it can make estimates variable. When \(\pi_i\) is small, observing unit \(i\) in the treatment arm is rare, and its treatment contribution is large if observed. The corresponding issue for the control mean occurs when \(1-\pi_i\) is small. Thus, probabilities close to zero or one can create unstable estimates even when all weights are technically defined.

For a design in which assignments are independent across units, the variance of each estimated mean has a simple form. The potential outcomes are treated as fixed, and the only randomness is in the assignment indicators:

\operatorname{Var}\{\widehat{\bar{Y}}_{\mathrm{HT}}(1)\} =\frac{1}{N^2}\sum_{i=1}^N\frac{1-\pi_i}{\pi_i}Y_i(1)^2, \qquad \operatorname{Var}\{\widehat{\bar{Y}}_{\mathrm{HT}}(0)\} =\frac{1}{N^2}\sum_{i=1}^N\frac{\pi_i}{1-\pi_i}Y_i(0)^2.

The treatment and control contributions for the same unit cannot both be observed: \(Z_i(1-Z_i)=0\). Under independent assignment across units, their covariance is \(-N^{-2}\sum_iY_i(1)Y_i(0)\). Consequently, the variance of their difference is

\operatorname{Var}(\widehat{\tau}_{\mathrm{HT}}) =\frac{1}{N^2}\sum_{i=1}^N \left\{ \frac{1-\pi_i}{\pi_i}Y_i(1)^2 +\frac{\pi_i}{1-\pi_i}Y_i(0)^2 +2Y_i(1)Y_i(0) \right\}.

This expression is specific to independent assignment across units. Under a dependent design, such as a design fixing the number treated, the variance also depends on the joint assignment probabilities for pairs of units. In general, if \(W_i=Z_iY_i(1)/\pi_i-(1-Z_i)Y_i(0)/(1-\pi_i)\), then the variance of the effect estimator is \(N^{-2}\sum_i\sum_j\operatorname{Cov}(W_i,W_j)\). Marginal probabilities alone do not generally determine all those covariances.

Worked Example: Comparing Variance at Two Assignment Designs

For a hypothetical two-unit population, let the potential outcomes be \(Y_1(1)=8\), \(Y_1(0)=2\), \(Y_2(1)=4\), and \(Y_2(0)=3\). Compare the variance of the Horvitz–Thompson effect estimator under independent assignments with probabilities \((0.20,0.80)\) against probabilities \((0.50,0.50)\). The estimator targets the same finite-population effect in either design, and the independence assumption is imposed for both calculations.

For probabilities \((0.20,0.80)\), the first unit's contribution inside the variance sum is

\frac{0.80}{0.20}(8^2)+\frac{0.20}{0.80}(2^2)+2(8)(2) =4(64)+0.25(4)+32 =289.

The second unit's contribution is

\frac{0.20}{0.80}(4^2)+\frac{0.80}{0.20}(3^2)+2(4)(3) =0.25(16)+4(9)+24 =64.

Since \(N^2=4\), the variance is \((289+64)/4=88.25\) squared outcome units. Under probabilities \((0.50,0.50)\), the contributions are \(1(64)+1(4)+32=100\) for unit 1 and \(1(16)+1(9)+24=49\) for unit 2. The variance is then \((100+49)/4=37.25\) squared outcome units. In this example, equal probabilities yield lower variance. Unequal probabilities do not always increase variance, however: their consequences depend on how assignment probabilities align with the potential outcomes and on the assignment design.

Interpreting Estimates and Avoiding Pitfalls

The Horvitz–Thompson estimate is a finite-population mean or effect in the same units as the outcome. Its unbiasedness is a repeated-assignment statement: if the same fixed population were assigned many times according to the same design, the average of the estimates would equal the target. It does not say that a single realized estimate equals the target, nor that the estimate has low variance.

A practical complication is that the variance depends on both potential outcomes for each unit, even though only one is observed. Variance estimation therefore requires care and information about the assignment design; the independent-assignment formula should not be applied automatically to a design with dependent assignments. In particular, known marginal probabilities suffice for the unbiasedness proof, but not in general for calculating the full variance.

Common mistake: “The treatment group’s raw average is an unbiased estimate of the population treatment mean, because treatment was randomized.” With unequal assignment probabilities, units were not equally likely to appear in that group, so its raw average need not target the finite-population mean. Use the Horvitz–Thompson weighted mean, and check that the probabilities used are the actual assignment probabilities. A second mistake is to treat a large inverse-probability weight as harmless: positivity ensures a defined estimator, but probabilities near zero or one can still produce high variance.

Specify the target.
State whether the goal is the finite-population treatment mean, control mean, or their difference.

Record the design probabilities.
For every unit, use its known treatment probability \(\pi_i\) and control probability \(1-\pi_i\).

Weight observed outcomes.
Use \(Z_iY_i/\pi_i\) for treatment and \((1-Z_i)Y_i/(1-\pi_i)\) for control, then average over all \(N\) units.

Match variance calculations to the design.
Use the independent-assignment variance only when assignments are independent; otherwise account for joint assignment probabilities.

Key takeaway: Horvitz–Thompson weighting uses each unit's inverse assignment probability to estimate finite-population potential-outcome means without bias under the stated assignment design. Unequal probabilities affect precision, so unbiasedness and variance must be considered separately.

Check Your Understanding

Answer each question using the finite-population and assignment-design perspective.

For a unit with treatment probability \(\pi_i=0.25\), what weight is applied to its observed outcome in the treatment-mean estimator, and what is its probability of contributing?
State the positivity condition needed to define both Horvitz–Thompson potential-outcome mean estimators.
Why does the unbiasedness argument not require assignments to different units to be independent?
Explain why a very small treatment probability can increase the variance of the treatment-mean estimator.
Why are unit-level assignment probabilities enough to establish unbiasedness but not necessarily enough to calculate variance?