Design-Based Variance From Joint Assignment Probabilities

Why Joint Assignment Probabilities Enter the Variance

A Horvitz–Thompson contrast weights each observed outcome by the inverse probability of its assignment. These weights can make the contrast unbiased for a finite-population treatment effect, but they do not by themselves determine its uncertainty. The design variance also depends on how assignments to different units co-vary. In a response-adaptive trial, that dependence is part of the assignment mechanism: later probabilities may depend on earlier trial history, and the relevant joint probabilities average over the complete randomization procedure.

The goal here is to express the variance in terms of fixed potential outcomes and first- and second-order assignment probabilities, then ask a separate question: can the outcome products in that expression be estimated from the realized trial? An exact variance formula and an estimable variance are not always the same thing.

Finite-Population Setup and Assumptions

Consider a fixed finite population of \(N\) units. Unit \(i\) has fixed potential outcomes \(Y_i(1)\) and \(Y_i(0)\), representing its outcome under treatment and control. Let \(Z_i=1\) indicate treatment and \(Z_i=0\) indicate control. The observed outcome is \(Y_i=Z_iY_i(1)+(1-Z_i)Y_i(0)\). The finite-population average treatment effect is \(\tau=N^{-1}\sum_i\{Y_i(1)-Y_i(0)\}\).

Probabilities and expectations below are taken over the trial’s random assignment mechanism, holding all potential outcomes fixed. Write \(\pi_i=P(Z_i=1)\), \(q_i=P(Z_i=0)=1-\pi_i\), and \(\pi_{ij}=P(Z_i=1,Z_j=1)\). Define \(p_i^1=\pi_i\), \(p_i^0=q_i\), and \(p_{ij}^{ab}=P(Z_i=a,Z_j=b)\), where \(a,b\in\{0,1\}\). These are marginal probabilities over the full design, not merely the probability conditional on one observed history. For example, \(p_{ij}^{10}=\pi_i-\pi_{ij}\) for distinct units.

Assumptions: The assignment mechanism is specified and its probabilities refer to the randomization actually used. Each unit has well-defined fixed potential outcomes, and the observed outcome equals the potential outcome for its assigned arm (consistency). For the inverse-probability-weighted contrast to be defined, \(0<\pi_i<1\) for every unit. No independence of assignments is assumed.

The estimator is

\hat{\tau}_{HT}=\frac{1}{N}\sum_{i=1}^{N}\left\{\frac{Z_iY_i(1)}{\pi_i}-\frac{(1-Z_i)Y_i(0)}{q_i}\right\}.

For each unit, the expected treated contribution is \(Y_i(1)\), since \(E(Z_i)/\pi_i=1\); the expected control contribution is \(Y_i(0)\), since \(E(1-Z_i)/q_i=1\). Thus \(E(\hat{\tau}_{HT})=\tau\). This unbiasedness statement requires the stated marginal probabilities to match the actual assignment design.

Variance Expansion

Let \(A_i^1=Z_i\) and \(A_i^0=1-Z_i\) be the assignment indicators for the two arms, and let \(s_1=1\) and \(s_0=-1\). The estimator is a sum of weighted assignment indicators. For arm labels \(a,b\), define their assignment-indicator covariance by

C_{ij}^{ab} =\operatorname{Cov}(A_i^a,A_j^b) =p_{ij}^{ab}-p_i^a p_j^b.

The following identity is obtained by expanding the variance of the sum: the covariance between each pair of weighted terms is the product of their fixed outcomes and weights, multiplied by the covariance of their assignment indicators.

Variance formula: Under the setup above, the exact design-based variance is

\operatorname{Var}(\hat{\tau}_{HT}) = \frac{1}{N^2} \sum_{i=1}^{N}\sum_{j=1}^{N} \sum_{a=0}^{1}\sum_{b=0}^{1} s_as_b \frac{Y_i(a)}{p_i^a} \frac{Y_j(b)}{p_j^b} C_{ij}^{ab}.

This expression includes \(i=j\): in that case \(p_{ii}^{11}=\pi_i\), \(p_{ii}^{00}=q_i\), and \(p_{ii}^{10}=p_{ii}^{01}=0\). For distinct units, all four joint assignment probabilities can be obtained from the first- and second-order probabilities. In particular, \(p_{ij}^{11}=\pi_{ij}\), \(p_{ij}^{10}=\pi_i-\pi_{ij}\), \(p_{ij}^{01}=\pi_j-\pi_{ij}\), and \(p_{ij}^{00}=1-\pi_i-\pi_j+\pi_{ij}\).

A useful equivalent form replaces each covariance by \(C_{ij}^{ab}/(p_i^a p_j^b)=p_{ij}^{ab}/(p_i^a p_j^b)-1\) and uses the unweighted product \(Y_i(a)Y_j(b)\). The covariance form above makes the weighting explicit; the equivalent form makes clear that the coefficient of each potential-outcome product is determined by the joint inclusion probability and the marginals.

Worked Examples

Worked Example: Checking the Variance by Enumerating a Two-Unit Design

Suppose, hypothetically, that \(N=2\), with fixed potential outcomes \(Y_1(1)=8\), \(Y_1(0)=2\), \(Y_2(1)=5\), and \(Y_2(0)=1\), all in outcome points. The design assigns exactly one unit to treatment, choosing either unit with probability \(0.5\). Therefore \(\pi_1=\pi_2=q_1=q_2=0.5\), and \(\pi_{12}=0\).

The two possible assignment paths give \(\hat{\tau}_{HT}=7\) when unit 1 is treated and \(\hat{\tau}_{HT}=3\) when unit 2 is treated:

\hat{\tau}_{HT}(1,0)=\frac{1}{2}\left(\frac{8}{0.5}-\frac{1}{0.5}\right)=7, \qquad \hat{\tau}_{HT}(0,1)=\frac{1}{2}\left(-\frac{2}{0.5}+\frac{5}{0.5}\right)=3.

The finite-population effect is \(\tau=\{(8-2)+(5-1)\}/2=5\), which is also the estimator’s design expectation. Direct enumeration gives variance \(0.5(7-5)^2+0.5(3-5)^2=4\) points squared.

Now isolate the ordered treatment-treatment term for \((i,j)=(1,2)\). Here \(C_{12}^{11}=\pi_{12}-\pi_1\pi_2=0-0.25=-0.25\). Its contribution is

N^{-2}C_{12}^{11}\frac{Y_1(1)}{\pi_1}\frac{Y_2(1)}{\pi_2} = \frac{1}{4}(-0.25)(16)(10)=-10.

The reverse-order term \((2,1)\) is also \(-10\), so the two ordered treatment-treatment terms contribute \(-20\) points squared. This is not the full variance: the diagonal and control and mixed-arm terms together contribute \(24\), leaving the enumerated total \(4\). The example illustrates why an individual component can be negative even though the complete variance cannot be.

Worked Example: Estimating a Pair Component When Joint Positivity Holds

Consider a hypothetical three-unit design with \(\pi_1=0.6\), \(\pi_2=0.5\), and \(\pi_{12}=0.4\). Suppose the fixed treatment potential outcomes for units 1 and 2 are \(Y_1(1)=3\) and \(Y_2(1)=4\) points. The ordered treatment-treatment component for \((1,2)\) is

\frac{1}{N^2}C_{12}^{11} \frac{Y_1(1)}{\pi_1}\frac{Y_2(1)}{\pi_2}, \qquad C_{12}^{11}=\pi_{12}-\pi_1\pi_2=0.4-(0.6)(0.5)=0.1.

Substitution gives \(\frac{1}{9}(0.1)(3/0.6)(4/0.5)=0.4444\) points squared. Because \(\pi_{12}=0.4>0\), both treatment outcomes can be observed together with positive probability. An unbiased estimator of this one component replaces the unknown product by its joint inverse-probability estimate:

\widehat{V}_{12}^{11} = \frac{1}{N^2}C_{12}^{11} \frac{\mathbf{1}(Z_1=1,Z_2=1)}{\pi_{12}} \frac{Y_1Y_2}{\pi_1\pi_2}.

When both units are treated, this estimate is \(\frac{1}{9}(0.1)(1/0.4)(3\cdot4)/(0.6\cdot0.5)=1.111\) points squared; otherwise it is zero. Its design expectation is \(0.4(1.111)=0.4444\), the component just calculated. This estimates one term, not automatically the entire variance, which also includes other pairs and arm combinations.

Worked Example: A Zero Joint Probability Blocks Product Estimation

In the two-unit design above, both units have positive marginal treatment probabilities, but \(\pi_{12}=0\): they are never treated together. The treatment-treatment covariance is \(C_{12}^{11}=-0.25\), so its ordered variance component depends on \(Y_1(1)Y_2(1)\). No realized assignment reveals that product, because the design never observes both treatment outcomes in the same trial.

An inverse-probability estimate of the product would require division by \(\pi_{12}\), which is division by zero. The exact component can still be written when the finite-population potential outcomes are specified, as in the previous example, but it is not generally estimable from observed outcomes under this design. Positive marginal probabilities alone do not guarantee that pairwise outcome products are observable.

Which Terms Are Estimable?

For distinct units and arm labels \(a,b\), a term involving \(Y_i(a)Y_j(b)\) can generally be estimated without bias from observed outcomes if \(p_{ij}^{ab}>0\). The indicator \(\mathbf{1}(Z_i=a,Z_j=b)\), divided by \(p_{ij}^{ab}\), has expectation one, so multiplying the observed product by that factor recovers the fixed product in expectation. This is the pairwise version of inverse-probability weighting.

If \(p_{ij}^{ab}=0\), the corresponding product is never observed jointly. The exact variance formula still has a coefficient for that product: in the equivalent form, \(p_{ij}^{ab}/(p_i^a p_j^b)-1=-1\). Unless the coefficient vanishes through another feature of the design or additional outcome information is available, that product cannot be estimated generally from the trial data. For a single unit, the two potential outcomes are mutually exclusive observations, so \(p_{ii}^{10}=p_{ii}^{01}=0\); the cross-potential-outcome product is not observed in any assignment.

Some terms do not require product estimation. If \(C_{ij}^{ab}=0\), that term contributes zero to the variance regardless of the potential outcomes. Also, special known restrictions on the potential outcomes can make an otherwise unavailable product recoverable. Without such a restriction, however, an exact algebraic variance expression is not a promise that every component has an unbiased data-based estimator.

Practical Checks and Limitations

Verify the design probabilities.
Check that the marginal and joint probabilities are calculated under the actual adaptive algorithm, integrating over possible histories as required. Confirm that the joint probabilities are compatible with the marginals.

Check marginal positivity.
Every arm used in the contrast requires positive marginal probability for each unit; otherwise the corresponding inverse-probability weight is undefined.

Check pairwise positivity term by term.
For an outcome product to be estimated by a pairwise inverse-probability method, the associated joint arm probability must be positive. Inspect zero or extremely small probabilities, not just the marginal assignment probabilities.

Check the complete sum.
Individual covariance components can be negative. The variance is the sum over all units, arms, and ordered pairs; do not interpret a single component as a variance or discard negative components.

These checks matter especially under adaptive allocation, where treating assignments as independent can give incorrect second-order probabilities and hence an incorrect variance. Exact design-based calculation uses the assignment mechanism and fixed potential outcomes; a model-based standard error instead adds outcome-model assumptions and should not be described as the same calculation. Repeatedly simulating the assignment algorithm can approximate design quantities when the mechanism is known, but it cannot reveal unobserved potential outcomes by itself. Nonestimable products and very small joint probabilities therefore remain genuine limitations, not computational inconveniences.

Common Mistake

Wrong reasoning: “Both units have positive treatment probabilities, so the product of their treatment outcomes is estimable.” Positive marginal probabilities only imply that each outcome could be observed separately. Correction: Check the joint probability \(p_{ij}^{11}\). If it is zero, the outcomes are never jointly observed; the pair product is not generally estimable from the trial, even though the corresponding exact variance term is well-defined for specified potential outcomes.

Key takeaway: First-order probabilities determine the inverse weights; second-order probabilities determine assignment covariances. The resulting finite-population variance is exact under the design, but estimating every term also requires the relevant joint assignment events to have positive probability.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If \(p_{ij}^{11}=\pi_i\pi_j\), what is \(C_{ij}^{11}\), and what does this imply for that treatment-treatment variance component?
For \(\pi_i=0.4\), \(\pi_j=0.5\), and \(\pi_{ij}=0.3\), calculate \(C_{ij}^{11}\).
Why can a positive marginal probability for each unit fail to make a pairwise outcome product estimable?
For one unit, what is \(p_{ii}^{10}\), and what does this imply about observing both potential outcomes?

Solutions. (1) \(C_{ij}^{11}=p_{ij}^{11}-\pi_i\pi_j=0\), so this component is zero. (2) \(C_{ij}^{11}=0.3-(0.4)(0.5)=0.10\). (3) The units may never receive the specified pair of assignments together; estimation of the product requires a positive joint probability for that event. (4) \(p_{ii}^{10}=P(Z_i=1,Z_i=0)=0\); a unit cannot be assigned to both arms in the same trial, so both potential outcomes are not jointly observed.