Design-Based Variance Estimation for Clustered Assignment

Why the Assignment Mechanism Matters

When treatment is assigned to whole clusters, the assignment mechanism—not an assumed distribution for individual errors—can determine the uncertainty in a treatment-effect estimate. The relevant random variation comes from which clusters are assigned to treatment. This perspective is especially useful when the clusters are the units randomized and the set of clusters is viewed as a fixed finite population.

The calculation below concerns complete randomization: a fixed number of clusters is selected at random for treatment, and the remaining clusters receive control. It gives a design-based variance, meaning that potential outcomes are treated as fixed and probability is taken over the allowed assignments. This is distinct from a model-based variance that follows from assumptions about the distribution or covariance of errors.

Definition: There are $J$ clusters. Cluster $j$ has fixed potential outcomes $Y_j(1)$ and $Y_j(0)$, the outcomes it would have under treatment and control. Its treatment effect is $\tau_j=Y_j(1)-Y_j(0)$. Under complete randomization, exactly $m$ of the $J$ clusters are assigned treatment, with every such assignment equally likely. The cluster-average treatment effect is $\tau=J^{-1}\sum_{j=1}^J\tau_j$.

For now, each cluster contributes one outcome, such as its mean outcome. Thus the target $\tau$ weights clusters equally. Assignment is the source of randomness; the potential outcomes do not change across assignments. The observed outcome is $Y_j(1)$ if cluster $j$ is treated and $Y_j(0)$ otherwise. This requires consistency and no interference between clusters: a cluster's potential outcome under its assigned condition does not depend on the assignments of other clusters.

The Difference in Cluster Means

Let $Z_j=1$ indicate that cluster $j$ is assigned treatment and $Z_j=0$ indicate control. The difference in observed cluster means is

\hat{\tau} = \frac{1}{m}\sum_{j=1}^J Z_jY_j(1) - \frac{1}{J-m}\sum_{j=1}^J(1-Z_j)Y_j(0).

Under complete randomization, each cluster has probability $m/J$ of treatment. Consequently, the treated mean is an unbiased estimate of the finite-population mean of $Y_j(1)$, and the control mean is an unbiased estimate of the finite-population mean of $Y_j(0)$. Their difference is therefore unbiased for $\tau$.

Theorem: Define $S_1^2$, $S_0^2$, and $S_\tau^2$ as the finite-population sample variances, each using denominator $J-1$, of $Y_j(1)$, $Y_j(0)$, and $\tau_j$, respectively. Under complete randomization of exactly $m$ clusters, $$ \operatorname{Var}_Z(\hat{\tau}) = \frac{S_1^2}{m} + \frac{S_0^2}{J-m} - \frac{S_\tau^2}{J}. $$

The subscript $Z$ emphasizes that this variance is over the assignment mechanism. The final term depends on the variation in the cluster treatment effects. Since $S_\tau^2$ involves both potential outcomes for each cluster, it cannot generally be computed from the observed data: each cluster reveals only one of its two potential outcomes.

Why the Variance Formula Has a Subtraction

A short derivation makes the role of treatment-effect heterogeneity clear. Write $S_{10}$ for the finite-population covariance between $Y_j(1)$ and $Y_j(0)$, using denominator $J-1$. Under complete randomization, the treated clusters are a simple random sample of size $m$, and the control clusters are their complement. The variance of the treated mean is $(J-m)S_1^2/(Jm)$; the variance of the control mean is $mS_0^2/[J(J-m)]$. The covariance between these two means is $-S_{10}/J$, because membership in one group rules out membership in the other. Therefore,

\begin{aligned} \operatorname{Var}_Z(\hat{\tau}) &= \frac{J-m}{Jm}S_1^2 + \frac{m}{J(J-m)}S_0^2 + \frac{2S_{10}}{J}\\ &= \frac{S_1^2}{m} + \frac{S_0^2}{J-m} - \frac{S_1^2+S_0^2-2S_{10}}{J}\\ &= \frac{S_1^2}{m} + \frac{S_0^2}{J-m} - \frac{S_\tau^2}{J}. \end{aligned}

The second equality uses the finite-population identity $S_\tau^2=S_1^2+S_0^2-2S_{10}$, which follows by expanding the variance of $Y_j(1)-Y_j(0)$. Since $S_\tau^2\geq 0$, treatment-effect heterogeneity reduces the variance relative to the first two terms. A variance formula that omits this unobserved term will be conservative on average.

Worked Example: Enumerating All Assignments

Consider four hypothetical clusters, with exactly two assigned treatment. The following table gives their fixed potential outcomes, in outcome points.

Cluster	$Y_j(1)$	$Y_j(0)$	$\tau_j$
1	8	5	3
2	7	6	1
3	9	4	5
4	10	9	1

The cluster-average treatment effect is $\tau=(3+1+5+1)/4=2.5$ points. There are $\binom{4}{2}=6$ equally likely assignments. For each, compute the treated clusters' mean $Y(1)$ minus the control clusters' mean $Y(0)$:

Treated clusters	Difference in means (points)
1 and 2	$(8+7)/2-(4+9)/2=1$
1 and 3	$(8+9)/2-(6+9)/2=1$
1 and 4	$(8+10)/2-(6+4)/2=4$
2 and 3	$(7+9)/2-(5+9)/2=1$
2 and 4	$(7+10)/2-(5+4)/2=4$
3 and 4	$(9+10)/2-(5+6)/2=4$

The average of these six estimates is $(1+1+4+1+4+4)/6=2.5$, matching $\tau$. Their assignment variance is \[ \{3(1-2.5)^2+3(4-2.5)^2\}/6=2.25 \] square points. The finite-population variances are $S_1^2=5/3$, $S_0^2=14/3$, and $S_\tau^2=11/3$. The theorem gives $(5/3)/2+(14/3)/2-(11/3)/4=2.25$ square points, exactly the enumerated variance.

Estimating the Design-Based Variance

The finite-population variance formula contains $S_\tau^2$, which is unobserved because no cluster supplies both potential outcomes. A usable estimator replaces $S_1^2$ and $S_0^2$ by the sample variances in the observed treatment and control groups, then omits the unavailable nonnegative term.

Definition: Let $s_1^2$ be the sample variance of the observed outcomes among the $m$ treated clusters, and $s_0^2$ the sample variance among the $J-m$ control clusters. The Neyman estimator of the design-based variance is $$ \widehat{\operatorname{Var}}_Z(\hat{\tau}) = \frac{s_1^2}{m} + \frac{s_0^2}{J-m}. $$ The estimated standard error is the square root of this quantity.

Each observed group is a simple random sample from its corresponding finite population of potential outcomes. Thus the sample variances estimate $S_1^2$ and $S_0^2$ without bias. Taking expectations over the assignment gives

E_Z\!\left[\widehat{\operatorname{Var}}_Z(\hat{\tau})\right] = \frac{S_1^2}{m}+\frac{S_0^2}{J-m} = \operatorname{Var}_Z(\hat{\tau})+\frac{S_\tau^2}{J}.

So this estimator is conservative in expectation, and is unbiased for the assignment variance when treatment effects are constant across clusters, so that $S_\tau^2=0$. “Conservative” here is a statement about its expectation over assignments, not a guarantee that it exceeds the true variance in every realized experiment.

Worked Example: Estimating the Variance From Observed Clusters

In a hypothetical experiment, six clusters are randomized, three to each group. Their observed cluster-average outcomes are 12, 15, and 13 points for treatment, and 9, 10, and 8 points for control. The difference in means is \[ \hat{\tau}=(12+15+13)/3-(9+10+8)/3=13.333-9=4.333 \] points.

For treatment, the mean is $13.333$, and the sum of squared deviations is $4.667$, giving $s_1^2=4.667/(3-1)=2.333$ square points. For control, the mean is 9 and the sum of squared deviations is 2, giving $s_0^2=2/(3-1)=1.000$ square points. Therefore,

\widehat{\operatorname{Var}}_Z(\hat{\tau}) = \frac{2.333}{3}+\frac{1.000}{3} = 1.111\ \text{square points}, \qquad \widehat{SE}(\hat{\tau})=\sqrt{1.111}=1.054\ \text{points}.

This is an estimate of assignment variance for the cluster-average contrast, under the stated randomization. It does not require the cluster outcomes to follow a normal distribution. Inference using this standard error may require additional justification, especially with few clusters; the calculation alone does not make a normal or $t$-based approximation exact.

Which Treatment Effect Is Being Estimated?

The difference in cluster means gives equal weight to each cluster. When cluster sizes differ, that is not necessarily the same target as the average effect for an individual. This is a choice of estimand, not a small-sample correction.

Worked Example: Unequal Cluster Sizes and the Target

Suppose four hypothetical clusters have sizes 10, 20, 30, and 40. Their cluster-average potential outcomes are the values in the earlier table: treatment means 8, 7, 9, and 10, and control means 5, 6, 4, and 9 points. The cluster effects are 3, 1, 5, and 1 points.

The equally weighted cluster-average effect is $(3+1+5+1)/4=2.5$ points. If the target instead gives each individual equal weight, the effect is $(10\cdot3+20\cdot1+30\cdot5+40\cdot1)/(10+20+30+40)=240/100=2.4$ points. These differ because the cluster with effect 5 contains 30 individuals, while the clusters with effect 1 have different sizes.

The variance formula in this tutorial applies to the equally weighted cluster-level difference in means. It should not be applied automatically to a differently weighted estimator or interpreted as uncertainty for the individual-weighted effect. The estimator, target, and assignment probabilities must be aligned.

Design-Based and Model-Based Uncertainty

In design-based analysis, the potential outcomes are fixed and the random assignment is the source of uncertainty. The variance formula follows from the probabilities of assigning clusters to each arm, together with the finite-population variation in potential outcomes. It does not require independent individual errors within a cluster, a particular intracluster correlation, or a correctly specified outcome regression.

A model-based analysis instead specifies a stochastic model for outcomes or errors and derives uncertainty under that model. Such a model can be useful, but it answers a different question and relies on its own assumptions. Cluster-robust covariance estimators address dependence in regression errors under their corresponding sampling or asymptotic framework; the finite-population design variance here is not merely a model for those errors. If assignment is blocked, uses unequal probabilities, or is otherwise not complete randomization, the formulas must reflect that actual design.

Common Mistakes

Calling the plug-in estimator exactly unbiased. It omits $S_\tau^2/J$. Its expectation equals the true design variance plus that nonnegative quantity, so it is conservative on average.
Treating potential outcomes as random errors in this derivation. The design-based calculation holds potential outcomes fixed; the assignment is random.
Counting individuals as independently randomized. When whole clusters are assigned, the randomization unit is the cluster. The number of individuals does not replace $J$, $m$, or $J-m$ in these formulas.
Ignoring the estimand when cluster sizes vary. An equally weighted contrast of cluster means targets an average across clusters, not automatically an average across individuals.

Key takeaway: Under complete randomization of clusters, the exact assignment variance subtracts the unobserved treatment-effect variance $S_\tau^2/J$. The usual estimator replaces the observable finite-population variances with within-arm sample variances and omits that term, making it conservative in expectation.

Check Your Understanding

Use the definitions and assumptions in this tutorial to answer the following.

What is held fixed, and what is random, in design-based variance estimation?
Why can $S_\tau^2$ not generally be calculated from the observed outcomes?
Under what condition is the Neyman variance estimator unbiased for the assignment variance?
In the unequal-size example, which target is estimated by equally weighting the four cluster effects, and how does it differ from the individual-weighted target?
Why does having many individuals within a small number of randomized clusters not make the number of assignment units large?

Cluster	\(Y_j(1)\)	\(Y_j(0)\)	\(\tau_j\)
1	8	5	3
2	7	6	1
3	9	4	5
4	10	9	1

Treated clusters	Difference in means (points)
1 and 2	\((8+7)/2-(4+9)/2=1\)
1 and 3	\((8+9)/2-(6+9)/2=1\)
1 and 4	\((8+10)/2-(6+4)/2=4\)
2 and 3	\((7+9)/2-(5+9)/2=1\)
2 and 4	\((7+10)/2-(5+4)/2=4\)
3 and 4	\((9+10)/2-(5+6)/2=4\)