Worked Analysis of a Blocked Randomized Trial

Why Analyze the Blocks Separately?

Blocking organizes a randomized trial into groups before treatment is assigned. The design then assigns treatment and control units separately within each block. This can improve precision, but it also makes the choice of weights important: a block containing many units should generally count more toward an average effect for the full trial population than a small block does. If the treatment allocation rates differ between blocks, a pooled difference in observed means can obscure that target.

This tutorial works through a two-arm trial with unequal block sizes. The estimand is the average effect for the finite population enrolled in the trial. The analysis estimates each block’s effect, combines those estimates using the blocks’ population shares, and assesses uncertainty under the randomization design. A separate small example shows how to conduct an exact randomization test for the sharp null.

Estimand and Block-Weighted Estimate

Let the trial population contain \(N\) units divided into \(H\) blocks. Unit \(i\) in block \(h\) has potential outcomes \(Y_{hi}(1)\) under treatment and \(Y_{hi}(0)\) under control. Its causal effect is \(Y_{hi}(1)-Y_{hi}(0)\). Only one potential outcome is observed for each unit, according to its randomized assignment.

Definition: The finite-population average treatment effect in block \(h\) is \(\tau_h=N_h^{-1}\sum_{i=1}^{N_h}[Y_{hi}(1)-Y_{hi}(0)]\), where \(N_h\) is the number of units in that block. The average treatment effect for the full trial population is \(\tau_N=\sum_{h=1}^{H}(N_h/N)\tau_h\), where \(N=\sum_h N_h\). The block-specific difference in observed means is \(\hat{\tau}_h=\bar{Y}_{1h}-\bar{Y}_{0h}\), and the estimator for the full population is \(\hat{\tau}_{\mathrm{block}}=\sum_{h=1}^{H}(N_h/N)\hat{\tau}_h\).

Here, \(\bar{Y}_{1h}\) and \(\bar{Y}_{0h}\) are the observed outcome means among treated and control units in block \(h\). The notation \(N_h/N\) is the block’s share of the trial population, not its share of the treated or control group. For the estimator and its variance to be defined, each block must contain at least one treated and one control unit. Within each block, treatment is assigned at random with fixed arm sizes; assignments are made independently across blocks.

Worked Example: Weighting Two Unequal Blocks

Consider a hypothetical trial with two blocks: 40 units in block A and 60 in block B. In block A, 20 units are assigned to each arm; the observed treatment and control means are 12.0 and 9.0 outcome points. In block B, 15 are treated and 45 are controls; their observed means are 8.0 and 6.5 points. Calculate the block-specific contrasts and the estimate for the full trial population.

The block-specific estimates are \(\hat{\tau}_A=12.0-9.0=3.0\) points and \(\hat{\tau}_B=8.0-6.5=1.5\) points. The population shares are \(40/100=0.4\) and \(60/100=0.6\). Thus,

\hat{\tau}_{\mathrm{block}} =0.4(3.0)+0.6(1.5) =1.2+0.9 =2.1\text{ points}.

This estimate weights each block by its share of the 100-unit population. Under the specified within-block random assignment, it is unbiased for the finite-population average treatment effect across those 100 units. It estimates an average for this trial population, not automatically an effect for a broader superpopulation.

The unequal allocation rates in this example also show why simply pooling all treated outcomes and all control outcomes can answer a different question. The pooled treated mean is \([20(12.0)+15(8.0)]/35=10.286\) points, and the pooled control mean is \([20(9.0)+45(6.5)]/65=7.269\) points. Their difference is about 3.016 points, not 2.1. The pooled comparison gives the blocks different implicit weights in the two arms. It is not the defined population-share-weighted estimator.

Randomization Uncertainty and a Conservative Variance

For each block, write \(S_{1h}^{2}\) and \(S_{0h}^{2}\) for the finite-population variances of the treatment and control potential outcomes, respectively. Write \(S_{\tau h}^{2}\) for the finite-population variance of the unit-level effects in that block. For a finite population of values \(a_i\), the variance convention is \(S^2=(N_h-1)^{-1}\sum_i(a_i-\bar{a})^2\). Let \(n_{1h}\) and \(n_{0h}\) be the fixed numbers assigned to treatment and control, and let \(s_{1h}^{2}\) and \(s_{0h}^{2}\) be the corresponding observed sample variances.

Theorem: Under complete random assignment within each block, the variance of the block-weighted estimator over the randomization distribution is \(\operatorname{Var}(\hat{\tau}_{\mathrm{block}})=\sum_h(N_h/N)^2[S_{1h}^{2}/n_{1h}+S_{0h}^{2}/n_{0h}-S_{\tau h}^{2}/N_h]\). Since \(S_{\tau h}^{2}\) cannot generally be calculated from observed data, a conservative variance estimator is \(\widehat{\operatorname{Var}}_{\mathrm{Neyman}}(\hat{\tau}_{\mathrm{block}})=\sum_h(N_h/N)^2[s_{1h}^{2}/n_{1h}+s_{0h}^{2}/n_{0h}]\). The corresponding standard error is the square root of this estimated variance.

The subtracted term in the exact variance is nonnegative, so dropping it yields a variance at least as large as the true randomization variance. The observed treatment and control sample variances estimate the corresponding finite-population potential-outcome variances under random sampling within each block. Consequently, the variance estimator is conservative in expectation. It is not an exact variance estimate in every realized trial, and “conservative” does not guarantee that it exceeds the true variance for every dataset.

Why the Variance Formula Holds

Within block \(h\), the treated units are a simple random sample of size \(n_{1h}\) from the block’s treatment potential outcomes, and the controls are the complementary random sample. The difference in their observed means therefore has expectation equal to the difference between the two finite-population potential-outcome means, which is \(\tau_h\). For the variance, the two sample means are dependent because the assignment places each unit in exactly one arm. The finite-population variance identity for this difference gives \(S_{1h}^{2}/n_{1h}+S_{0h}^{2}/n_{0h}-S_{\tau h}^{2}/N_h\). Independent assignments across blocks make the variance of the weighted sum the sum of the block variances multiplied by squared weights.

These steps also establish the target and the role of the weights: each block contrast is unbiased for its own finite-population effect, and weighting those contrasts by \(N_h/N\) gives expectation \(\sum_h(N_h/N)\tau_h=\tau_N\). Omitting the unobserved effect-variance term produces the stated conservative variance estimator.

Worked Example: Calculating a Conservative Standard Error

Continue the hypothetical two-block trial. Suppose the observed sample variances are \(s_{1A}^{2}=16\), \(s_{0A}^{2}=9\), \(s_{1B}^{2}=25\), and \(s_{0B}^{2}=16\), in squared outcome points. Use the observed arm sizes and block shares to estimate the standard error of the weighted estimate.

For block A, the variance contribution before weighting is \(16/20+9/20=1.25\). Its weighted contribution is \((40/100)^2(1.25)=0.16(1.25)=0.200\) squared points. For block B, the corresponding calculation is \(25/15+16/45=1.6667+0.3556=2.0222\) squared points, weighted by \((60/100)^2=0.36\), giving \(0.728\) squared points. Thus,

\widehat{\operatorname{Var}}_{\mathrm{Neyman}}(\hat{\tau}_{\mathrm{block}}) =0.200+0.728 =0.928\text{ points}^2, \qquad SE=\sqrt{0.928}=0.9633\text{ points}.

The estimate is 2.1 points with estimated standard error 0.9633 points. If a large-sample normal approximation is justified, an illustrative 95% interval is \(2.1\pm1.96(0.9633)\), or approximately \([0.2119,\,3.988]\) points. This is a normal-approximation interval, not an exact randomization interval. Whether the approximation is suitable depends on the design and the amount of information in the blocks; the calculation alone does not establish that condition. The interval procedure does not assign a 95% probability to this particular fixed finite-population effect being inside the computed endpoints.

Exact Randomization Test of the Sharp Null

A Fisher randomization test assesses the sharp null hypothesis that treatment changes no unit’s outcome: \(H_0:Y_{hi}(1)=Y_{hi}(0)\) for every unit. It is called sharp because it specifies every unit’s missing potential outcome under the null. Under \(H_0\), the observed outcomes can be treated as fixed while treatment labels are reassigned according to the trial’s actual randomization scheme. For a two-sided test, the p-value is the proportion of allowed assignments whose statistic is at least as extreme as the observed statistic, often measured by absolute value. When all allowed assignments are enumerated, this is an exact randomization p-value under the sharp null and the stated assignment mechanism.

Worked Example: Enumerating a Blocked Randomization Test

In a small hypothetical trial, block A has four units, two assigned to treatment, with observed outcomes 2, 4, 6, and 8 points. Block B has three units, one assigned to treatment, with outcomes 1, 3, and 5 points. The observed treated units have outcomes 6 and 8 in block A and 5 in block B. Test the sharp null using the absolute value of the population-share-weighted difference in means.

The observed block contrasts are \(7-3=4\) points in block A and \(5-2=3\) points in block B. Since the block sizes are four and three, the observed statistic is

T_{\mathrm{obs}}=\frac{4}{7}(4)+\frac{3}{7}(3)=\frac{25}{7}=3.571\text{ points}.

Under the sharp null, the outcomes remain fixed and only the assignments vary. There are \(\binom{4}{2}=6\) allowed treatment pairs in block A and \(\binom{3}{1}=3\) allowed assignments in block B, giving \(6(3)=18\) joint assignments. In block A, the possible treatment-pair sums are 6, 8, 10, 10, 12, and 14. For a pair sum \(q\), the contrast is \(q/2-(20-q)/2=q-10\), so its possible values are \(-4,-2,0,0,2,4\). In block B, assigning outcome \(y\) to treatment gives contrast \(y-(9-y)/2\), with possible values \(-3,0,3\).

For any assignment, the weighted statistic is \((4d_A+3d_B)/7\). The observed absolute statistic corresponds to \(|4d_A+3d_B|=25\). Among the 18 assignments, only the combination \(d_A=4,d_B=3\) reaches \(25\), and only \(d_A=-4,d_B=-3\) reaches \(-25\). Thus two assignments are at least as extreme as the observed one, counting ties, and

p=\frac{2}{18}=0.1111.

At a 5% significance level, this test does not reject the sharp null: the observed statistic is not unusual enough under the trial’s blocked assignment mechanism to meet that threshold. This does not establish that every unit has exactly zero effect, nor does the p-value give the probability that the null is true. The test concerns the sharp null; it is not, without further argument, a test of the weaker null that the average effect alone is zero.

Common Pitfall: Using the Wrong Weights

A frequent error is to average block estimates equally, or to pool treated outcomes and control outcomes without accounting for block sizes and allocation rates, then describe the result as the average effect for the full trial population. Those calculations generally use weights other than \(N_h/N\). The correction is to name the target first, calculate each block contrast, and then apply the weights corresponding to that target. For the finite-population average effect, these are the trial population shares. A different estimand may require different weights, but it should be stated explicitly rather than inferred from a convenient calculation.

A second interpretive caution is that randomization-based inference is tied to the assignment design. The standard error above quantifies uncertainty over assignments with fixed treatment counts inside each block. The exact Fisher test likewise reassigns treatment labels within each block while preserving those counts. Neither calculation should be replaced by a procedure that treats the assignments as if they had been generated by a different design.

Key takeaway: For a blocked trial, define the population target, estimate each block’s treatment contrast, and combine those estimates using target-population weights. The Neyman standard error accounts for within-block randomization and is conservative because unit-level treatment-effect variation is not identified from observed outcomes. A Fisher randomization test preserves the original block-specific assignment counts and, for the sharp null, compares the observed statistic with all assignments allowed by the design.

Check Your Understanding

Try these questions before reviewing the calculations.

A trial population has 30 units in block A and 70 in block B. What weights define its finite-population average treatment effect?
Why can a pooled difference in observed means target something other than the finite-population average treatment effect when allocation rates vary across blocks?
What unobserved term is omitted from the Neyman variance estimator, and why does omitting it make the estimator conservative in expectation?
In a Fisher randomization test for the sharp null, which features of the assignment mechanism must be preserved when generating assignments?
What does a two-sided exact randomization p-value of 0.04 mean in context, and what does it not mean?