Chance Imbalance in Small Randomized Experiments

Why Randomized Groups Can Differ at Baseline

Random assignment gives each eligible experimental unit a chance to receive each treatment according to a stated allocation procedure. It does not promise that the groups will look identical before treatment begins. This distinction is especially visible in a small experiment: a few units assigned one way rather than another can produce a noticeable difference in age, initial symptom score, or another baseline characteristic.

A baseline characteristic is measured before treatment is assigned or begins. Because it precedes treatment, the treatment cannot have caused its value. If a characteristic differs between the assigned groups, the difference may be a consequence of the particular allocation that happened to be drawn. That is chance imbalance. It is not, by itself, evidence that the allocation process favored one group.

Definition: Chance imbalance is a difference between randomized groups in a pre-treatment characteristic that occurs because of the particular random allocation drawn. Systematic treatment-selection bias occurs when the assignment process gives treatment status according to a rule related to characteristics of the units, so some units are preferentially assigned to a treatment.

The key evidence is the assignment procedure, not simply whether the groups happen to differ. Under a properly carried-out random allocation, an imbalance can occur even though no unit was deliberately favored. Conversely, a group difference produced by assigning treatment based on baseline characteristics is not converted into randomization merely because the groups were given treatment labels afterward.

A Chance Imbalance in a Small Experiment

Consider a hypothetical experiment with eight participants: four have a high baseline symptom category and four have a low category. Exactly four participants are assigned to treatment and four to control. Suppose the allocation is made by selecting four of the eight participants uniformly at random; every possible set of four has the same chance of being assigned to treatment.

Let \(K\) be the number of high-category participants assigned to treatment. If \(K=3\), the treatment group has three high-category participants out of four, while the control group has one out of four. The difference in the groups' high-category proportions is \(3/4-1/4=0.50\), even though assignment was random.

Worked Example: Probability of a Noticeable Category Imbalance

For the eight participants above, calculate the probability that the absolute difference between the treatment and control proportions in the high category is at least 0.50.

Check the assignment mechanism. There are four high-category and four low-category participants. Treatment receives four participants, and every four-person set is equally likely. Thus the total number of possible treatment groups is \(\binom{8}{4}=70\).

If treatment receives \(k\) high-category participants, control receives \(4-k\) high-category participants. The absolute difference in proportions is \( \left|k/4-(4-k)/4\right|=|k-2|/2 \). This is at least 0.50 when \(k=0,1,3,\) or \(4\).

For a given \(k\), the number of possible treatment groups is \(\binom{4}{k}\binom{4}{4-k}\): choose \(k\) of the four high-category participants and \(4-k\) of the four low-category participants. Therefore,

P\left(\left|\hat{p}_{T}-\hat{p}_{C}\right|\geq 0.50\right) = \frac{ \binom{4}{0}\binom{4}{4} +\binom{4}{1}\binom{4}{3} +\binom{4}{3}\binom{4}{1} +\binom{4}{4}\binom{4}{0} }{\binom{8}{4}} = \frac{1+16+16+1}{70} = \frac{34}{70} \approx 0.486

So, under this particular allocation procedure, almost 49% of allocations produce an absolute difference of at least 0.50 in the high-category proportions. This probability describes the allocation mechanism for these eight fixed participants; it does not mean that every randomization will produce such a difference. It does show why a seemingly large baseline difference is quite plausible in a small experiment.

The example also shows why “randomized” and “balanced” are not synonyms. Randomization makes the allocation unpredictable according to the specified procedure. Balance describes how similar the groups happen to be on characteristics. Randomization can lead to balance, but it does not require it in the realized experiment.

Equal on Average Does Not Mean Equal in This Allocation

A useful result makes the distinction precise. Suppose there are \(N\) experimental units with fixed baseline values \(z_1,\ldots,z_N\). A completely randomized allocation assigns exactly \(n_T\) units to treatment, with every subset of that size equally likely; the other \(n_C=N-n_T\) units receive control. The baseline values are fixed before assignment. The treatment-group mean is therefore random because the assigned set is random.

Result: Under complete random assignment, the expected treatment-group mean and expected control-group mean for any fixed baseline characteristic both equal the overall mean of that characteristic. Consequently, the expected treatment-minus-control baseline difference is zero. This is an average over all allocations allowed by the procedure; it does not assert that the difference in the allocation actually drawn is zero.

Here is a short justification. For each unit \(i\), let \(I_i=1\) if unit \(i\) is assigned to treatment and \(I_i=0\) otherwise. Since exactly \(n_T\) of the \(N\) units are selected uniformly, \(P(I_i=1)=n_T/N\). The treatment mean can be written as \(\bar{Z}_T=(1/n_T)\sum_{i=1}^{N} I_i z_i\). Taking its expectation gives:

E(\bar{Z}_T) = \frac{1}{n_T}\sum_{i=1}^{N} E(I_i)z_i = \frac{1}{n_T}\sum_{i=1}^{N}\frac{n_T}{N}z_i = \frac{1}{N}\sum_{i=1}^{N}z_i = \bar{Z}

The control group is the complement of the treatment group and is also a uniformly selected subset of size \(n_C\). The same reasoning gives \(E(\bar{Z}_C)=\bar{Z}\). Subtracting the two expectations yields \(E(\bar{Z}_T-\bar{Z}_C)=0\). This is a statement about repeated allocations under the design, not a guarantee about the single allocation used in a particular experiment.

Worked Example: A Difference in a Continuous Baseline Score

In a hypothetical experiment, eight participants have baseline scores of 2, 3, 4, 5, 7, 8, 9, and 10 points. Four are assigned to treatment and four to control by complete random assignment. One possible draw assigns scores 2, 3, 8, and 9 to treatment, leaving 4, 5, 7, and 10 in control. Find and interpret the difference in baseline means.

Check the mechanism and measurement. The scores are measured before assignment, so treatment cannot have produced them. The procedure assigns four of eight participants uniformly, so this particular allocation is one of the \(\binom{8}{4}=70\) possible allocations. A difference in this draw is compatible with random assignment.

The treatment-group mean is:

\bar{x}_T=\frac{2+3+8+9}{4}=\frac{22}{4}=5.50\text{ points}

The control-group mean is:

\bar{x}_C=\frac{4+5+7+10}{4}=\frac{26}{4}=6.50\text{ points}

The treatment-minus-control difference is \(5.50-6.50=-1.00\) point. In this allocation, the treatment group started one point lower on average. This calculation describes the observed baseline groups; it does not show that treatment caused the difference, nor does it show that assignment was biased. The random allocation can produce this particular partition even though all 70 possible treatment groups were equally likely.

When Assignment Creates Systematic Selection

Now compare random allocation with an assignment rule that deliberately uses the baseline score. Suppose eight hypothetical participants have baseline scores of 1 through 8 points, and the four highest-scoring participants are assigned to treatment while the four lowest are assigned to control. The treatment group has scores 5, 6, 7, and 8; the control group has scores 1, 2, 3, and 4. This rule makes the baseline difference predictable rather than leaving it to random allocation.

Worked Example: Baseline-Based Assignment

Calculate the baseline mean difference under this rule, and consider what would happen if treatment had no effect and each participant's post-treatment score simply remained equal to their baseline score.

Check the assignment rule. Treatment is assigned to the four highest baseline scores. Therefore, this is not the complete random assignment used in the earlier examples: treatment status depends directly on the characteristic being compared.

The baseline means are:

\bar{x}_T=\frac{5+6+7+8}{4}=6.50\text{ points}, \qquad \bar{x}_C=\frac{1+2+3+4}{4}=2.50\text{ points}

Thus the baseline treatment-minus-control difference is \(6.50-2.50=4.00\) points. Under the stated no-effect scenario, the post-treatment scores remain the same as the baseline scores, so the post-treatment difference is also 4.00 points. That difference arises from the assignment rule and the groups' starting values, not from a treatment effect. This illustrates systematic treatment-selection bias: the rule itself creates treatment groups that differ on a relevant pre-treatment characteristic.

An observed difference alone cannot reveal which of these mechanisms produced it. To distinguish them, ask how assignment was actually carried out. Was a random procedure used as planned, or did staff, participants, or a decision rule direct particular units to particular treatments? A realized imbalance under random assignment is a chance result of the procedure. A baseline-related assignment rule is a feature of the procedure itself.

What a Baseline Difference Does—and Does Not—Tell You

A baseline imbalance may matter when interpreting a small experiment because the groups have started at different observed values. It can make a comparison less precise or complicate the interpretation of a particular outcome difference. However, the imbalance does not establish that random assignment failed, and it is not proof that the observed outcome difference is caused by the baseline characteristic. The design and the outcome analysis must be considered separately.

Also, no finite random allocation guarantees balance on every characteristic, including characteristics that were not measured. Randomization supports a fair comparison by making treatment assignment independent of pre-assignment characteristics in expectation over the allocation procedure. It does not arrange participants so that every treatment group is a miniature copy of every other group. A small sample offers fewer units over which chance differences can average out, which is why imbalance can be especially conspicuous in small experiments.

A careful description reports the observed difference in context and avoids treating it as a verdict on the randomization. For example, “The treatment group had a mean baseline score one point lower than control” accurately describes a result. “The one-point difference proves treatment was not randomized” does not follow from that result. Evidence about the assignment protocol is needed to assess whether allocation was random or systematically directed.

Common Mistake

Wrong reasoning: “The randomized groups differ at baseline, so random assignment must have failed.” This confuses a possible result of random allocation with the allocation process itself.
Correction: Random assignment does not guarantee identical baseline groups. Check the assignment procedure: a difference can arise by chance when the stated random mechanism is followed; systematic treatment-selection bias arises when the assignment rule depends on participant characteristics.
Another wrong reasoning: “Randomization guarantees balance, so any baseline difference is harmless.” Randomization balances group means only in expectation across possible allocations. A realized difference can still be substantial and should be described accurately.

Key takeaway: Judge randomization by how treatment was assigned, not by whether the groups happened to match perfectly. Random allocation allows chance imbalance in a particular experiment while making baseline group means equal on average across the allocations it permits.

Check Your Understanding

Answer these questions using the distinction between the allocation process and the groups produced by one allocation.

In a randomized experiment, the treatment group has a higher average baseline age than control. Does that observation alone show that assignment was not random? Explain.
Eight participants include four with a high baseline category and four with a low category. Four are assigned to treatment uniformly at random. If three high-category participants enter treatment, what are the treatment and control proportions in the high category, and what is their difference?
State in words what it means that the expected treatment-minus-control difference in a fixed baseline characteristic is zero under complete random assignment.
A researcher assigns the participants with the highest baseline scores to treatment. Why is this different from an imbalance produced by a random allocation?
Why is it incorrect to say that randomization guarantees identical groups on both measured and unmeasured characteristics?