Reading Chance Imbalance in a Randomized Trial

Why Baseline Groups Can Differ After Randomization

Random assignment makes treatment groups comparable by design, but it does not force their observed baseline characteristics to be identical. A randomized trial is one realization of an assignment process: which participants happen to receive each assignment can produce differences in age, disease severity, or other pretreatment characteristics. Such differences may arise even when the allocation was generated and implemented correctly.

The key distinction is between chance imbalance and systematic treatment selection. In a properly randomized trial, assignment does not systematically favor one group based on baseline covariates. In a particular trial, however, random variation can leave the groups unequal. A difference by itself is therefore not evidence that randomization failed. Conversely, a table of similar observed characteristics cannot prove that randomization was carried out correctly.

Definition: A baseline covariate is a characteristic measured before treatment assignment or before treatment can affect it. Chance imbalance is an observed difference in a baseline covariate between randomized groups that can occur because of the particular random allocation, even when the allocation mechanism was valid.

Let $Z=1$ denote assignment to treatment and $Z=0$ assignment to control, and let $X$ be a baseline covariate. Under random assignment, $Z$ is generated independently of participants' pretreatment characteristics according to the specified allocation mechanism. This does not imply that the realized sample means or proportions of $X$ must match exactly. It means that the allocation mechanism does not systematically favor participants with particular values of $X$.

What Randomization Guarantees on Average

A simple derivation shows the distinction between an average guarantee and an exact-balance guarantee. Suppose $N$ participants are assigned by complete randomization: exactly $n_1$ are assigned to treatment and $n_0=N-n_1$ to control, with every allocation of that size equally likely. Treat the $N$ baseline values $X_i$ as fixed, and write $\bar X$ for their overall mean. The difference in group means is $D=\bar X_1-\bar X_0$.

\bar X = \frac{n_1\bar X_1+n_0\bar X_0}{N} \qquad\Longrightarrow\qquad D = \frac{N}{n_0}(\bar X_1-\bar X).

Under complete randomization, the treatment group is a simple random sample of size $n_1$ from the $N$ fixed values. Its mean has randomization expectation $\bar X$ and variance $(1-n_1/N)S_X^2/n_1$, where $S_X^2=\sum_{i=1}^N(X_i-\bar X)^2/(N-1)$. Using the expression for $D$, its expectation and variance follow:

E(D)=0, \qquad \operatorname{Var}(D) =\left(\frac{N}{n_0}\right)^2 \frac{1-n_1/N}{n_1}S_X^2 =S_X^2\left(\frac{1}{n_1}+\frac{1}{n_0}\right).

Thus the difference has expectation zero over repeated allocations, but its variance is generally positive. A realized difference is not surprising merely because it is nonzero. Larger samples tend to reduce random variation in the difference, while more variable covariates tend to increase it. The derivation concerns the specified complete-randomization design; other assignment schemes, such as blocked randomization, have their own allocation distributions.

Describe the Size of Imbalance

A raw difference is expressed in the covariate's units: years for age, for example, or percentage points for a binary characteristic. Standardized mean differences (SMDs) express the difference relative to a measure of covariate spread, which can help compare imbalance across variables with different units. An SMD describes the observed difference; it is not a test statistic and has no universal cutoff that proves a design is sound or unsound.

Definition: For a continuous baseline covariate, one common SMD is the difference in group means divided by the pooled within-group standard deviation: $$ \operatorname{SMD} =\frac{\bar X_1-\bar X_0} {\sqrt{\{(n_1-1)s_1^2+(n_0-1)s_0^2\}/(n_1+n_0-2)}}. $$ For a binary covariate with group proportions $\hat p_1$ and $\hat p_0$, a common SMD is $$ \operatorname{SMD} =\frac{\hat p_1-\hat p_0} {\sqrt{\{\hat p_1(1-\hat p_1)+\hat p_0(1-\hat p_0)\}/2}}. $$ The sign indicates which group has the larger mean or proportion; the absolute value describes the magnitude.

Worked Example: Standardizing a Difference in a Binary Covariate

Suppose a hypothetical trial assigns 500 participants to each group. At baseline, 12% of the treatment group and 8% of the control group have a particular diagnosis. Assume the diagnosis was measured before assignment and that the stated proportions are the observed group proportions. The raw difference is $0.12-0.08=0.04$, or 4 percentage points.

Using the binary SMD formula, the denominator is $ \sqrt{\{0.12(0.88)+0.08(0.92)\}/2} $. The two variance terms are $0.1056$ and $0.0736$, so the denominator is $ \sqrt{0.0896}\approx0.29933 $. Therefore,

\operatorname{SMD} =\frac{0.12-0.08}{\sqrt{\{0.12(0.88)+0.08(0.92)\}/2}} =\frac{0.04}{0.29933} \approx 0.134.

The treatment group has the higher diagnosis proportion, and the standardized difference is about 0.134. This summarizes the size of the observed difference on a standardized scale. It does not tell us whether the allocation was valid, nor does it establish that the groups differ in a way that changes a treatment-effect estimate.

Worked Example: A Continuous Covariate and a Baseline Test

Consider a hypothetical trial with 100 participants assigned to each group. Mean age is 54.2 years in the treatment group and 51.8 years in the control group; the corresponding sample standard deviations are 10.0 and 9.5 years. Assume age was measured before treatment and assignment followed the planned randomized procedure.

The pooled standard deviation is $ \sqrt{\{99(10.0^2)+99(9.5^2)\}/198}=\sqrt{95.125}\approx9.753 $ years. Thus the SMD is $ (54.2-51.8)/9.753\approx0.246 $. The treatment group's mean age is 2.4 years higher, an observed difference of about 0.246 pooled standard deviations.

For illustration, a conventional two-sample $t$-test gives $SE=\sqrt{10.0^2/100+9.5^2/100}=\sqrt{1.9025}\approx1.379$ years and $t=2.4/1.379\approx1.74$. With approximately 198 degrees of freedom, the two-sided p-value is about 0.083. At $\alpha=0.05$, this test does not reject $H_0$ of equal population means.

That result is not evidence that the groups are exactly balanced, or that randomization has been proven successful. The test asks how compatible a difference at least this large is with its statistical null model; it does not audit how assignments were generated. The estimated difference and its units remain important whether or not the p-value crosses a threshold.

Why Baseline Significance Tests Do Not Verify Randomization

A baseline test is sometimes described as a test of whether randomization “worked.” That description is misleading. A test examines a chosen feature of the observed data under a null model; randomization is a feature of the assignment process. A large p-value does not establish that the process was valid. A small p-value can occur by chance under valid randomization, especially when many characteristics are examined.

Worked Example: Many Baseline Tests

Suppose investigators test 20 baseline covariates at the 0.05 significance level. For a simple illustration, assume all 20 null hypotheses are true and the test results are independent. For each test the probability of a p-value below 0.05 is 0.05. The probability that none of the 20 p-values is below 0.05 is $0.95^{20}$, so the probability of at least one such result is

1-0.95^{20}\approx 0.6415.

Thus even under these simplifying assumptions, there is about a 64% chance of at least one “significant” result. Baseline variables are often correlated, so the independence assumption may not hold; the calculation illustrates why a single small p-value among many tests should not automatically be treated as proof of failed randomization. Nor does a collection of large p-values certify that allocation was properly implemented.

For assessing baseline comparability, report group summaries and consider standardized differences and graphical displays rather than relying on a sequence of significance tests. For continuous variables, inspect means and spreads as well as distributions when shape or outliers matter. For categorical variables, show counts and proportions. A small mean difference can conceal important distributional differences, and a single summary cannot reveal every feature of the data.

Confirm what was measured.
Check that the variables are genuinely pretreatment and that their definitions, units, and missing-data patterns are clear.

Summarize differences descriptively.
Report group means or proportions, raw differences, and useful standardized differences. Examine distributions where summaries may hide relevant structure.

Audit the assignment procedure if warranted.
If patterns are unexpectedly large or suggest allocation problems, review the randomization sequence, concealment, enrollment timing, and records of assignments. The audit concerns the design and implementation, not just a p-value.

Interpretation, Comparisons, and Limitations

In a randomized trial, baseline differences are interpreted against the assignment mechanism: they can be chance variation from a valid allocation, or they can prompt investigation of a possible implementation problem. In an observational comparison, treatment groups were not created by random assignment, so observed baseline differences may reflect how treatment was selected. The same descriptive summaries can be useful in both settings, but they answer different design questions. A balance table alone cannot establish the assumptions needed for a causal comparison in either setting.

Baseline summaries also have limitations. They cover measured characteristics, and no finite table proves equality on every relevant feature. SMDs depend on the chosen denominator and summarize only a particular contrast; they do not encode every aspect of a distribution. A difference that is small relative to a standard deviation may still be important in context, while a statistically detectable difference need not be consequential. Interpretation should combine design knowledge, substantive context, and transparent descriptive reporting.

Common Mistakes

“The groups differ, so randomization failed.” A nonzero baseline difference can occur under valid random assignment. Review the allocation process before drawing a conclusion about implementation.
“The baseline p-values are large, so randomization worked.” Failure to reject a null hypothesis is not proof of a valid procedure or exact balance.
“One small p-value proves selection bias.” Chance and multiple testing can produce small p-values even with valid randomization. Consider the full pattern and, if needed, audit the design records.
“A small SMD means no issue is possible.” An SMD is a descriptive summary, not a universal pass/fail rule; examine units, distributions, and the study context.

Key takeaway: Randomization makes baseline covariates independent of assignment in expectation under the design; it does not guarantee exact balance in the realized groups. Describe observed differences, use diagnostics thoughtfully, and assess randomization through the assignment process rather than baseline significance tests alone.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In a valid completely randomized trial, must the observed treatment and control means of every baseline covariate be equal? Explain.
A binary covariate has proportions 0.20 in treatment and 0.10 in control. What is the raw difference in percentage points, and which group has the higher proportion?
What does a baseline-test p-value of 0.40 establish about whether randomization was implemented correctly?
Why can testing many baseline covariates produce at least one small p-value even when assignment was valid?
Name two useful steps for investigating an unexpectedly large baseline imbalance.

Solutions. (1) No. Randomization balances in expectation across allocations, but a particular valid allocation can produce unequal sample means. (2) The raw difference is $0.20-0.10=0.10$, or 10 percentage points, with the higher proportion in the treatment group. (3) It establishes neither that randomization was correctly implemented nor that the groups are equivalent; it is a result from a test under its null model. (4) Each test has some chance of a small p-value under its null, and conducting many tests increases the chance of seeing at least one by chance. (5) Check measurement and data records, and audit the assignment sequence, concealment, enrollment timing, and implementation records.