Tutorials › ANOVA › Baseline Balance Tables and Misleading Significance Tests

Randomization Tests and Replication in ANOVA · Tutorial 127 of 595

Baseline Balance Tables and Misleading Significance Tests

Summarize pre-treatment characteristics by randomized group and interpret apparent differences without treating significance tests as proof of randomization success.

Beginner 10 min read

What You'll Learn

  • Choose clear descriptive summaries for quantitative and categorical baseline characteristics
  • Calculate and interpret differences and standardized mean differences
  • Explain why random assignment does not guarantee identical groups in a particular sample
  • Understand why a non-significant baseline test does not prove balance
  • Recognize how many baseline tests can produce apparently significant differences by chance
  • Report baseline information without using p-values as a test of randomization success

Why Describe Baseline Characteristics?

Random assignment creates treatment groups by chance. It supports fair comparisons because, across the possible assignments allowed by the design, pre-treatment characteristics are not deliberately allocated to favor one group. It does not promise that the groups will have identical characteristics in the particular experiment that was run. With a modest sample, chance can leave one group older, or with a different mix of baseline health characteristics, than another.

A baseline balance table makes those observed characteristics visible. It summarizes variables measured before treatment was given, separately for each randomized group. Its purpose is descriptive: readers can see who entered each group and the size and direction of observed differences. A table is useful whether or not those differences appear large. It is not a certificate that the randomization process worked.

Definition: A baseline characteristic is a participant characteristic measured before treatment begins. A baseline balance table presents descriptive summaries of such characteristics by randomized group. Observed balance means that the measured baseline characteristics in the groups are similar by the chosen descriptive comparisons; it does not mean the groups are identical on every measured or unmeasured feature.

A sound table begins with the randomization design and the variables measured before assignment or treatment. For a quantitative characteristic, a mean and standard deviation are useful when the distribution is reasonably symmetric and not dominated by extreme values. If a distribution is strongly skewed, a median and interquartile range can describe its center and spread more clearly. For a categorical characteristic, report the count and percentage in each category. Include group sample sizes and units, and use the same summary convention across groups.

The table should make the comparison easy to understand. Counts and percentages describe categorical characteristics; means, standard deviations, medians, and interquartile ranges describe quantitative ones. A difference in means or percentages can also be reported when it helps readers see the direction and size of a disparity. These summaries describe the observed participants. They do not, by themselves, establish why a difference occurred.

Worked Descriptive Comparisons

Worked Example: Comparing a Quantitative Baseline Characteristic

In a hypothetical randomized experiment, age was recorded before treatment for five participants assigned to Treatment T and five assigned to Control C. The ages, in years, were 42, 39, 45, 50, 44 in T and 38, 47, 41, 46, 43 in C. Calculate and interpret the group means, sample standard deviations, and difference in means.

For T, the mean age is \( \bar{x}_T=(42+39+45+50+44)/5=220/5=44.0 \) years. The squared deviations from 44 are \(4,25,1,36,0\), so the sample variance is \(66/(5-1)=16.5\) years squared and the sample standard deviation is \(s_T=\sqrt{16.5}=4.06\) years.

For C, the mean age is \( \bar{x}_C=(38+47+41+46+43)/5=215/5=43.0 \) years. The squared deviations from 43 are \(25,16,4,9,0\), so \(s_C=\sqrt{54/(5-1)}=\sqrt{13.5}=3.67\) years.

The observed mean difference, calculated as T minus C, is \(44.0-43.0=1.00\) year. A concise table entry could show T as 44.0 (SD 4.06) years and C as 43.0 (SD 3.67) years, with a mean difference of 1.00 year. This describes a difference in the observed sample; it does not show that assignment failed or that treatment caused the age difference. Age was measured before treatment.

For some comparisons, a standardized difference can help communicate size on a scale that is not tied to the original measurement units. One common descriptive measure for a quantitative characteristic is the difference in group means divided by a pooled within-group standard deviation. It is a summary, not a hypothesis test. In the example, the pooled standard deviation is \( \sqrt{[(5-1)(4.06^2)+(5-1)(3.67^2)]/(5+5-2)} \), which, using the unrounded variances, is \( \sqrt{(4(16.5)+4(13.5))/8}=\sqrt{15}=3.87 \) years. The standardized difference is \(1.00/3.87=0.258\). State which group is subtracted from which; otherwise the direction is unclear. A standardized value does not have a universal cutoff that proves balance or imbalance.

Worked Example: Comparing a Categorical Baseline Characteristic

In another hypothetical experiment, 40 of 100 participants assigned to T and 30 of 100 assigned to C reported having a particular pre-treatment characteristic. Describe the percentages and their difference, then calculate a standardized difference for the binary characteristic.

In T, the percentage is \(40/100 \times 100\%=40\%\); in C it is \(30/100 \times 100\%=30\%\). The observed difference is \(40\%-30\%=10\) percentage points, with a higher percentage in T. Percentage points are appropriate here: the difference is not simply a “10% increase.”

Let \(p_T=0.40\) and \(p_C=0.30\). A commonly used standardized difference for a binary characteristic is \( (p_T-p_C)/\sqrt{[p_T(1-p_T)+p_C(1-p_C)]/2} \). Substituting the values gives \( (0.40-0.30)/\sqrt{[0.40(0.60)+0.30(0.70)]/2}=0.10/\sqrt{0.225}=0.211 \). Thus, the difference is 0.211 on this standardized scale. The raw difference of 10 percentage points remains important to report because it is immediately interpretable in context. Neither summary determines whether randomization succeeded.

For a categorical characteristic with more than two categories, show each category’s count and percentage within each group. Percentages should use the relevant group size as their denominator, and the table should say so. Combining categories solely to make the table look more balanced can hide useful information; any regrouping should be substantively justified and explained.

Why Baseline Significance Tests Mislead

A significance test of a baseline difference asks a narrow question under a specified statistical model: if that model’s null hypothesis were true, how often would a test statistic at least as extreme as the observed one occur? A large p-value does not demonstrate that the groups are equivalent, that all important characteristics are balanced, or that randomization was carried out correctly. It may simply reflect imprecise data or a small sample.

A small p-value is not proof that randomization failed either. Proper random assignment can produce an unusual-looking allocation by chance. A baseline test assesses a difference relative to a reference model or assignment mechanism; it does not directly inspect whether the assignment procedure was followed. That question is addressed by documenting and auditing the randomization process, not by asking whether baseline p-values exceed a threshold.

Worked Example: A Significant Baseline Difference Can Occur by Chance

Suppose a hypothetical experiment randomly assigns 100 participants to each of two groups. Before treatment, the mean age is 51.4 years (SD 10.0) in T and 48.6 years (SD 10.0) in C. Calculate the standardized mean difference and the approximate two-sided Welch t-test result. Assume the age measurements are independent between participants and that the large samples make the mean comparison reasonably approximated by the t procedure.

The mean difference, T minus C, is \(51.4-48.6=2.80\) years. The observed standard error for the difference is \( SE=\sqrt{10.0^2/100+10.0^2/100}=\sqrt{2}=1.414 \) years. Therefore, \(t=2.80/1.414=1.98\). With equal sample sizes and these standard deviations, the Welch degrees of freedom are approximately 198, giving a two-sided p-value of about 0.049.

The standardized mean difference is \(2.80/10.0=0.280\), because the two observed standard deviations are both 10.0 years. This example has a p-value just below 0.05, yet it does not establish that the randomized assignment was defective. Even a valid random process sometimes produces such a difference. The result reports how unusual the observed statistic is under the test’s reference model; it is not the probability that randomization failed, nor the probability that the null hypothesis is true.

Testing many baseline variables makes threshold-based interpretations still less reliable. If 20 independent null hypotheses are tested at \(\alpha=0.05\), the probability of at least one p-value below 0.05 is \( 1-(1-0.05)^{20}=1-0.95^{20}\approx0.642 \). So even when all 20 null hypotheses are true and the tests are independent, there is about a 64.2% chance of at least one “significant” result. Real baseline variables may be related, so the independence calculation will not generally describe an actual table exactly. The key point remains: scanning many p-values invites chance findings, while a collection of non-significant results still cannot prove equivalence.

How to Present the Table

A practical baseline table prioritizes transparent description over significance labels. Include the randomized groups as columns, identify the number assigned to each group, and report a sensible summary for each pre-treatment characteristic. Where useful, add a difference with a clear direction and units, or a standardized difference. Explain summaries that may not be familiar to readers, such as whether a percentage uses all participants in that group as its denominator.

Do not use a baseline p-value as a pass-or-fail rule for randomization, and do not select an analysis only because one baseline comparison crossed a significance threshold. If the design used blocks or other restrictions, describe the allocation method and make the table informative about the groups as randomized. The table should not replace a record of how allocation was generated and implemented.

1
Identify baseline variables.
Include characteristics measured before treatment, and state their units or categories clearly.
2
Choose descriptive summaries.
Use means and standard deviations, or medians and interquartile ranges when appropriate, for quantitative variables; use counts and within-group percentages for categorical variables.
3
Show differences in context.
When useful, give a difference with its direction and units. Treat standardized differences as descriptive scales, not significance tests.
4
Interpret without a pass/fail test.
Use the table to understand the observed sample. Assess whether randomization was implemented from the procedure and its documentation, not from baseline p-values.

Common Mistake

Wrong reasoning: “Every baseline p-value is above 0.05, so the groups are balanced and randomization succeeded.” This treats a failure to detect a difference as proof that no meaningful difference exists. A small sample or noisy measurement can produce a large p-value even when the observed group summaries differ appreciably.

Correction: Report the baseline characteristics descriptively, including differences that matter in the context of the study. A non-significant test does not establish equivalence, and a significant test does not establish failed randomization. Random assignment protects the comparison through its design and procedure; it does not force identical realized groups.

Key takeaway: A baseline balance table describes the characteristics observed in each randomized group. Its summaries help readers assess the size and direction of chance differences, but significance tests cannot certify that randomization succeeded or failed.

Check Your Understanding

Answer these questions using the principles above.

  1. Why can two groups created by proper random assignment have different average ages in the realized sample?
  2. A table reports a baseline mean difference of 2.0 years with a p-value of 0.30. What does the p-value fail to establish?
  3. For a binary characteristic, 25% of T and 20% of C have the characteristic. What is the difference in percentage points, and which group has the higher percentage?
  4. Why can testing many baseline characteristics produce at least one small p-value even when randomization was proper?
  5. Name two kinds of information, other than baseline p-values, that help readers assess whether the randomization procedure was implemented as intended.
Clinical Biostats