Why a Balance Test Is Not a Balance Score
After matching or weighting, it is tempting to test each covariate for a difference between treatment groups and call the groups balanced when the tests are not statistically significant. That rule confuses two questions. A significance test asks whether the data are compatible with a specified null hypothesis, such as equal population means. A balance assessment asks whether the observed covariate distributions are sufficiently similar for the intended analysis. The first question depends on sample size and test assumptions; the second is a diagnostic judgment about the size and pattern of remaining differences.
A very large sample can make a negligible difference statistically significant. A small sample can leave a substantial difference statistically non-significant. Neither result, by itself, says whether the covariate imbalance is practically important. This is why p-values should accompany, not replace, descriptive measures such as standardized differences, variance ratios, and distributional plots.
Sample Size Changes the Test, Not the Observed Difference
For a two-group comparison of means, a common test statistic is the observed difference in sample means divided by its estimated standard error. Under a Welch two-sample t-test, which does not require equal group variances, the statistic is
Here, \(\bar{x}_1-\bar{x}_0\) is the observed mean difference, and \(s_1^2/n_1+s_0^2/n_0\) estimates its variance. When the group sizes increase while the difference and standard deviations stay fixed, the denominator generally shrinks. The same difference can therefore produce a larger absolute test statistic and a smaller p-value.
A standardized mean difference puts the observed difference on a scale of covariate variability. For the examples below, define \(s_p=\sqrt{(s_1^2+s_0^2)/2}\) and use \((\bar{x}_1-\bar{x}_0)/s_p\). This is a descriptive effect-size summary; it is not a test statistic. Its denominator should be chosen consistently for the balance assessment.
Worked Example: A Negligible Difference With a Small P-Value
Suppose a hypothetical matched sample has 100,000 people in each group. A continuous baseline covariate has sample means 50.0 and 50.1 units, and sample standard deviations of 10.0 units in both groups. Use a two-sided Welch test of equal means.
The observed mean difference is \(50.1-50.0=0.1\) unit. Its estimated standard error is
Thus \(t=0.1/0.04472=2.236\). With approximately 199,998 degrees of freedom, the two-sided p-value is about 0.02535. At a conventional \(\alpha=0.05\), this test rejects equal means. Yet the standardized mean difference is only
The data provide evidence of a nonzero mean difference under the test model, but the observed difference is only 0.010 pooled standard deviations. The small p-value does not turn that small descriptive difference into a large imbalance.
The key calculation is the ratio of the difference to its standard error. Holding the observed difference fixed, multiplying both group sizes by four divides each variance contribution \(s_g^2/n_g\) by four, so the standard error is halved and the absolute test statistic doubles. The observed difference itself has not changed. This is the basic reason that significance is not a fixed threshold for acceptable balance.
A Non-Significant Result Can Hide a Large Difference
The converse occurs when the sample is small. A substantial standardized difference may yield a wide sampling distribution for the test statistic and a large p-value. Failure to reject equal means is not proof that the means are equal, nor does it establish that the groups are balanced.
Worked Example: A Large Standardized Difference With a Non-Significant Test
Consider a hypothetical matched sample with four people in each group. A baseline covariate has means 50 and 54 units and standard deviations of 5 units in both groups. Apply a two-sided Welch test.
The mean difference is \(54-50=4\) units. The estimated standard error is
The test statistic is \(t=4/3.536=1.131\). With \(df=6\), the two-sided p-value is approximately 0.300. At \(\alpha=0.05\), the test does not reject equal means. But the standardized mean difference is
A difference of 0.800 pooled standard deviations is substantial as a descriptive contrast, even though this small-sample test is not statistically significant. The appropriate conclusion is that the test did not detect a difference at the chosen significance level; it is not that the groups have been shown to be balanced.
What the Test Assumes—and What It Tests
A p-value is meaningful only in relation to its null hypothesis and the conditions used to calculate it. For a two-sample t-test, the observations within each group should be independent under the sampling model. Welch’s test allows unequal variances, but its reference distribution relies on an appropriate approximation for the sampling distribution of the difference in means. With small samples, strong skewness or outliers can make that approximation unreliable. With large samples, the mean can often be estimated well despite moderate non-normality, but dependence and a poorly chosen variance estimator remain problems.
Matching and weighting can create structures that an ordinary two-independent-samples test does not represent. For example, if matched pairs or matched sets are treated as independent individual observations, a test that ignores the within-set dependence may calculate an incorrect standard error. Repeated observations or clustered assignment can cause similar problems. Weighting also changes how observations contribute to estimates; applying an unweighted test to weighted data does not automatically give a valid test for the weighted comparison. The test must reflect the design and dependence structure if inferential p-values are to be interpreted.
Even a correctly calculated test addresses only the feature in its null hypothesis. A test of equal means can miss differences in spread, skewness, tails, or subgroups when the means happen to be similar. A test of equal proportions addresses a binary covariate, but sparse categories can make large-sample approximations unreliable. A test of one marginal covariate at a time also does not summarize joint patterns across covariates. Distributional and multivariate diagnostics are needed to look for those features.
Worked Example: Many Tests Produce Chance Findings
Suppose, hypothetically, that 20 covariates are perfectly balanced in the populations being compared. If 20 independent tests are each conducted at \(\alpha=0.05\), each test has a 0.05 probability of a false rejection under its null. The probability that none rejects is \(0.95^{20}\). Therefore, the probability of at least one false rejection is
So, even when every null hypothesis is true, there is about a 64.15% chance that at least one of these 20 tests will produce a p-value below 0.05, under the stated independence assumption. If the tests are dependent, this calculation no longer applies exactly, but the general issue remains: interpreting a collection of unadjusted p-values as if each were the only test can overemphasize chance findings. Conversely, applying a multiple-testing correction does not make p-values a comprehensive measure of balance; it changes a testing rule, not the descriptive differences themselves.
Use Descriptive Diagnostics Alongside Tests
Balance assessment is primarily descriptive: report how different the observed groups are on relevant covariates, and inspect the shapes and patterns of those differences. Standardized differences make mean differences comparable across measurement scales and do not become statistically significant merely because the sample size grows. They still have limits: a small standardized mean difference does not rule out unequal variances or different distribution shapes. Variance ratios, quantile-quantile plots, empirical distribution comparisons, and checks of interactions or subgroups can reveal patterns that a mean test cannot.
| Approach | What it summarizes | What it can miss or distort |
|---|---|---|
| Significance test | Evidence against a specified null, conditional on the test model and assumptions | Practical size of the difference; features not included in the null; effects of sample size and dependence |
| Standardized difference | Observed mean or proportion difference on a standardized scale | Differences in spread, tails, shape, or joint patterns |
| Distributional or multivariate diagnostic | Patterns such as spread, quantiles, tails, or joint covariate structure | May be harder to reduce to one number; interpretation still depends on context |
The earlier material on standardized-difference thresholds, variance ratios, skewness and tail imbalance, quantile-quantile plots, empirical-distribution diagnostics, and multivariate balance provides tools for these comparisons. No single diagnostic proves that all relevant differences are absent. Read the diagnostics together and judge differences in light of the covariates’ substantive importance and the planned analysis.
A practical sequence is to examine descriptive summaries and plots first, then use a test only when a specific inferential question and an appropriate test model justify it. Report the test’s null hypothesis, method, and p-value rather than using “not significant” as a synonym for “balanced.” If a test and descriptive diagnostic appear to disagree, investigate why: sample size, distribution shape, sparse categories, dependence, or the fact that the test concerns only one feature may explain the discrepancy.
Common Mistakes and Limits
- “The p-value exceeds 0.05, so balance is established.” A non-significant result is not evidence that the difference is zero or acceptably small. Describe the observed difference and its scale.
- “A small p-value means the imbalance is large.” A small p-value can accompany a tiny difference when the sample is large. Report a descriptive effect size and inspect the distribution.
- “A mean test checks the entire covariate distribution.” It tests a hypothesis about a mean, not equality of every distributional feature. Examine spread, shape, tails, and relevant joint patterns separately.
- “The standard two-sample test works after any matching or weighting.” It may fail to represent matched dependence, clustering, or the role of weights. Use a method consistent with the design if making an inferential claim.
- “Correcting many tests solves balance assessment.” A correction addresses a chosen error-rate goal for tests; it does not replace descriptive diagnostics or determine whether an observed difference is practically important.
Significance tests have a legitimate but limited role: they can quantify evidence against particular equality hypotheses when their assumptions and design are handled appropriately. They are not universal pass-fail criteria for matching quality. Descriptive diagnostics also have limitations and do not certify the absence of all imbalance; their value is that they show the magnitude and pattern of observed differences more directly.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- If both group sizes in a two-sample mean test are multiplied by four while the means and standard deviations stay fixed, what happens to the standard error and the test statistic?
- Two groups have means differing by 3 units and a chosen standardizing standard deviation of 6 units. What is the standardized mean difference, and does that value depend directly on sample size?
- Under 10 independent true null hypotheses tested at \(\alpha=0.05\), what is the probability of at least one false rejection?
- Why might an ordinary independent-samples test be inappropriate for observations arranged in matched pairs?
Solutions. (1) Each variance contribution \(s_g^2/n_g\) is divided by four, so the standard error is halved. The numerator is unchanged, so the absolute test statistic doubles. (2) The standardized difference is \(3/6=0.5\). The observed standardized difference does not directly depend on sample size, although its sampling uncertainty does. (3) The probability is \(1-0.95^{10}=0.4013\), or about 40.13%, assuming independent tests. (4) Members of a matched pair may be dependent. An independent-samples test ignores that structure, so its standard error and p-value may not be valid; the analysis must account for the pairing.