Why Separate Balance From Sample Size?
A balance diagnostic should describe the size of a covariate difference on a scale that is interpretable across variables. A significance test instead asks whether the data are compatible with a specified null hypothesis, taking sampling uncertainty into account. Because that uncertainty generally decreases as sample size grows, a test can give very different results at different sample sizes even when the observed difference relative to the covariate’s spread is unchanged.
This distinction is one reason standardized mean differences (SMDs) are used to describe covariate balance. But “does not depend on sample size” needs careful qualification. Sample size does not appear explicitly in the basic SMD formula. However, the means and scale are calculated from data, and a conventional sample standard deviation can itself change under exact replication because its divisor is \(n-1\). The precise claim is therefore that SMDs are not mechanically multiplied or divided by sample size in the way a test statistic is; they are not guaranteed to be exactly invariant to every change in sample size or data.
For a common pooled sample-standard-deviation convention, with group sizes \(n_1,n_0\) and sample standard deviations \(s_1,s_0\), the scale is $$ s_p=\sqrt{\frac{(n_1-1)s_1^2+(n_0-1)s_0^2}{n_1+n_0-2}}. $$ Then \(\mathrm{SMD}=(\bar{x}_1-\bar{x}_0)/s_p\). The choice of denominator is addressed by the earlier material on choosing a denominator for standardized mean differences; here, the key point is to state the convention and use it consistently. Once the observed means and chosen scale are given, there is no separate factor involving \(n_1\) or \(n_0\) in the SMD formula.
What “No Mechanical Sample-Size Effect” Means
Imagine two hypothetical data sets with the same difference between group means and the same reference scale. The SMD is the same, whether those summaries came from a small or large sample. This is a statement about the formula conditional on those summaries, not a claim that sample size has no influence on the data summaries themselves. In repeated samples, the estimated means and standard deviations vary; larger samples usually make those estimates less variable.
The contrast with a standard error explains why significance tests behave differently. For independent observations in two groups, a Welch two-sample test uses the estimated standard error $$ SE(\bar{x}_1-\bar{x}_0)=\sqrt{\frac{s_1^2}{n_1}+\frac{s_0^2}{n_0}}. $$ Under the equal-variance pooled-\(t\) model, where \(s_1=s_0=s\) and \(n_1=n_0=n\), the test statistic for a difference from zero is $$ t=\frac{\bar{x}_1-\bar{x}_0}{s\sqrt{2/n}} =\left(\frac{\bar{x}_1-\bar{x}_0}{s}\right)\sqrt{\frac{n}{2}} =\mathrm{SMD}\sqrt{\frac{n}{2}}. $$ This algebra shows that if the observed SMD is held fixed in this equal-size setting, the test statistic increases with \(\sqrt{n}\). The exact \(t\) distribution under the null requires independent observations and normally distributed outcomes with equal variances; with large samples, the test is often justified approximately under less restrictive distributional conditions. Neither a small p-value nor a large one is a direct measure of the size of the covariate difference.
Worked Example: The Same SMD, Different Test Results
Suppose two hypothetical independent groups have a covariate difference of \(0.4\) units and a standard deviation of \(2\) units in each group. Compare a design with 25 observations per group to one with 400 per group. For this illustration, use the equal-variance pooled-\(t\) model.
In either design, the SMD is $$ \mathrm{SMD}=\frac{0.4}{2}=0.200. $$ Thus the standardized difference is 0.200 in both sets of summaries.
With 25 observations per group, the standard error is $$ SE=\sqrt{\frac{2^2}{25}+\frac{2^2}{25}} =\sqrt{0.32}=0.5657, $$ so \(t=0.4/0.5657=0.7071\), with \(df=25+25-2=48\). The two-sided p-value is approximately 0.483. This test does not provide evidence against \(H_0\) at \(\alpha=0.05\).
With 400 observations per group, $$ SE=\sqrt{\frac{2^2}{400}+\frac{2^2}{400}} =\sqrt{0.02}=0.1414, $$ so \(t=0.4/\sqrt{2^2/400+2^2/400}\approx2.828\), with \(df=798\). The two-sided p-value is approximately 0.0048. This test gives evidence against \(H_0\) at \(\alpha=0.05\).
The test results differ because the larger sample gives a smaller standard error, not because the observed standardized difference became smaller. The p-values concern compatibility with the null hypothesis of equal means under the test model; they do not establish whether the covariate difference is substantively small or whether the groups are balanced on other features.
A Finite-Sample Caveat: Exact Replication
The phrase “SMDs do not depend on sample size” can be misleading if it is taken to mean that copying every observation leaves an SMD exactly unchanged. With the conventional sample variance, \(s^2=\sum_i(x_i-\bar{x})^2/(n-1)\), exact replication multiplies the sum of squared deviations and changes the degrees-of-freedom divisor. The sample variance generally changes under exact replication when the within-group sum of squares is nonzero, even though the group mean does not.
For two groups using the pooled sample standard deviation, let \(N=n_1+n_0\), and let \(A=(n_1-1)s_1^2+(n_0-1)s_0^2\), the sum of the groups’ within-group sums of squares. The original pooled variance is \(s_p^2=A/(N-2)\). If every observation in both groups is copied \(k\) times, the means stay fixed and the within-group sums of squares become \(kA\). The replicated pooled variance is therefore $$ s_{p,\mathrm{rep}}^2=\frac{kA}{kN-2} =s_p^2\frac{k(N-2)}{kN-2}. $$ The multiplier is generally not 1. In particular, exact replication does not create independent information: the copied observations are duplicates. This calculation describes how the conventional sample-SD formula responds to the altered data table, not a valid way to increase the study’s effective sample size.
Worked Example: Replication Changes a Pooled Sample SD
Consider hypothetical covariate values of 0 and 20 in group 1, and 0 and 0 in group 0. Initially, each group has \(n=2\). The means are \(\bar{x}_1=10\) and \(\bar{x}_0=0\). Group 1 has sample variance $$ s_1^2=\frac{(0-10)^2+(20-10)^2}{2-1}=200, $$ while group 0 has \(s_0^2=0\). The pooled sample variance and SMD are $$ s_p^2=\frac{(2-1)(200)+(2-1)(0)}{2+2-2}=100,\qquad \mathrm{SMD}=\frac{10-0}{\sqrt{100}}=1.000. $$
Now copy every value five times. Each group has \(n=10\); the means remain 10 and 0. Group 1’s sum of squared deviations is \(5(200)=1000\), so its replicated sample variance is \(1000/(10-1)=111.111\). Group 0’s sample variance remains zero. The pooled variance and SMD are $$ s_{p,\mathrm{rep}}^2=\frac{1000}{18}\approx55.5556, \qquad \mathrm{SMD}_{\mathrm{rep}}=\frac{10}{\sqrt{1000/18}}\approx1.342. $$
The SMD changed because the pooled sample-SD denominator changed under the \(n-1\) convention; the mean difference did not. This does not contradict the absence of an explicit sample-size multiplier in the SMD formula. It does show why exact invariance should not be claimed without specifying the variance convention. A variance calculated with divisor \(n\) would be preserved by exact replication, but that is a different convention.
Using the Distinction in Balance Assessment
A significance test is usually a poor primary summary of covariate balance. Its p-value combines the observed difference with its estimated sampling uncertainty, and that uncertainty depends on sample size. Consequently, a small difference may yield a small p-value in a very large sample, while a larger difference may yield a non-small p-value in a small sample. The SMD makes the size of the mean difference relative to a stated scale more visible without directly scaling it by the sample size.
This does not make an SMD a complete description of balance. It summarizes a mean difference, so it can fail to reveal differences in spread, tails, or distributional shape. It also depends on the selected denominator, and both its numerator and denominator are estimates when calculated from a sample. Use it as a descriptive diagnostic alongside a clear account of the groups and the covariate, not as proof that all relevant features are balanced.
Several checks help identify misuse:
- Check the denominator. State whether the scale is pooled, based on a specified group, or otherwise defined. Do not compare SMDs calculated with inconsistent conventions as if they were identical summaries.
- Separate effect size from evidence against a null. A p-value that changes with sample size does not mean the SMD must have changed, and a non-small p-value does not certify balance.
- Check what is being held fixed. The algebraic comparison assumes the same observed mean difference and scale. Different samples usually have different estimates, so their SMDs may differ.
- Do not count duplicates as independent observations. Exact replication changes sample-SD formulas but supplies no new independent information.
Raw mean differences remain useful when the covariate’s units are directly meaningful. Standardized differences support comparisons across covariates measured in different units. Significance tests answer a different question about a null model and sampling uncertainty. Neither scale-free summaries nor tests alone address every feature of a covariate distribution.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Two groups have a mean difference of 3 units and a chosen scale of 12 units. What is the SMD? If the same summaries occur in a larger sample, what is the SMD?
- Under an equal-variance pooled-\(t\) model, the two groups each have \(n=100\), the mean difference is 0.3, and the common standard deviation is 1.5. Find the SMD and test statistic. Repeat the test-statistic calculation for \(n=900\) per group if the difference and standard deviation stay fixed.
- Why does copying every observation exactly not necessarily preserve a pooled SMD based on sample standard deviations?
- What can and cannot be concluded when a large study gives a small p-value for a covariate mean difference?
Solutions. (1) \(\mathrm{SMD}=3/12=0.250\). If the mean difference and scale stay at those values, it remains 0.250; sample size has no separate multiplier in the formula. (2) The SMD is \(0.3/1.5=0.200\). At \(n=100\), \(SE=1.5\sqrt{2/100}=0.2121\), so \(t=0.3/0.2121=1.414\). At \(n=900\), \(SE=1.5\sqrt{2/900}=0.07071\), so \(t=0.3/0.07071=4.243\). The SMD stays fixed under the stated summaries, while the test statistic increases. (3) Replication multiplies within-group sums of squares by the replication factor \(k\), but changes the sample-variance degrees of freedom from \(N-2\) to \(kN-2\). Therefore the pooled sample variance, and potentially the SMD, changes. (4) The test gives evidence against equal means under its assumptions, but the p-value alone does not quantify the difference’s size or establish balance on other distributional features. A large sample can make a relatively small difference statistically detectable.