Presenting Covariate Balance Diagnostics

Why Balance Diagnostics Need Careful Reporting

A matching or weighting analysis changes the composition or influence of the comparison groups. A reader therefore needs to see whether the measured covariates became more similar after adjustment, and which differences remain. Reporting only that “balance was achieved” hides the size and pattern of those differences. Reporting only baseline characteristics without a post-adjustment comparison does not show what the adjustment accomplished.

A useful balance report presents comparable pre- and post-adjustment summaries, states how standardized mean differences (SMDs) were calculated, identifies any threshold used, and names covariates that exceed it after adjustment. Because an SMD for a mean does not describe every feature of a distribution, suitable distributional summaries should accompany it when they are relevant.

Definition: A standardized mean difference expresses the difference between two group means or proportions in units of a specified standard deviation. For a continuous covariate $X$, a common convention is $$ \mathrm{SMD}_X=\frac{\bar{x}_1-\bar{x}_0}{\sqrt{(s_1^2+s_0^2)/2}}, $$ where subscripts 1 and 0 refer to the treated and comparison groups. For a binary covariate with proportions $p_1$ and $p_0$, a common convention is $$ \mathrm{SMD}_X=\frac{p_1-p_0}{\sqrt{\{p_1(1-p_1)+p_0(1-p_0)\}/2}}. $$ The sign indicates direction; balance thresholds are usually applied to the absolute value.

For a pre/post comparison, a useful convention is to calculate the denominator from the original, pre-adjustment groups and hold it fixed for both SMDs. This makes the before-and-after values comparable: a change in the SMD then reflects a change in the group difference, rather than a change in the scale used to standardize it. Other denominator conventions exist, especially in weighted analyses, so a report should state which one it uses. The worked examples below use the fixed pre-adjustment denominator.

What to Show Alongside the SMD

SMDs are dimensionless, which makes them useful for comparing differences across covariates measured in different units. They do not, however, replace the underlying summaries. A reader should be able to see what the groups looked like, not just a single standardized number. For continuous covariates, report group means and standard deviations, and consider medians and quantiles or a distribution plot when skewness, outliers, or tail differences matter. For binary and multi-category covariates, report the group proportions for each relevant category as well as their standardized differences.

The common rule of thumb $|\mathrm{SMD}|<0.10$ is a descriptive threshold, not a universal boundary between adequate and inadequate balance. If a report uses it, state it explicitly and identify post-adjustment covariates with $|\mathrm{SMD}|\geq0.10$. A stricter or otherwise justified threshold may be appropriate in a particular analysis, but it should not be chosen after inspecting results merely to make the diagnostics look favorable.

Balance diagnostics are descriptive comparisons, not hypothesis tests. A small p-value can reflect a modest difference in a large sample, while a large p-value can reflect limited precision in a small sample. Thus, “the groups did not differ significantly” is not a substitute for reporting the size of the covariate differences. Nor does good measured-covariate balance demonstrate that unmeasured confounding is absent or that the causal assumptions hold.

Worked Example: Reporting a Continuous Covariate Before and After Matching

Suppose a hypothetical study compares treated and comparison patients. Before matching, mean age is 54 years (standard deviation 12) in the treated group and 50 years (standard deviation 10) in the comparison group. After matching, the corresponding means are 52 and 51 years. For this illustration, the original pooled denominator is held fixed after matching.

First calculate the pre-adjustment denominator: $$ \sqrt{\frac{12^2+10^2}{2}} =\sqrt{\frac{144+100}{2}} =\sqrt{122} \approx 11.05\text{ years}. $$

The pre-adjustment SMD is $$ \frac{54-50}{11.05} =\frac{4}{11.05} \approx 0.362. $$ The mean age difference is 4 years, or about 0.362 original pooled standard deviations.

Using that same denominator after matching gives $$ \mathrm{SMD}_{\mathrm{post}} =\frac{52-51}{11.05} =\frac{1}{11.05} \approx 0.0905. $$ The post-matching mean difference is 1 year, or about 0.091 original pooled standard deviations. Under the stated $|\mathrm{SMD}|<0.10$ convention, age is below the threshold after matching. A concise report would still give the group means and standard deviations, the fixed-denominator convention, and the threshold; the SMD alone does not reveal whether the age distributions differ in spread or shape.

Binary and Multi-Category Covariates

For a binary covariate, such as a recorded history of smoking, the SMD compares group proportions. For a nominal covariate with several categories, one approach is to calculate an SMD for each level using that level’s proportions and the binary-proportion denominator. Report the proportions and level-specific SMDs together. The level-specific differences are related: because category proportions sum to one in each group, their signed differences sum to zero. They should not be interpreted as independent tests.

Worked Example: Reporting a Three-Level Covariate

Consider a hypothetical three-category baseline covariate. Before adjustment, the proportions in categories A, B, and C are 0.50, 0.30, and 0.20 among treated patients, and 0.60, 0.25, and 0.15 among comparison patients. After matching, they are 0.49, 0.28, and 0.23 versus 0.50, 0.27, and 0.23. Use each category’s pre-adjustment denominator for both pre- and post-matching SMDs.

For category A, the pre-adjustment denominator and SMD are $$ \sqrt{\frac{0.50(0.50)+0.60(0.40)}{2}} =\sqrt{0.245} \approx 0.4950, \qquad \frac{0.50-0.60}{0.4950}\approx -0.202. $$ After matching, the difference is $0.49-0.50=-0.010$, so the SMD is $$ \frac{-0.010}{0.4950}\approx -0.0202. $$

For category B, the pre-adjustment denominator is $$ \sqrt{\frac{0.30(0.70)+0.25(0.75)}{2}} =\sqrt{0.19875} \approx 0.4458. $$ Its pre-matching SMD is $(0.30-0.25)/0.4458\approx 0.1122$. After matching, the difference is $0.28-0.27=0.010$, giving an SMD of $0.010/0.4458\approx 0.0224$.

For category C, the pre-adjustment denominator is $$ \sqrt{\frac{0.20(0.80)+0.15(0.85)}{2}} =\sqrt{0.14375} \approx 0.3791. $$ Its pre-matching SMD is $(0.20-0.15)/0.3791\approx 0.1319$. After matching, the proportions are both 0.23, so the SMD is $0/0.3791=0$.

The largest absolute level-specific SMD fell from approximately 0.202 before matching (category A) to 0.0224 after matching (category B). All three post-matching values are below 0.10. A clear report would show the proportions for all categories in both groups and the level-specific differences, rather than only reporting the smallest or largest value.

Mean Balance Does Not Guarantee Distributional Balance

Two groups can have the same mean but very different variances, quantiles, or tails. This can happen even when the mean SMD is exactly zero. For continuous covariates, useful supplementary diagnostics include group standard deviations or variances, a variance ratio, selected quantiles, and distribution plots. The choice depends on the covariate and the features that could matter for the analysis. A single summary cannot establish that the entire distributions match.

Worked Example: A Zero SMD With Different Spreads

Suppose hypothetical post-weighting summaries for a continuous covariate show a mean of 10 units in both groups. Before weighting, both groups had standard deviations of 2 units, so the fixed SMD denominator is $$ \sqrt{\frac{2^2+2^2}{2}}=2. $$ The post-weighting SMD is $(10-10)/2=0$. By the mean criterion alone, the groups have no mean difference.

Now suppose the post-weighting standard deviations are 3 units in the treated group and 1.5 units in the comparison group. The post-adjustment variance ratio, defined here as treated variance divided by comparison variance, is $$ \frac{3^2}{1.5^2}=\frac{9}{2.25}=4. $$ The treated-group variance is four times the comparison-group variance, despite the zero mean SMD. A report should therefore not claim that the distributions are balanced based on this SMD alone. It should show the spread diagnostics and, where useful, quantiles or a plot. These summaries describe measured covariate distributions; they do not by themselves establish the validity of a causal estimate.

A Practical Reporting Sequence

A balance summary is most useful when its calculations and interpretation can be checked. The following sequence makes the reporting decisions explicit.

Name the comparison and covariates.
Identify the treated and comparison groups and report the covariates assessed, including important nonlinear terms or categories used in the analysis.

Show pre- and post-adjustment summaries.
Give group-specific means and standard deviations or proportions, as appropriate. Label whether post-adjustment summaries are matched-sample or weighted summaries.

Define the SMD calculation.
State the formula or denominator convention, whether it is held fixed across the pre/post comparison, and whether reported values are signed or absolute.

State the assessment threshold and exceptions.
If using a rule such as $|\mathrm{SMD}|<0.10$, name it and list every post-adjustment covariate at or above the threshold. Add spread or distributional diagnostics when mean balance is insufficient.

For example, a report could say: “Using the original pooled standard deviation as a fixed denominator, the absolute age SMD decreased from 0.362 to 0.091. We used 0.10 as a descriptive assessment threshold; age was below that threshold after matching. Post-matching variances and quantiles were also examined because mean balance alone does not describe the full age distribution.” If some covariates remain above the chosen threshold, name them and give their values rather than saying simply that balance was adequate.

Common Mistakes and Limits

Using a different denominator without explaining it. A pre/post change may then reflect both a changed difference and a changed scale. State the convention; a fixed pre-adjustment denominator makes the comparison especially transparent.
Treating 0.10 as a theorem. It is a common descriptive rule of thumb, not a guarantee of negligible confounding or a universal standard. Report the actual values and explain the chosen convention.
Using significance tests as the balance assessment. A p-value depends on sample size and does not directly communicate the magnitude of imbalance. Report descriptive differences and distributions instead.
Reporting only an overall maximum. A maximum can conceal which covariates remain imbalanced. Identify the covariates and give their post-adjustment summaries and SMDs.
Equating a small mean SMD with identical distributions. Differences in spread, skewness, tails, or category composition can remain. Pair mean-based measures with suitable summaries or plots.

Balance diagnostics assess measured covariates under the reported adjustment. They do not show that unmeasured confounding is absent, determine whether the adjustment was otherwise appropriate, or establish the causal interpretation of an effect estimate. A complete report presents the diagnostics as evidence about measured balance and describes their limitations plainly.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A covariate has a post-adjustment SMD of $-0.14$. Under a stated threshold of 0.10, does it meet the threshold? What should the report say?
Why can a fixed pre-adjustment SMD denominator help when comparing balance before and after adjustment?
Two groups have identical means but different standard deviations. What additional diagnostic could reveal this difference?
Why is a non-significant balance-test p-value not sufficient evidence of balance?
For a three-category covariate, why should the report show each level’s proportions rather than only one category’s SMD?

Solutions. (1) No. Its absolute SMD is $|-0.14|=0.14$, which is at or above 0.10; name the covariate and report its value and group summaries. (2) It keeps the scale constant, so the pre/post SMD comparison reflects the change in the group difference rather than a changed denominator. (3) Report group standard deviations or a variance ratio; quantiles or a distribution plot may also help. (4) The p-value depends on sample size and does not directly quantify the difference’s magnitude; descriptive standardized differences and distributional summaries are more informative. (5) Category proportions sum to one, so level differences are related, and a single level may not reveal imbalances in the others. Show all relevant proportions and level-specific summaries.