Assessing Covariate Balance After Adjustment

Why Assess Balance After Adjustment?

Matching and weighting aim to make the covariate distributions of trial participants and external controls more comparable. Balance diagnostics check whether that aim was achieved for the measured covariates used in adjustment. They should be computed both before and after adjustment: the first set describes the initial differences, and the second shows which differences remain.

This assessment concerns covariates, not outcomes. In an external-control analysis, the groups might be trial participants and patients selected from routine-care records. For clarity, let group 1 denote trial participants and group 0 external controls. Balance is assessed for each measured covariate using the observations retained or assigned positive weight by the adjustment procedure. The choice of target population matters: diagnostics should reflect the matched or weighted populations relevant to the intended comparison.

A covariate can have similar group means while differing substantially in spread, tail behavior, or category composition. No single summary establishes balance. Standardized differences, variance ratios, and distributional plots provide complementary views.

Definition: Covariate balance is similarity between the distributions of measured baseline covariates in the groups being compared, after the specified matching or weighting. It is a design diagnostic: balance on measured covariates does not establish balance on unmeasured covariates, remove measurement error, or prove that the groups are exchangeable.

Weighted Summaries and Standardized Differences

For a group \(g\), let \(w_{ig}\) be the nonnegative analysis weight for individual \(i\), and let \(x_{ig}\) be that individual's covariate value. The weighted mean and weighted variance used in these diagnostics can be defined as

\bar{x}_{g,w}=\frac{\sum_i w_{ig}x_{ig}}{\sum_i w_{ig}}, \qquad s^2_{g,w}=\frac{\sum_i w_{ig}(x_{ig}-\bar{x}_{g,w})^2}{\sum_i w_{ig}}.

These descriptive, population-style weighted moments use the total weight as the variance denominator. Multiplying all weights within a group by the same positive constant does not change either summary. With no weighting, set each observation's weight to 1. In a matched sample, use the weights or retained observations that define the matched comparison; do not calculate one group from a different target population than the other.

For a continuous covariate, a standardized mean difference (SMD) expresses the difference in group means in units of a standard deviation. A useful before-and-after comparison keeps the denominator fixed at its pre-adjustment pooled value:

\operatorname{SMD}_{x} = \frac{\bar{x}_{1}-\bar{x}_{0}} {\sqrt{(s^2_{1}+s^2_{0})/2}}.

Here, \(\bar{x}_{g}\) and \(s^2_g\) are the unweighted group mean and variance before adjustment. After adjustment, replace the numerator's group means with their weighted or matched means but retain the same pre-adjustment denominator. This convention makes the before-and-after standardized differences directly comparable: a change in the SMD then reflects a change in the mean difference, rather than a changing scale. State the convention used, since alternative denominators also occur in practice. The sign indicates which group has the larger mean; the magnitude indicates the size of the difference in standard-deviation units.

Worked Example: Continuous Covariate Before and After Weighting

In a hypothetical comparison, the baseline age summaries are: 100 trial participants with mean age 58 years and standard deviation 12 years, and 100 external controls with mean age 54 years and standard deviation 10 years. After weighting, the weighted means are 57 and 56 years, and the weighted variances are 121 and 110 years squared, respectively. Compute the continuous-covariate SMD before and after weighting and the post-weighting variance ratio.

Before weighting. The pooled variance scale is \((12^2+10^2)/2=(144+100)/2=122\) years squared, so its square root is approximately 11.05 years. Thus,

\operatorname{SMD}_{\text{before}} = \frac{58-54}{\sqrt{122}} = \frac{4}{11.05} \approx 0.362.

The trial participants' mean age is 0.362 pre-adjustment pooled standard deviations higher than the external controls' mean age.

After weighting. Keep the original 11.05-year denominator: \(\operatorname{SMD}_{\text{after}}=(57-56)/11.05\approx 0.0905\). The weighted mean difference is 1 year, or about 0.091 of the original pooled standard deviation. The variance ratio, defined here as group 1 divided by group 0, is \(121/110=1.10\). The weighted age variances are therefore similar, though not identical. The SMD and ratio describe different features: a small difference in means does not by itself imply equal spreads.

For a binary covariate, calculate the weighted prevalence in each group using the weighted mean of its 0/1 indicator. A common standardized difference uses a pooled Bernoulli standard deviation calculated from the two group prevalences:

\operatorname{SMD}_{\mathrm{binary}} = \frac{\hat{p}_1-\hat{p}_0} {\sqrt{\{\hat{p}_1(1-\hat{p}_1)+\hat{p}_0(1-\hat{p}_0)\}/2}}.

The same formula can be used with weighted prevalences after adjustment. As with continuous covariates, retaining the pre-adjustment denominator when comparing before and after is one defensible convention; be explicit about it. If a category is extremely rare, the Bernoulli standard deviation can be very small, making the SMD unstable or undefined when both prevalences are at a boundary. Report the prevalences as well as the standardized difference in such cases.

Worked Example: Balance on a Binary Risk Factor

Suppose a hypothetical baseline risk factor is present in 40% of 200 trial participants and 25% of 240 external controls. After weighting, its prevalence is 40% in trial participants and 38% in external controls. Calculate the SMD before and after adjustment.

Before adjustment. The pooled Bernoulli variance is \(\{0.40(0.60)+0.25(0.75)\}/2=(0.2400+0.1875)/2=0.21375\). Its square root is approximately 0.4623, giving \(\operatorname{SMD}_{\text{before}}=(0.40-0.25)/0.4623\approx 0.324\).

After weighting. Keep the pre-adjustment denominator, 0.4623: \(\operatorname{SMD}_{\text{after}}=(0.40-0.38)/0.4623\approx 0.0433\). The weighted prevalence difference is 0.02, or 2 percentage points, and the standardized difference is small. The relevant calculation uses the weighted proportions, not the original group sizes as if the weighted observations were an unweighted sample.

Variance Ratios and Distributional Checks

For a continuous covariate, define the variance ratio as the variance in group 1 divided by the variance in group 0:

\operatorname{VR}_{x}=\frac{s^2_{1}}{s^2_{0}}.

After weighting or matching, use the corresponding weighted or matched variances. A ratio of 1 indicates equal variances; ratios above or below 1 indicate greater variance in group 1 or group 0, respectively. The ratio has no units. It can reveal unequal spread that an SMD misses, but it does not describe skewness, multimodality, or tail differences. If the denominator variance is zero, the ratio is undefined; report the group distributions directly rather than forcing a numerical ratio.

Distributional plots are therefore an important complement. For continuous covariates, compare weighted histograms or density plots, and consider weighted empirical cumulative distribution functions (ECDFs). An ECDF shows, at each value, the proportion of total group weight at or below that value. Compare the location, spread, tails, and shape of the curves. For binary and multi-level categorical variables, compare weighted category proportions, preferably in a common bar chart or table. A plot can expose a mismatch that is hidden by a mean or a single variance ratio.

Worked Example: A Multi-Level Categorical Covariate

Consider hypothetical clinical sites classified as A, B, or C. Before adjustment, the proportions are 0.50, 0.30, and 0.20 among trial participants, and 0.30, 0.50, and 0.20 among external controls. After weighting, the external-control proportions are 0.48, 0.32, and 0.20; the trial proportions remain 0.50, 0.30, and 0.20. Calculate category-specific standardized differences before and after adjustment.

Treat each category as a 0/1 indicator and apply the binary SMD formula. For category A before adjustment, the denominator is \(\sqrt{\{0.50(0.50)+0.30(0.70)\}/2}=\sqrt{0.2300}\approx0.4796\). Thus, its SMD is \((0.50-0.30)/0.4796\approx0.417\). For category B, the denominator is also \(\sqrt{0.2300}\), and the SMD is \((0.30-0.50)/0.4796\approx-0.417\). For category C, the difference and SMD are 0.

After weighting, category A has a difference of 0.02 and denominator \(\sqrt{\{0.25+0.48(0.52)\}/2}=\sqrt{0.2498}\approx0.4998\), so its SMD is approximately 0.0400. For category B, the denominator is \(\sqrt{\{0.21+0.32(0.68)\}/2}=\sqrt{0.2138}\approx0.4624\), giving \((0.30-0.32)/0.4624\approx-0.0433\). Category C remains at 0. The weighted category proportions and these small category-specific differences indicate much closer site composition. Because the category proportions sum to 1, their differences are dependent; the set of proportions and a plot are more informative than treating each SMD as an independent test.

How to Conduct and Interpret the Assessment

Calculate diagnostics for all important baseline covariates identified for adjustment, not only those with large initial differences. Include clinically important transformations and interactions where imbalance in those features could matter; for example, a similar average age does not ensure a similar proportion above a clinically relevant age threshold. Review balance before and after adjustment using the same covariate definitions and the same denominator convention for standardized differences.

Specify the comparison.
Identify the groups, the target population represented after matching or weighting, and the weights or matched observations that define it.

Summarize each covariate.
For continuous covariates, report group means and variances; for categorical covariates, report category proportions. Show the unadjusted and adjusted summaries.

Compare complementary diagnostics.
Calculate standardized differences, use variance ratios for continuous covariates, and inspect plots for distributional differences.

Investigate remaining discrepancies.
Check coding, missingness, weighting or matching implementation, and covariate forms. If a material mismatch remains, report it and reconsider the adjustment rather than declaring balance from one summary.

An absolute SMD below 0.1 is often used as a rough warning threshold for a small mean or proportion difference; it is a convention, not a guarantee of adequate balance. Context matters, and even a small SMD for a strong confounder may warrant attention. Similarly, there is no universal variance-ratio cutoff that establishes acceptable balance. Report the diagnostics themselves rather than turning any threshold into a pass/fail proof.

Do not use ordinary hypothesis tests as the primary balance criterion. Their p-values depend on sample size and, after weighting, can be difficult to interpret using conventional unweighted formulas. A large p-value does not demonstrate balance, and a small p-value does not quantify the size or importance of an imbalance. Standardized differences describe magnitude on a common scale; plots reveal distributional features. The earlier material on Diagnosing Positivity and Covariate Overlap addresses whether comparable covariate patterns are available, whereas balance diagnostics assess the distributions produced by the chosen adjustment. Neither diagnostic substitutes for the other.

Derivation and Limits of the Standardized Difference

The standardization follows directly from changing the units of a mean difference. If a continuous covariate is converted from \(x\) to \(y=a+bx\), where \(b>0\), then the difference in means becomes \(b(\bar{x}_1-\bar{x}_0)\), and the pooled standard deviation becomes \(b\sqrt{(s_1^2+s_0^2)/2}\). Therefore,

\frac{\bar{y}_1-\bar{y}_0} {\sqrt{(s_{y,1}^2+s_{y,0}^2)/2}} = \frac{b(\bar{x}_1-\bar{x}_0)} {b\sqrt{(s_{x,1}^2+s_{x,0}^2)/2}} = \operatorname{SMD}_{x}.

Thus, the SMD is unchanged by a change in measurement units, such as years to months, and is useful for comparing covariates measured on different scales. The absolute value is unchanged if the groups are reversed; the sign reverses. This derivation also shows why a fixed pre-adjustment denominator is useful for comparisons over time: if the denominator were recalculated after adjustment, changes in spread could alter the SMD even when the mean difference did not change.

An SMD is still only a summary of a mean or proportion difference. Two groups can have the same mean and variance but different distributional shapes. Matching may also discard observations or weighting may emphasize a subset of the data; good measured-covariate balance does not by itself show that the resulting comparison is precise. Effective sample size is a separate diagnostic. Finally, adjustment diagnostics cannot correct unmeasured confounding, systematic differences in covariate measurement, or outcome differences caused by other features of the data sources.

Common Mistakes and Practical Checks

Calling a nonsignificant test proof of balance. A p-value is affected by sample size and does not measure the size of an imbalance. Report standardized differences and inspect distributions instead.
Checking means alone. Similar means can conceal different variances, tails, or category proportions. Add variance ratios and distributional plots.
Changing the SMD denominator between before and after. This makes the comparison harder to interpret. State the denominator convention and use a fixed pre-adjustment scale when directly comparing changes.
Reporting only a multi-level category's single summary. A multi-level variable needs category proportions and category-specific indicators or a plot; one overall number can hide which levels differ.
Equating balance with absence of confounding. Balance applies to measured covariates included in the assessment. It does not demonstrate balance on unobserved factors or establish causal comparability.

As a practical check, confirm that every reported weighted summary uses the intended weights, that the same covariate coding is used before and after adjustment, and that the direction of each difference is clearly defined. Interpret diagnostics together and in clinical context; do not tune an adjustment solely to improve a numerical threshold while ignoring the target population or the assumptions behind the comparison.

Check Your Understanding

Answer each question, then compare with the brief solutions.

A continuous covariate has pre-adjustment group means of 10 and 8, with standard deviations 3 and 4. What is its signed SMD if group 1 is the group with mean 10?
After adjustment, a binary covariate has weighted prevalences 0.30 in group 1 and 0.28 in group 0. What does the difference tell you, and what else should you report?
A continuous covariate has a post-adjustment variance ratio of 1.25, defined as group 1 variance divided by group 0 variance. Interpret it.
Why can equal means fail to establish distributional balance?
Why is a large p-value from a balance test not evidence that the covariates are balanced?

Solutions. (1) The fixed pooled standard deviation is \(\sqrt{(3^2+4^2)/2}=\sqrt{12.5}\approx3.536\). The SMD is \((10-8)/3.536\approx0.566\), so group 1 has the higher mean by about 0.566 pooled standard deviations. (2) The weighted prevalence difference is \(0.30-0.28=0.02\), or 2 percentage points. Report both group prevalences and, if using the binary SMD, its denominator convention and value; also inspect other relevant covariates. (3) Group 1's weighted variance is 1.25 times group 0's weighted variance, or 25% greater. This says nothing by itself about differences in means or distributional shape. (4) Equal means do not require equal variances, tails, or shapes; compare variance summaries and plots as well. (5) A p-value depends on sample size and does not quantify the magnitude of the difference. Balance should be assessed using effect-size summaries and distributional checks, not a nonsignificance conclusion.