Check Covariate Balance After Adjustment

Why Check Balance After Weighting?

Propensity weighting aims to make exposure groups comparable with respect to measured pre-exposure covariates. But calculating weights does not itself show that this aim has been met. A weighting model can be fitted and weights applied while important covariate differences remain, or while weighted distributions become unstable because a few observations have great influence. Balance checks provide evidence about what the adjustment accomplished.

The focus is on the covariates used to adjust the comparison, especially those selected as confounders from the causal question. Check their distributions before and after weighting. The resulting diagnostics describe measured covariate comparability in the weighted sample; they do not show that unmeasured confounding is absent or prove that a causal effect has been identified.

Standardized Differences

For a continuous covariate \(X\), first calculate the unweighted mean and sample standard deviation in each exposure group. Use these to form a pooled, pre-weighting standard deviation:

Here group 1 is the exposed group and group 0 is the unexposed group; \(n_1,n_0\) and \(s_1,s_0\) are their sample sizes and standard deviations. The unweighted SMD is \((\bar{x}_1-\bar{x}_0)/s_{\mathrm{pooled}}\). After weighting, calculate each group’s weighted mean and subtract them, but keep the same pre-weighting pooled standard deviation in the denominator. This fixed scale makes the before-and-after values directly comparable.

For a binary covariate, treat the proportion with the characteristic as a mean of zero-one values. Let \(\hat{p}_1\) and \(\hat{p}_0\) be the unweighted group proportions. The usual pre-weighting pooled scale is:

The binary SMD is the difference in proportions divided by this scale. For the weighted SMD, replace the proportions in the numerator with weighted proportions, retaining the unweighted denominator. For a categorical covariate with several levels, compare the weighted and unweighted proportions at each level, commonly by coding each level as a separate zero-one indicator.

The sign says which group has the larger mean or proportion. The magnitude, often summarized as an absolute SMD, describes the size of the difference on a standardized scale. Because an SMD is unitless, it can help compare imbalances across covariates measured in different units. It is a descriptive diagnostic, not a hypothesis test and not a universal pass-or-fail rule. In particular, a small SMD for a mean does not ensure similar variances, tails, or shapes.

Weighted Summaries and Distribution Plots

For an exposure group with weights \(w_i\), the weighted mean of a continuous covariate is \(\sum_i w_iX_i/\sum_i w_i\). The weighted proportion with a binary characteristic is \(\sum_i w_iI(X_i=1)/\sum_i w_i\), where \(I(X_i=1)\) equals one when the characteristic is present and zero otherwise. These summaries give each observation influence in proportion to its weight. Use the weights appropriate to the target population and the chosen exposure contrast; do not switch weighting schemes when making the before-and-after comparison.

Plots complement summaries by showing more of the distribution. For continuous covariates, compare group-specific histograms, density plots, or empirical cumulative distribution functions before and after weighting. For binary or categorical covariates, compare group proportions for each category. Show weighted distributions after adjustment, and label axes and groups clearly. Plotting a covariate against the propensity score can also help reveal areas with few observations from one group, but a covariate-balance plot is not a substitute for checking overlap.

A balance plot can display absolute SMDs for all covariates, with separate marks for before and after weighting. It makes patterns easy to scan: perhaps most imbalances shrink, while one covariate remains noticeably different. A reference line can be useful as a visual guide, but no single cutoff determines whether adjustment is adequate in every application. Interpret the size and pattern of the remaining differences in context.

Worked Diagnostics

Reading Residual Imbalance and Overlap

Residual imbalance means that some measured covariate distributions still differ after weighting. It can arise because the propensity model did not adequately represent the assignment patterns relevant to those covariates, because the weighting procedure or analysis sample was implemented incorrectly, or because the observed data provide limited overlap. A remaining SMD does not by itself identify which explanation applies. Review the model, variable coding, weights, and distribution plots before deciding what to do.

Poor overlap is especially important when people with certain covariate profiles are rare in one exposure group. Weighting can then give a few observations very large influence. Look for extreme or highly concentrated weights, sparse regions in group-specific covariate or propensity-score plots, and weighted distributions dominated by a small number of observations. Such patterns can make a comparison unstable even if some average SMDs appear small. A balance assessment and an overlap assessment inform one another, but they answer distinct questions: balance asks how similar the weighted covariate distributions are, while overlap asks whether both groups are represented across relevant profiles.

For a set of covariates, inspect each variable that matters for the adjustment strategy rather than reporting only an average balance score. Include nonlinear features or important categories where relevant; a close mean of a continuous variable does not ensure balance in its tails or in a clinically meaningful threshold. If the weighting targets a particular population, interpret balance with respect to that target and the corresponding weights.

Why the Diagnostics Matter—and a Common Mistake

Balance checks are an important part of assessing whether a propensity-weighted comparison has achieved its measured-covariate objective. They can expose a mismatch that a summary of the outcome alone would not reveal and can flag unstable weighting before drawing conclusions. They do not establish exchangeability: comparable measured covariates cannot demonstrate that all relevant confounders were measured and adequately handled. Nor do balance checks by themselves justify a causal interpretation; the causal identification assumptions and target population still matter.

Common mistake: “Every post-weighting SMD is below 0.1, so the groups are balanced and the causal effect is unbiased.” This reasoning treats a rule of thumb as a guarantee, considers only standardized mean differences, and extends a measured-covariate diagnostic to unmeasured confounding. A better conclusion is narrower: “The post-weighting differences in the reported measured covariate means and proportions are small on the chosen scales; plots show [describe what they show]. These diagnostics do not rule out unmeasured confounding or other limitations.” State the diagnostic results and the remaining assumptions separately.

Do not rely on baseline significance tests as a substitute for these checks. A p-value for a group difference depends on sample size as well as the size of the difference, whereas the purpose here is to describe covariate comparability. Report interpretable differences, plots, and weight behavior rather than treating a non-significant test as evidence of balance.