Covariate Balance Diagnostics After Transport Weighting

Why Check Balance After Transport Weighting?

Transport weighting changes how much each source-study participant contributes to summaries intended to describe a target population. A weighted analysis is more credible when the weighted source covariates resemble the target covariates, especially for variables that modify treatment effects or predict outcomes. Balance diagnostics assess that resemblance; they do not establish that the treatment effect is transportable or that the assumptions behind weighting hold.

A useful diagnostic set combines standardized differences, plots of distributions, and effective sample size (ESS). These answer different questions. Standardized differences summarize selected contrasts, plots show more of the distribution, and ESS indicates how concentrated the weights are. None is a substitute for checking whether target covariate values are represented in the source.

The target summary may come from observed target-population data or from a representative target sample. If the target sample itself requires sampling weights, its summary must account for them. The examples below treat target summaries as known or correctly estimated, and use transport weights for the source participants.

Standardized Differences

A standardized mean difference (SMD) reports a mean difference in units of a chosen reference standard deviation. To make pre-weighting and post-weighting comparisons interpretable, choose and state the standardizer, and keep it fixed across those comparisons. Here, the continuous-covariate standardizer is the target-population standard deviation \(s_{T,X}\).

For a binary covariate represented by an indicator, use the target standard deviation of that indicator, \(\sqrt{p_T(1-p_T)}\). For a categorical covariate with several levels, compare each level's indicator proportion using its target standard deviation. A positive SMD means the weighted source summary is higher than the target summary; a negative SMD means it is lower. The SMD is not a percentage difference and does not have units of the original covariate.

For example, when a target proportion is 0.50, its indicator standard deviation is \(\sqrt{0.50(1-0.50)}=0.50\). A weighted-source proportion of 0.40 is 0.10 proportion points lower, which is \(0.10/0.50=0.20\) target standard deviations lower. The corresponding SMD, with weighted source minus target in the numerator, is \(-0.20\).

The standardizer matters. Some analyses use a pooled standard deviation or another prespecified reference scale. Those choices can be reasonable, but the definition must be explicit and consistent. SMDs are useful for comparing covariates measured in different units; they should not be read as tests of statistical significance. A threshold such as an absolute SMD below 0.1 is a common heuristic, not a universal guarantee of adequate balance.

Plots Reveal More Than Mean Balance

A mean can match while the distributions differ substantially. For continuous covariates, compare weighted-source and target histograms or density plots, and weighted empirical cumulative distribution functions (ECDFs). Check central location, spread, skewness, tails, and gaps in support. For categorical variables, compare weighted-source and target proportions for every level, not just one selected contrast. When scientifically relevant, inspect important transformations, thresholds, and interactions as well as the original covariates.

Use comparable axes and scales so differences are visible. A plot with a large mass in one region can hide a small but clinically important tail; ECDFs and displays focused on the tail can help. Two-dimensional plots or cross-tabulations can reveal joint covariate patterns that separate one-variable-at-a-time checks miss. Plotting is diagnostic, not a formal proof that all relevant aspects of the distributions match.

Effective Sample Size and Weight Concentration

The raw source sample size does not describe how much information remains when a few participants receive most of the total weight. A common weight-concentration summary is the Kish effective sample size. It is computed from the transport weights, not from the target sample size.

One justification comes from the variance of a weighted mean. Write \(q_i=w_i/\sum_j w_j\), so the normalized weights sum to 1. If outcomes are independent with common variance \(\sigma^2\), the variance of their weighted mean is \(\sigma^2\sum_iq_i^2\). Equating this with the variance \(\sigma^2/n_{\mathrm{eff}}\) of an unweighted mean based on \(n_{\mathrm{eff}}\) equally weighted observations gives \(n_{\mathrm{eff}}=1/\sum_iq_i^2\), which is the stated formula. This is a useful concentration summary, not a literal count of independent participants for every outcome or model.

How to Diagnose Problems and Choose Comparisons

These diagnostics are complementary to common alternatives. A hypothesis test for differences can depend strongly on sample size: a small discrepancy may be statistically detectable in a large sample, while a practically important discrepancy may not be detected in a small one. SMDs describe magnitude on a reference scale without that same sample-size dependence, but compress each comparison to a summary. Distribution plots retain information about shape and support, but require judgment and can be hard to compare if axes or bins differ. ESS summarizes weight concentration, but does not identify which covariates are imbalanced or where support is absent.

If diagnostics are concerning, possible responses include revisiting the weighting specification, using a different weighting or design strategy, or narrowing the target population to a region with adequate support. Each changes the analysis or its target and requires justification. Trimming extreme weights may reduce variance but can introduce bias or change the effective target; it does not create observations in unsupported regions. No adjustment should be presented as solving a positivity problem without examining what population is then represented.

Common Mistake and Limitations

Wrong reasoning: “The weighted means match and all reported SMDs are below 0.1, so the populations are balanced and transport is justified.” This conclusion overstates what the summaries establish. They cover only the covariates and contrasts examined; they can miss distributional differences, interactions, sparse regions, or a target tail absent from the source. Balance on measured covariates also cannot verify unmeasured assumptions or demonstrate that effects are exchangeable across populations.

Correction: Treat SMDs as one part of a diagnostic assessment. State the standardizer and examine the full distributions and support, weight concentration, and scientifically important joint patterns. Report remaining discrepancies and limitations. In particular, acceptable average balance does not establish overlap: weighting can match a mean by emphasizing observed source values while leaving target values unsupported.

A further limitation is that estimated target summaries and estimated weights have uncertainty. Simple balance summaries are descriptive and do not automatically account for that uncertainty. They also cannot tell whether a small residual imbalance is consequential for a particular treatment effect. Interpretation must return to the target population, the estimand, and the covariates needed for the transport argument.