Weighted Standardized Mean Differences

Why Weighting Changes a Balance Diagnostic

A standardized mean difference (SMD) compares group means on a scale that is not tied to the measurement units of a covariate. When observations receive weights, the means and variances in that comparison must describe the weighted distributions, not the original unweighted samples. The resulting SMD answers a specific question: how far apart are the weighted group means, relative to a chosen standard deviation?

The weights determine which observations contribute more to each group’s summary. For example, treatment weights can reweight each treatment group toward a specified target population; sampling weights can make a sample represent a target population under the sampling design. Thus, a weighted SMD is not a generic property of the original data. It is tied to the weights and target distributions used to calculate it.

The variance definition above describes the weighted empirical distribution: each observation contributes in proportion to its weight, and the weights are divided by their sum. It is not an unbiased estimator of a superpopulation variance, nor is that its purpose in this descriptive balance diagnostic. Other SMD conventions use a different prespecified reference standard deviation. Whichever denominator convention is used, state it and apply it consistently when comparing balance summaries.

Multiplying every weight in one group by the same positive constant leaves that group’s weighted mean and variance unchanged, because the constant cancels from each ratio. Relative weights within the group, however, matter. Weights need not sum to the same total across groups: each group’s mean and variance use its own sum of weights.

What Distribution Do the Weights Represent?

For treatment weighting, the weight assigned to an observation is generally chosen to make each treatment group represent a target covariate distribution. For instance, with estimated propensity score \(e(X_i)=P(A_i=1\mid X_i)\), the familiar inverse-probability weights for an average treatment effect are \(1/e(X_i)\) for treated observations and \(1/[1-e(X_i)]\) for controls. For an average treatment effect among the treated, a common choice is weight 1 for treated observations and \(e(X_i)/[1-e(X_i)]\) for controls. These formulas illustrate why an SMD must be interpreted relative to a target: changing the weighting scheme can change the weighted group means and variances.

Sampling weights have a different source. A sampling weight commonly reflects how many units in a target population an observed unit represents. A weighted SMD then compares the covariate distributions represented by the sampled members of the two groups. If both treatment and sampling weights are used, the combined weight must be justified by the design and target of the analysis; simply multiplying arbitrary weights does not guarantee a meaningful target.

The calculation itself does not require a causal model. But to interpret treatment-weighted summaries as representing a particular causal target, the weights must be appropriate for that target. In causal applications, this includes requiring adequate positivity for the target population and a valid basis for the estimated weights. Large or effectively infinite weights can make a weighted summary unstable or undefined. Balance on measured covariates does not by itself establish that unmeasured confounding is absent.

Why the Weighted Variance Uses the Weighted Mean

The weighted variance summarizes dispersion around the center of the weighted empirical distribution. This center is the weighted mean, not the unweighted mean. One direct justification is that the weighted mean minimizes the total weighted squared deviation. Let \(W=\sum_i w_i\), \(\bar{x}_w=\sum_i w_ix_i/W\), and let \(c\) be any candidate center. Expanding around \(\bar{x}_w\) gives

The cross-product term vanishes because \(\sum_i w_i(x_i-\bar{x}_w)=0\). Since \(W(c-\bar{x}_w)^2\geq0\), the weighted sum of squared deviations is smallest at \(c=\bar{x}_w\). Dividing that minimized sum by \(W\) gives the descriptive weighted variance used above.

This derivation also explains why calculating deviations from the unweighted mean would describe a different center and would not be the variance of the weighted empirical distribution. For some sampling-design or model-based purposes, variance estimation requires additional corrections. Those inferential variance estimators are distinct from the descriptive weighted variance used here to standardize a balance difference.

Sampling Weights and the Target Represented

Checks, Comparisons, and Limitations

A reliable weighted balance calculation starts by making the target explicit. For each group, identify the weight attached to every observation, check that the total weight is positive, and then calculate the mean and variance using that same set of weights. Computing an unweighted mean or variance at either stage breaks the alignment between the summary and the weighted distribution.

• Check the weight application. Recalculate the weighted mean from the weighted sum and total weight. Confirm that observations with larger weights have the intended relative influence.
• Check the scale. A variance is in squared units, so the denominator takes a square root before it is used with a mean difference. The SMD itself has no measurement units.
• Check the denominator convention. The examples use the average of the two weighted variances. A different prespecified reference standard deviation can produce a different SMD; record the choice rather than comparing incompatible values.
• Check extreme weights. A few large weights can make a weighted summary depend heavily on a small number of observations. Inspect the weights and the observations carrying most of the total weight, and assess whether the target distribution remains supported by the data.

Weighted and unweighted SMDs are not competing calculations of one fixed quantity: they describe different distributions. An unweighted SMD describes the observed sample groups; a weighted SMD describes the groups after applying the specified reweighting. Matching and weighting also create different analysis samples or distributions, so the relevant balance summaries must reflect the design actually used.

A weighted SMD is a useful, compact diagnostic for a difference in means, but it is not a complete test of balance. Equal means can coexist with differences in variance, skewness, tails, or subgroups. A small SMD does not prove that all features of the covariate distributions are similar, and it does not establish causal identification. In addition, when weights are highly variable, the weighted SMD may be numerically precise-looking while being driven by a few observations. Pair it with inspection of the weights and other distributional diagnostics appropriate to the analysis.