Diagnosing Balance Under Extreme Weights

Why Balance Needs a Weight-Concentration Check

A weighted mean can look reassuring even when only a few observations contribute most of the information. That matters because balance diagnostics describe the weighted sample, while extreme weights can make that sample depend heavily on a small number of units. The resulting summary may be mathematically correct but fragile: small changes to those units, their measured covariates, or the weights can substantially change the result.

This tutorial focuses on diagnosing that fragility. The aim is not to replace weighted standardized mean differences (SMDs), variance ratios, or distributional checks, but to interpret them alongside summaries of how the weights are distributed. Weight concentration does not by itself prove that an analysis is biased, nor does a lack of concentration establish that a causal estimate is valid.

Definition: For group $g$, let $w_i\geq 0$ be the analysis weight for unit $i$, and let $X_i$ be a covariate. The normalized weighted mean is $$ \bar{X}_{w,g}=\frac{\sum_{i\in g} w_iX_i}{\sum_{i\in g}w_i}. $$ The normalized contribution of unit $i$ is $q_i=w_i/\sum_{j\in g}w_j$. These contributions sum to 1, and $\bar{X}_{w,g}=\sum_{i\in g}q_iX_i$.

For example, with inverse-probability weights for the average treatment effect, a common choice is $w_i=1/e_i$ for treated units and $w_i=1/(1-e_i)$ for untreated units, where $e_i=P(A_i=1\mid X_i)$ is the estimated propensity score. These weights can be large when a treated unit has a small estimated propensity score or an untreated unit has a large one. Other weighting schemes target other populations; diagnose the weights actually used, separately within each treatment group.

The calculations below require nonnegative finite weights and a positive total weight in each group being summarized. A zero weight contributes nothing. If all weights are zero, the weighted mean is undefined. Multiplying every weight in a group by the same positive constant leaves its normalized weighted mean and weight shares unchanged.

Summarize How Much the Weights Concentrate

No single concentration measure is sufficient in every setting. A useful first check is the largest normalized contribution, $\max_i q_i$. Also inspect the share held by the largest few weights, such as the top 5 or top 10, and compare the distribution of weights across treatment groups. These summaries reveal whether a small set of units has substantial influence.

Definition: The Kish effective sample size for a group with nonnegative weights is $$ n_{\mathrm{eff}}=\frac{\left(\sum_{i\in g}w_i\right)^2}{\sum_{i\in g}w_i^2} =\frac{1}{\sum_{i\in g}q_i^2}. $$ It is a measure of weight concentration, not the number of units literally present in the data.

The effective sample size (ESS) equals the actual group size when all weights are equal. It becomes smaller as weights become more unequal. The ESS helps assess whether a weighted mean is supported by many similarly contributing observations or by a smaller effective set. It is not a universal threshold for declaring an analysis adequate, and it does not account for every feature of the covariates or outcome.

Why Effective Sample Size Measures Concentration

A short derivation explains the ESS interpretation. Suppose, just for this justification, that the observed covariate values are independent with a common variance $\sigma^2$, and treat the weights as fixed. The variance of the weighted mean is then

\operatorname{Var}(\bar{X}_{w,g}) =\operatorname{Var}\left(\sum_{i\in g}q_iX_i\right) =\sum_{i\in g}q_i^2\operatorname{Var}(X_i) =\sigma^2\sum_{i\in g}q_i^2 =\frac{\sigma^2}{n_{\mathrm{eff}}}.

An unweighted mean of $m$ independent observations with the same variance has variance $\sigma^2/m$. Equating these expressions gives $m=1/\sum_iq_i^2=n_{\mathrm{eff}}$. This derivation is an analogy under its stated assumptions, not a general variance formula for causal estimators: it omits, among other things, uncertainty from estimating the weights and dependence or clustering in the data.

Worked Example: A Near-Zero SMD With Concentrated Weights

Consider hypothetical covariate values for four treated and four untreated units. The treated weights are $1,1,1,6$; the untreated weights are all 1.

Group	Covariate values	Weights
Treated	0, 1, 2, 10	1, 1, 1, 6
Untreated	4, 6, 8, 10	1, 1, 1, 1

The treated weighted mean is $(1(0)+1(1)+1(2)+6(10))/(1+1+1+6)=63/9=7$. The untreated weighted mean is $(4+6+8+10)/4=7$. Thus, the difference in weighted means is zero, so the weighted SMD is zero whenever its standardizing denominator is defined and positive.

But the treated weights are concentrated. The largest normalized contribution is $6/9=0.667$, and $n_{\mathrm{eff}}=9^2/(1^2+1^2+1^2+6^2)=81/39=2.077$, despite four treated units in the data. For comparison, the untreated ESS is 4. The descriptive weighted variances, using the sum of weights as denominator, are $164/9=18.22$ for the treated group and $20/4=5$ for the untreated group, a variance ratio of $18.22/5=3.644$. The equal means therefore do not show that the distributions are alike, and the treated mean depends substantially on the unit with covariate value 10.

Worked Checks for Sensitivity to Individual Units

A useful diagnostic question is: Would the summary change materially if the largest-weight observation were absent? This is not a proposal to discard that unit. It is a way to see whether the reported balance summary is sensitive to one observation. Also compare the full weighted covariate distribution with its unweighted counterpart and inspect other diagnostics such as variance ratios, quantiles, and empirical distribution comparisons.

Worked Example: One Observation Dominates a Weighted Mean

Suppose a hypothetical group has covariate values $0,0,1,2,10$, with weights $1,1,1,1,16$. The total weight is 20, so the weighted mean is $$ \bar{X}_w=\frac{1(0)+1(0)+1(1)+1(2)+16(10)}{20} =\frac{163}{20}=8.15. $$

The largest weight accounts for $16/20=0.80$, or 80% of the total weight. The ESS is $20^2/(1+1+1+1+256)=400/260=1.538$, although five units were observed. If the largest-weight unit is omitted solely for a sensitivity calculation, the remaining weighted mean is $(0+0+1+2)/4=0.75$. The large change from 8.15 to 0.75 shows that the full weighted mean is highly dependent on that one unit. It does not justify removing the unit from the primary analysis; it signals a need to investigate the weights, the unit's covariates, and the support of the propensity scores.

In balance assessment, do this kind of check separately by treatment group and for important covariates. A small overall SMD can coexist with a large maximum weight share or low ESS. Conversely, a low ESS does not tell you which covariate is imbalanced: it describes the weights, not the covariate distributions. Both kinds of information are needed.

Worked Example: What Capping Weights Changes

Return to treated covariate values $0,1,2,10$ and weights $1,1,1,6$. As an illustrative sensitivity calculation, cap the largest weight at 3, giving weights $1,1,1,3$. The capped weighted mean is $$ \bar{X}_{w,\mathrm{cap}}=\frac{1(0)+1(1)+1(2)+3(10)}{1+1+1+3} =\frac{33}{6}=5.5. $$

The uncapped mean was 7. The capped weights have ESS $6^2/(1+1+1+9)=36/12=3$, compared with $81/39=2.077$ before capping. Thus capping reduces concentration and shifts this covariate mean by $-1.5$ units. The calculation illustrates a trade-off, not an automatic correction: changing weights changes the weighted population being summarized and may alter balance for other covariates. A capped analysis should be treated as a sensitivity analysis or as part of a clearly specified alternative design, with the resulting balance and target population interpreted accordingly.

Diagnostics, Alternatives, and Limits

A practical diagnosis combines several views rather than relying on one reassuring number. Report group-specific ESS and maximum weight share; consider the share in the largest few weights; examine weighted covariate means and distributional diagnostics; and check how important summaries change under leave-one-out calculations. Very small estimated propensity scores or values near 1 can help explain large inverse-probability weights, but the weights and scores should be checked against the analysis specification and observed data.

The main alternatives answer different questions. A weighted SMD summarizes a difference in means on a standardized scale; it can be near zero even when weights are concentrated or distributions differ. A variance ratio, quantile comparison, or empirical distribution diagnostic can reveal features that a mean difference misses, but none directly measures how many observations carry the weighted summary. ESS and maximum-share checks assess concentration, but do not establish covariate balance. Weight capping can reduce concentration, while changing the weighting target or using a different design may be preferable in some applications; each choice has consequences for the estimand and balance and must be justified rather than selected simply to improve a diagnostic.

Extreme weights often signal limited overlap: some observed covariate patterns make one treatment group difficult to represent using the other. They can also arise from propensity-score model problems or data errors. Investigate before choosing a remedy. Even excellent measured-covariate balance and acceptable concentration do not verify conditional exchangeability, correct measurement, or other assumptions needed for a causal interpretation. A balance diagnostic is evidence about measured covariates under the chosen weights, not proof of all causal assumptions.

Common pitfall: “The weighted SMD is near zero, so the weights are safe.” This reasoning checks the difference in means but ignores how that mean was formed. Also inspect weight shares and ESS, then look for distributional differences and sensitivity to high-weight units. A reassuring mean balance statistic can coexist with unstable or concentrated weighting.

Check Your Understanding

Try each question, then compare with the solutions below.

Three units have weights $1,1,4$. What is the largest normalized contribution and the ESS?
Two groups have the same weighted covariate mean. Does that establish that the weights are not concentrated? Explain.
A group has weights $2,2,2,2$. Calculate its ESS and state what it indicates about concentration.
In a sensitivity calculation, capping weights changes a covariate mean. Does this establish that the capped analysis is preferable? Why or why not?

Solutions. (1) The total weight is 6, so the largest share is $4/6=0.667$. The ESS is $6^2/(1+1+16)=36/18=2$, below the three observed units. (2) No. Equal weighted means describe one feature of the covariate distributions, not how weight is allocated. Calculate weight shares and ESS, and examine distributional diagnostics. (3) The ESS is $8^2/(2^2+2^2+2^2+2^2)=64/16=4$, equal to the observed group size because all weights are equal. (4) No. Capping changes the weights and can change both the target population and balance. Compare the capped and uncapped diagnostics and state the target and rationale before preferring either analysis.