Effective Sample Size Under High-Dimensional Weights

Why Counted Observations Can Overstate Information

Propensity weighting can retain every eligible person in an analysis while giving some people far more influence than others. A nominal sample of hundreds may therefore contain substantially less information for a weighted comparison than its row count suggests. This issue can be especially consequential with high-dimensional propensity scores: many measured features may help characterize treatment assignment, yet the resulting weights can be highly unequal when some treatment patterns are rare or overlap is weak.

The effective sample size (ESS) summarizes the information loss due to unequal weights in a particular weighted sample. It is a useful diagnostic alongside the overlap and weight-instability checks described in the earlier material on overlap and weight instability. ESS does not establish that confounding has been controlled, nor does it directly give the standard error of a causal estimate.

Deriving the Measure

Suppose \(Y_1,\ldots,Y_n\) are independent observations with common mean \(\mu\) and common variance \(\sigma^2\), and the weights are fixed, nonnegative, and not all zero. The weighted mean is \(\bar{Y}_w=\sum_i w_iY_i/\sum_i w_i\). Independence gives its variance directly:

An equally weighted sample of size \(m\) under these same conditions has mean variance \(\sigma^2/m\). Equating the two variance expressions yields \(m=n_{\mathrm{eff}}\). This derivation explains the measure, and also its limits: unequal outcome variances, dependence between observations, estimated weights, or outcome-model adjustment can make the actual variance differ from the simple equal-variance calculation.

To see the concentration interpretation, define each normalized weight as \(q_i=w_i/\sum_j w_j\). Then \(\sum_i q_i=1\), and \(n_{\mathrm{eff}}=1/\sum_i q_i^2\). The sum of squared normalized weights is a concentration index: it is large when a few observations carry much of the total weight. Equal weights give \(q_i=1/n\), so \(n_{\mathrm{eff}}=n\); increasingly unequal weights reduce ESS. For nonnegative weights, \(1\le n_{\mathrm{eff}}\le n\).

A second useful expression compares ESS with the coefficient of variation of the weights. Let \(\bar{w}=\sum_i w_i/n\) and define the empirical weight variance using divisor \(n\) as \(v_w=\sum_i(w_i-\bar{w})^2/n\). Since \(\sum_iw_i^2=n(v_w+\bar{w}^2)\), substitution gives \(n_{\mathrm{eff}}=n/(1+\mathrm{CV}_w^2)\), where \(\mathrm{CV}_w^2=v_w/\bar{w}^2\). Thus, weight dispersion and ESS express related information in different forms; the stated divisor matters when calculating the relationship.

Calculate ESS for the Relevant Weighted Samples

In a binary-treatment analysis, calculate ESS separately for the treated and control observations using their respective analysis weights. For example, with \(A_i=1\) for treated and propensity score \(e_i=P(A_i=1\mid X_i)\), unstabilized inverse-probability weights for the average treatment effect are \(1/e_i\) for treated people and \(1/(1-e_i)\) for controls. Stabilized versions multiply weights within an arm by a common factor; this does not change that arm’s ESS. In either case, arm-specific ESS reveals whether one side of the comparison is particularly concentrated.

This calculation assumes the weights are nonnegative, finite, and have a positive sum in the set being summarized. A subgroup ESS is calculated by restricting to that subgroup and using the weights of its members. The resulting value describes weight concentration in that subgroup; it should not be confused with the subgroup’s raw count or with the overall ESS.

ESS becomes more informative when reported with subgroup summaries. Subgroups can reveal a weak point that an overall average conceals: a group may have a substantial raw count but very low ESS after weighting. For each important subgroup, examine the raw count, the sum of weights, ESS, and concentration indicators such as the largest normalized weight. In a treatment comparison, do this separately by arm as well. Small subgroup ESS warns that estimates for that subgroup may be unstable, but the exact uncertainty still requires an appropriate variance calculation.

Diagnostics, Alternatives, and Limits

Use ESS as one part of a diagnostic picture, not as a pass-or-fail rule. A very small ESS relative to the raw count signals weight concentration and potential precision loss. Examine the weight distribution, largest normalized weights, and ESS within relevant arms and subgroups. Also assess covariate balance after weighting: a favorable ESS does not show that measured covariates are balanced, and good balance does not ensure that the weights are stable. In high-dimensional settings, overall summaries can hide residual imbalance in particular feature blocks, so interpret ESS alongside the balance diagnostics appropriate to the analysis.

Several alternatives answer related but different questions. The raw sample size counts observations without considering their weights. The maximum normalized weight identifies the single most influential observation, while ESS summarizes concentration across all observations. The weight coefficient of variation gives another summary of dispersion and is algebraically linked to ESS under the definition above. None of these measures replaces a variance estimator for the target effect. For example, an estimated standard error must account for the estimator and, when relevant, the propensity-score estimation procedure and dependence structure.

Truncating or capping extreme weights can increase ESS by reducing their influence, but it changes the weights and therefore generally changes the weighted analysis. It is not a free precision improvement: it may trade reduced variance for altered bias or a different approximation to the intended target. Compare the resulting balance and effect estimates, describe the chosen rule, and avoid choosing a threshold solely because it produces a preferred ESS. Alternative weighting targets can also change the weight distribution, but their estimands and assumptions must be considered rather than treating a higher ESS as automatically preferable.

The Kish calculation is most directly justified for a weighted mean of independent, equal-variance observations with fixed weights. In causal analyses, weights are often estimated from observed covariates, and the outcome variance may vary across people. Clustering or repeated observations also violate the simple independence setup. Accordingly, the number should be described as a weight-based effective sample size, not as the actual number of independent observations, a guarantee of a particular confidence-interval width, or evidence that the causal assumptions hold.

A Practical Reporting Sequence