Which Weighted Populations Are Being Compared?
A balance summary is meaningful only after specifying the populations it describes. In a survey-weighted treatment comparison, two kinds of weights can enter: survey weights, which account for how sampled units represent a target population, and treatment-adjustment weights, which reweight treated or control units to target a common covariate distribution. These weights serve different purposes. Survey weights alone do not generally make treatment groups comparable.
Let \(A_i\) indicate treatment for unit \(i\), with \(A_i=1\) for treated and \(A_i=0\) for control. Let \(X_i\) denote a baseline covariate, and let \(d_i\) be a survey weight, often based on the inverse probability of selection and possibly adjusted for nonresponse or calibration. A survey-weighted summary within group \(a\) describes that group after accounting for the survey design. It does not, by itself, describe the same covariate distribution in both treatment groups.
When \(w_i=d_i\), the comparison is between the survey-weighted treated and control groups in the target population represented by the survey weights. Their summaries may differ because treatment is associated with \(X\), by chance or by design. When a treatment-adjustment factor \(q_i\) is also used, the analysis weight is \(w_i=d_iq_i\). The resulting comparison has a different interpretation: the treatment factors are intended to reweight the groups toward a specified target population.
Weighted Mean Differences and Standardized Differences
For a continuous covariate, a basic diagnostic is the difference in weighted means, \(\bar X_{1,w}-\bar X_{0,w}\), reported in the covariate’s units. A standardized mean difference (SMD) expresses that difference relative to a reference standard deviation. One possible convention is to use the square root of the average of the two groups’ weighted variances:
This definition uses the same analysis weights for each group’s mean and variance. Other reference-SD conventions are also used, so an analysis should state which convention it uses and apply it consistently when comparing specifications. The SMD is unitless; the unstandardized difference retains the original units. Neither summary alone establishes that the full covariate distributions are similar.
For a binary covariate \(X\), its weighted mean is the weighted proportion with \(X=1\). The weighted variance under the descriptive definition above is \(\hat p_w(1-\hat p_w)\). Thus the same SMD formula can be used, with the two weighted proportions and their corresponding variances.
Worked Example: Survey-Weighted Age Balance
Consider a hypothetical sample with survey weights and ages shown below. First use the survey weights alone to compare the represented treated and control groups.
| Group | Survey weight \(d_i\) | Age (years) |
|---|---|---|
| Treated | 2 | 20 |
| Treated | 1 | 40 |
| Control | 1 | 22 |
| Control | 3 | 38 |
The treated weighted mean age is \((2(20)+1(40))/(2+1)=80/3=26.67\) years. The control weighted mean is \((1(22)+3(38))/(1+3)=136/4=34.00\) years. The difference, treated minus control, is \(-7.333\) years.
For the treated group, the weighted variance is \(\{2(20-26.667)^2+1(40-26.667)^2\}/3=88.889\) years squared. For controls it is \(\{1(22-34)^2+3(38-34)^2\}/4=48.00\) years squared. The reference SD is \(\sqrt{(88.889+48.00)/2}=8.273\) years, so the SMD is \(-7.333/8.273=-0.886\). These survey-weighted groups differ in mean age by 7.333 years, or 0.886 reference SDs. This is a comparison of the represented treated and control groups, not proof that either group represents the same covariate population as the other.
Combining Survey and Treatment Weights
For an average treatment effect (ATE) target, the usual inverse-probability treatment factors are \(1/e(X_i)\) for treated units and \(1/[1-e(X_i)]\) for controls, where \(e(X)=P(A=1\mid X)\) is the propensity score. Combining these with survey weights gives analysis weights \(d_i/e(X_i)\) for treated units and \(d_i/[1-e(X_i)]\) for controls. The intended target is the survey-represented population, reweighted so that both treatment groups represent its covariate distribution.
This interpretation relies on conditions. The survey weights must appropriately account for selection into the observed sample for the target population of interest. Treatment probabilities must be correctly specified or consistently estimated for the balancing argument below. Positivity requires \(0<e(X)<1\) in the target population, with adequate probability of each treatment for covariate patterns being compared. These weighting formulas do not, by themselves, establish that treatment assignment is unconfounded for an outcome analysis; that is a separate causal assumption.
The balancing justification applies to more than means. It applies to any measurable function of the covariates, including indicators of sets, which identify the covariate distribution.
To verify the first identity, conditional on \(X\), the expected value of \(I(A=1)\) is \(e(X)\). Therefore \(E[I(A=1)h(X)/e(X)\mid X]=h(X)e(X)/e(X)=h(X)\). For controls, \(E[I(A=0)\mid X]=1-e(X)\), so the second expression equals \(h(X)[1-e(X)]/[1-e(X)]=h(X)\). Taking \(h(X)\) to be the indicator of any covariate set shows that the two weighted groups have the same target covariate distribution in expectation. In a finite sample, estimated weights and random variation mean balance will generally be approximate, not exact.
Worked Example: A Binary Covariate With Combined Weights
In a hypothetical analysis, the following combined weights \(d_iq_i\) are used. The treatment-adjustment factor \(q_i\) has already been applied to the survey weight \(d_i\). Calculate and interpret the weighted balance in a binary baseline covariate \(X\).
| Group | Covariate value \(X\) | Combined weight \(d_iq_i\) |
|---|---|---|
| Treated | 1 | 8 |
| Treated | 0 | 4 |
| Control | 1 | 3 |
| Control | 0 | 9 |
The treated weighted proportion with \(X=1\) is \(8/(8+4)=0.6667\); the control weighted proportion is \(3/(3+9)=0.2500\). Their difference is \(0.4167\), or 41.67 percentage points. The weighted variances are \(0.6667(1-0.6667)=0.2222\) and \(0.2500(1-0.2500)=0.1875\). The reference SD is \(\sqrt{(0.2222+0.1875)/2}=0.4526\), giving \(\operatorname{SMD}_w=0.4167/0.4526=0.9206\).
This example shows how to calculate balance from the weights actually used. It does not show good balance: the weighted prevalence remains substantially different. A correctly motivated weighting scheme does not guarantee close balance in every finite sample, especially with estimated treatment probabilities, limited overlap, or influential weights. The group-specific weighted proportions and the SMD should be reported as diagnostics, not replaced by a claim that weighting necessarily succeeded.
Weighting Changes the Target
The difference between survey weights and combined weights is visible even within a single treatment group. Survey weights represent the survey target; adding a treatment factor changes how that group contributes to a treatment-comparison target. For example, inverse-probability weighting for the ATE multiplies control survey weights by \(1/[1-e(X)]\), while treated survey weights are multiplied by \(1/e(X)\). These are not simply more precise versions of the original survey weights.
Worked Example: A Control Mean Before and After Treatment Weighting
Suppose three hypothetical control units have covariate values \(X=0,1,2\), survey weights \(d_i=1,2,1\), and propensity scores \(e(X_i)=0.20,0.60,0.80\), respectively. Compare the survey-weighted mean with the control mean using the ATE treatment factors.
With survey weights alone, the mean is \(\{1(0)+2(1)+1(2)\}/(1+2+1)=4/4=1.000\). The control ATE factors are \(1/(1-e)=1.25,2.50,5.00\). The combined weights are therefore \(1.25,5.00,5.00\), and the combined weighted mean is \(\{1.25(0)+5(1)+5(2)\}/(1.25+5+5)=15/11.25=1.333\).
The shift from 1.000 to 1.333 reflects a change in the weighted distribution of controls, not a change in the observed \(X\) values. In an ATE analysis, the corresponding treated group is also reweighted, and both weighted groups are intended to represent the survey target population. The control calculation alone cannot establish that they do so; balance must be checked across groups.
Diagnostics, Alternatives, and Limitations
A useful diagnostic report states the target population, the survey-weight definition, the treatment-weight formula and estimand, and the convention used for each summary. For each covariate, inspect weighted means or proportions and SMDs, alongside distributional summaries or plots where relevant. Earlier material on quantile-quantile plots, empirical distribution diagnostics, nonlinear terms, and sparse covariates applies to the weighted distributions as well: a small mean difference does not rule out differences elsewhere in a distribution or in important covariate functions.
Also inspect weight ranges and concentration separately by treatment group. Large combined weights can make summaries depend heavily on a few observations; propensity scores near zero or one signal potential positivity problems. Check that the intended survey and treatment components were multiplied, rather than accidentally substituted for one another. Where the survey design supplies strata, clusters, or replicate weights, retain that information for design-based uncertainty calculations. A simple weighted balance calculation is descriptive and does not automatically provide a valid uncertainty interval for a complex survey.
The main alternatives answer different questions. Survey-weighted comparisons without treatment adjustment describe the treated and control groups in the survey target population; they are appropriate descriptive comparisons but do not generally target a shared covariate distribution. Treatment adjustment without survey weights generally targets the analyzed sample’s population, not necessarily the survey target. Matching can create a different comparison population by restricting or pairing units, while outcome-model adjustment targets its stated population through a model. Whichever approach is chosen, the balance diagnostic must use the weights and population that correspond to that approach.
- Common mistake: “The survey weights are large for underrepresented people, so the weighted treated and control groups are balanced.” Survey weighting addresses representation in the survey target; it does not cancel differences in treatment probabilities. The correction is to calculate group-specific weighted summaries and, if a common treatment target is intended, evaluate the appropriate combined weights.
- Another pitfall: Treating a small SMD as proof of complete balance. An SMD concerns a particular location summary and scaling choice. Inspect distributions and substantively important transformations as well.
- Limitation: Good measured-covariate balance does not show that unmeasured confounding is absent, that the propensity model is correct in every relevant respect, or that the outcome effect is identified.
Check Your Understanding
Answer each question, then compare with the solutions.
- What population does a comparison using survey weights alone describe, and why does it not necessarily yield treatment balance?
- A binary covariate has weighted proportions 0.40 in the treated group and 0.25 in controls. Calculate the difference in percentage points.
- Why must a balance report distinguish survey weights from treatment-adjustment weights?
- What does the conditional identity for an indicator \(h(X)\) establish about inverse-probability weighting?
Solutions. (1) It compares the survey-represented treated and control groups; treatment can still be associated with covariates, so their represented distributions can differ. (2) The difference is \(0.40-0.25=0.15\), or 15 percentage points. (3) Survey weights address representation in the survey target, whereas treatment factors reweight groups toward an estimand-specific target; combining them changes the comparison being made. (4) Applying the identity to indicators of covariate sets establishes equality of the weighted covariate distributions in expectation under the stated propensity and positivity conditions, not merely equality of their means.