Diagnosing Weighted and Unweighted Discrepancies

Why Compare Weighted and Unweighted Summaries?

A weighted summary and an unweighted summary describe the same survey records using different contributions from each record. The unweighted mean gives every record equal contribution. A weighted mean gives more contribution to records with larger weights. A difference between the two is therefore a useful diagnostic: it shows that the outcome values and the weighting of records combine differently from equal weighting.

The comparison does not, by itself, show that either estimate is wrong or that weighting has removed bias. To interpret it, ask what records have larger weights, whether their outcomes differ, and how much of the weighted total is represented by each relevant group. Those questions help distinguish a change in group composition from a change in the weighting of outcomes within groups.

For this comparison to be informative, calculate both summaries for the same records and the same outcome definition. Keep the units and the records’ inclusion in the calculation consistent. A difference can arise because the weighted and unweighted calculations represent different mixtures of groups, because weights vary among records within groups, or both. Scaling all weights by the same positive constant does not change the weighted mean.

A Decomposition of the Difference

Suppose the records are divided into mutually exclusive groups \(g=1,\ldots,G\), such as age categories or regions. Write \(n_g\) for the number of sample records in group \(g\), \(W_g=\sum_{i\in g}w_i\) for its weight total, and \(W=\sum_g W_g\). Let \(\bar{y}_g\) be the unweighted mean in group \(g\), and let \(\bar{y}_{wg}=\sum_{i\in g}w_i y_i/W_g\) be its weighted mean. Define the unweighted sample share \(p_g=n_g/n\) and the weighted share \(P_g=W_g/W\). Then

The first expression combines group means using the sample’s group shares. The second combines weighted group means using each group’s share of the total weight. Add and subtract \(\sum_g P_g\bar{y}_g\) to obtain an exact decomposition:

The first term is a composition component: it reflects a difference between the weighted and unweighted group shares, applied to the unweighted group means. The second is a within-group component: it reflects differences between the weighted and unweighted means within groups, averaged using the weighted group shares. This is an accounting identity, not a causal explanation. The choice of groups matters: a useful grouping is one that helps explain how the weights and outcome values vary.

Worked Comparisons

Why the Decomposition Works

The identity separates two sources of change without requiring an approximation. Since the records form a partition, the unweighted mean is the sum of group means multiplied by sample shares: each group contributes the fraction \(n_g/n\) of the records. Likewise, the weighted mean is the sum of weighted group means multiplied by weight shares: each group contributes the fraction \(W_g/W\) of the total weight. Subtract the first sum from the second and insert \(\sum_g P_g\bar{y}_g\):

In the first term, the group means are held at their unweighted values while the shares change from \(p_g\) to \(P_g\). In the second term, the weighted shares are held fixed while the within-group means change from \(\bar{y}_g\) to \(\bar{y}_{wg}\). This particular decomposition uses unweighted group means as the reference for the composition component. Another decomposition could use a different reference, so report the definition used when presenting components.

When the Overall Difference Conceals Subgroup Patterns

A small overall difference does not show that weights have little effect in every subgroup: positive and negative changes can offset. Conversely, a large overall difference need not imply that every subgroup changes substantially; a composition shift between groups with different outcome levels can produce it. Subgroup comparisons and the decomposition help make these possibilities visible.

Common Mistakes and Interpretation

• Calling a weighted–unweighted gap a bias estimate. The gap compares two summaries of the observed records. It does not measure the error of either estimate relative to an unknown population value.
• Explaining a gap using weight variation alone. Weight variation is not enough: if high- and low-weight records have similar outcomes, reweighting may change the summary little. Examine how outcomes vary across the weight pattern.
• Assuming the weighted share is a literal head count. A group’s weighted share is its share of the sum of weights, not its fraction of sampled respondents. Describe it as weighted representation, not as the number of people observed.
• Interpreting a small aggregate gap as no subgroup effect. Differences in opposite directions can cancel. Compare group-specific summaries when subgroup patterns matter.
• Treating the comparison as an uncertainty assessment. The difference alone does not provide a standard error, confidence interval, or significance test. Those require appropriate survey-design variance methods.