Weighted Versus Unweighted Sample Summaries

Why Two Summaries Can Differ

A survey data file contains records from people or units that were selected and provided usable responses. An unweighted summary gives each responding record the same influence. It therefore describes the responding records as they appear in the file. But if some population units were more likely to be selected—or more likely to respond after selection—those records may not represent population units equally. A weighted summary gives records different influence to reflect their representation of the target population.

The distinction is about the question being answered. “What proportion of these respondents reported the outcome?” calls for a direct summary of the respondents. “What proportion of the target population has the outcome?” is a population-estimation question, for which survey weights may be appropriate. Weighting can help align the respondent data with the population, but it does not automatically eliminate nonresponse, coverage, or measurement error.

Definition: Let the target population contain \(N\) units, and let \(y_i\) be the value of a variable for population unit \(i\). An unweighted sample summary gives each responding record equal influence. A weighted summary gives record \(i\) a nonnegative weight \(w_i\), intended to represent how many population units it stands for, or how much population influence it should receive. The responding-record distribution is the distribution among the records with usable responses; the target population distribution is the distribution among all units in the population of interest.

Weights commonly begin with design weights. If unit \(i\) has known probability \(\pi_i\) of being selected, its design weight is \(1/\pi_i\). For example, a selection probability of \(0.25\) gives a design weight of \(4\): a selected unit represents four population units under that design. If weights are used for respondents, they may also need to account for the process that produced response. The design weight alone does not generally correct for differential nonresponse.

Unweighted and Weighted Summaries

For a binary variable, such as whether a respondent experienced an outcome, the unweighted proportion is the number of respondents with the outcome divided by the number of respondents with a recorded value. A weighted proportion instead gives each record influence proportional to its weight. For a quantitative variable, the unweighted mean is the ordinary average of the respondent values; a weighted mean averages those values using their weights.

\hat{p}_{\text{unweighted}}=\frac{\sum_{i\in R} y_i}{n_R}, \qquad \hat{p}_{\text{weighted}}=\frac{\sum_{i\in R} w_i y_i}{\sum_{i\in R} w_i}, \qquad \bar{x}_{\text{weighted}}=\frac{\sum_{i\in R} w_i x_i}{\sum_{i\in R} w_i}

Here \(R\) denotes the set of responding records, \(n_R\) is its size, and \(y_i\) is coded as 1 for the outcome and 0 otherwise. The weighted proportion is therefore a weighted average of the zero-one values. The unweighted proportion uses the same values but gives each response equal influence. These formulas summarize the records at hand; interpreting a weighted result as a population estimate requires additional design and response conditions.

Worked Example: Unequal Selection Rates and an Outcome Proportion

Suppose a hypothetical population has 80 units in group A and 20 in group B. A survey selects 40 units from A and 5 from B, and all selected units respond. Among the A respondents, 10 report a specified outcome; among the B respondents, 4 report it. Compare the unweighted respondent proportion with an estimate that accounts for the unequal selection rates.

The inclusion probabilities are \(40/80=0.5\) for A and \(5/20=0.25\) for B. Their inverse-probability weights are \(1/0.5=2\) and \(1/0.25=4\), respectively. The unweighted proportion among the 45 respondents is

\hat{p}_{\text{unweighted}}=\frac{10+4}{40+5} =\frac{14}{45}=0.3111.

For the weighted summary, the outcome records contribute \(10(2)+4(4)=36\) weighted units. The full weighted respondent count is \(40(2)+5(4)=100\). Thus,

\hat{p}_{\text{weighted}}=\frac{36}{100}=0.3600.

The unweighted result says that 31.11% of these responding records report the outcome. Under the stated design and full-response conditions, the weighted estimate is 36.00% for the target population. The difference arises because group A, which is 80% of the population, forms \(40/45=88.89\%\) of the responding records. The weights give the less heavily sampled group B greater influence.

Why Weighting Can Target the Population

The inverse-probability principle has a simple justification. Let \(I_i=1\) if population unit \(i\) is selected and \(I_i=0\) otherwise. Under a probability-sampling design, \(P(I_i=1)=\pi_i\). For a fixed population value \(y_i\), the expected contribution of unit \(i\) to the inverse-probability weighted total is

E\left(\frac{I_i y_i}{\pi_i}\right) =\frac{y_i}{\pi_i}E(I_i) =\frac{y_i}{\pi_i}\pi_i =y_i.

Adding these contributions across the population gives an unbiased estimator of the population total when all selected units provide the required values. Dividing by a known population size \(N\) then gives an unbiased estimator of the population mean. In practice, a normalized weighted mean often divides the weighted sum of observed values by the sum of observed weights. That ratio is generally not exactly unbiased in a finite sample, although it is a standard population estimator under appropriate conditions.

For respondents rather than all selected units, a further condition is needed: the weights must appropriately account for the chance of response, or response must be ignorable within the groups or models used to construct the weights. In practical terms, respondents with similar adjustment characteristics must be representative of the nonrespondents they are intended to represent. If response still depends on the outcome after those adjustments, weighting may leave bias.

Worked Example: A Weighted Mean of a Quantitative Variable

Consider five hypothetical respondents. Three belong to group A, each with weight 1, and report values of 2, 4, and 6 hours. Two belong to group B, each with weight 3, and report values of 10 and 12 hours. Calculate the unweighted and weighted means.

The unweighted mean gives all five records equal influence:

\bar{x}=\frac{2+4+6+10+12}{5} =\frac{34}{5}=6.8\text{ hours}.

The weighted sum of values is \(1(2+4+6)+3(10+12)=12+66=78\). The sum of the weights is \(3(1)+2(3)=9\). Therefore,

\bar{x}_{\text{weighted}} =\frac{78}{9} =8.667\text{ hours}.

The unweighted mean is the average among these five respondents. The weighted mean gives group B more influence because each of its records represents three units under the stated weights, compared with one unit for each A record. If those weights correctly represent the target population, 8.667 hours is the population-oriented estimate; the calculation alone does not establish that the weights are appropriate.

Selection Rates Do Not Determine Respondent Representation

A higher sampling rate makes a group more common in the selected sample relative to its population share, all else equal. It does not by itself guarantee that the group is more common among respondents. Different response rates can offset or reverse the sampling-rate difference. This matters when deciding whether design weights alone are enough or whether respondent weights need an additional nonresponse adjustment.

Worked Example: Response Rates Can Reverse Sample Representation

Suppose a hypothetical population contains 60 units in group A and 40 in group B. The design selects 30 A units and 10 B units, so the sampling rates are 50% and 25%. Only 20% of selected A units respond, while 90% of selected B units respond. Assume, for illustration, that the response rates and within-group response patterns are adequately represented by these group-specific rates. How are the groups represented among respondents, and what weights would account for both stages?

The expected respondent counts are \(30(0.20)=6\) in A and \(10(0.90)=9\) in B. Thus, A makes up \(6/15=40\%\) of respondents, even though it is 60% of the population and was sampled at twice B’s rate. Unequal response has more than offset the sampling-rate difference.

The probabilities of both selection and response are \(0.50(0.20)=0.10\) for A and \(0.25(0.90)=0.225\) for B. The corresponding inverse probabilities are \(10\) and \(1/0.225=4.444\). The expected weighted respondent counts are \(6(10)=60\) and \(9(4.444)\approx40\), restoring the population group counts in expectation under the stated assumptions. A weight based only on selection would be \(2\) for A and \(4\) for B; it would not address the groups’ different response rates.

This calculation does not show that a group-level response adjustment always removes nonresponse bias. It illustrates what the adjustment is intended to do when response probabilities are known or suitably estimated and response is adequately described within the adjustment groups.

Conditions, Interpretation, and Common Pitfalls

A weighted summary is an estimate for a specified target population, not merely a corrected version of every unweighted statistic. Its interpretation depends on the target population, the variables being summarized, the construction of the weights, and whether the response process is adequately handled. A useful check is to state explicitly: “This unweighted proportion describes the responding records,” or “This weighted proportion estimates the proportion in the target population under the stated weighting assumptions.”

Weighting is most directly helpful when unequal selection or response probabilities are related to characteristics that also vary with the outcome. If all records have the same weight, weighted and unweighted averages are identical. If groups differ in weights but have the same outcome distribution, weighting may change little. Conversely, a large change after weighting signals that the weighted records contribute differently; it does not, by itself, prove which estimate is closer to the truth.

Mistake: “The group was sampled at a higher rate, so it must be overrepresented among respondents.” That conclusion ignores response. Comparable response rates would support the claim, but a sufficiently lower response rate can offset or reverse the difference. Check both stages.
Mistake: “A weighted result is automatically unbiased.” Weights address only the selection and response mechanisms they correctly represent. They do not automatically correct unmeasured nonresponse differences, coverage gaps, or inaccurate answers.
Mistake: “The unweighted result is wrong.” It correctly describes the responding records. It may not estimate the target population well when records have unequal representation.
Mistake: “The difference between weighted and unweighted results proves bias.” A difference shows that the summaries give records different influence. It does not establish which result is closer to the population value without assumptions or external evidence.

Key takeaway: Unweighted summaries describe responding records with equal influence. Weighted summaries give records unequal influence to estimate a target population when the weights appropriately reflect selection and, when needed, response. Unequal selection rates alone are not enough to determine how respondents represent the population.

Check Your Understanding

Answer each question by distinguishing what the records show from what can be inferred about the target population.

A survey has 30 respondents from a group that is 20% of the population. What can you conclude from the respondent count alone about that group’s representation in the population?
In a two-group survey, one group is sampled at a higher rate but has a lower response rate. What additional information is needed to determine its share among respondents?
For a binary outcome, explain in words how the weighted proportion differs from the unweighted proportion.
Why can inverse selection-probability weights fail to correct bias if response probabilities differ across groups?
If all respondent weights are equal, how do the weighted and unweighted means compare? Explain why.