Nonresponse and Survey Participation

Introduction

A probability sample can be drawn carefully and still produce an observed survey sample that differs from the intended one. Some selected units may never be reached, some may decline, and some who participate may leave particular questions unanswered. These are distinct points at which participation can fail, and they have different implications for what the collected data represent.

The key question is not simply how many people replied. It is which selected units participated, which questions they answered, and whether those patterns are related to the quantities being studied. A response rate describes participation under a stated definition; by itself, it does not establish that respondents represent nonrespondents or quantify any resulting distortion.

Definition: Unit nonresponse occurs when a sampled unit does not participate in the survey at the level required by the study’s stated response rule. Item nonresponse occurs when a participating unit does not provide a usable answer to a particular survey item. A respondent is a sampled unit counted as having responded under the prespecified unit-response rule.

Where Participation Can Fail

Participation is a process, not a single yes-or-no event. A sampled unit may be ineligible under the study’s target-population definition; an eligible unit may be difficult to contact; a contacted unit may decline; and a participant may begin but not complete the questionnaire. A participant who completes most of it may still omit a particular item. These stages should be recorded separately where possible, because one overall count can conceal where and for whom participation fell short.

The study should specify in advance what counts as a unit response. For example, it might count any interview with a minimum set of substantive answers as a response, while separately reporting fully completed questionnaires. This prevents the definition from changing after the response pattern is known. It also makes the denominator interpretable: rates should be calculated over sampled units known to be eligible under the chosen rule, with exclusions and unresolved eligibility cases described rather than silently discarded.

Draw the sample.
Record the selected units and, where relevant, their inclusion probabilities from the sampling design.

Establish eligibility and contact.
Track whether each selected unit belongs to the target population and whether contact was attempted and achieved.

Record participation status.
Distinguish refusal, noncontact, partial participation, and participation meeting the stated unit-response rule.

Track answers by item.
Among participants asked each question, record which provided a usable answer and which did not.

Describe the observed data.
Compare available characteristics across participation groups, while recognizing that measured comparisons cannot rule out differences in unobserved characteristics.

Unit response and item response have different denominators. A unit response rate concerns selected eligible units and the survey’s unit-level response rule. An item response rate concerns participants who were asked a particular item and whether they supplied a usable answer. If a questionnaire uses different routing for different participants, the item denominator should include only those eligible and asked that item.

Definition: For a survey with $N_E$ sampled units classified as eligible and $N_R$ units meeting the prespecified response rule, the unit response rate is $N_R/N_E$. For an item asked of $N_A$ participating units, with $N_Q$ usable answers, the item response rate is $N_Q/N_A$. Each reported rate should state its numerator, denominator, and relevant eligibility or completion rules.

How Response Can Change the Observed Sample

Earlier material on inclusion probabilities distinguishes the probability that a unit is selected from the population from the characteristics of the resulting sample. Survey participation adds another selection stage: a selected unit may or may not respond. Let $R_i=1$ if unit $i$ responds under the unit-response rule and $R_i=0$ otherwise. Let $Y_i$ be the survey outcome of interest. The observed respondent mean is calculated from units with $R_i=1$, not from all sampled units.

If response is unrelated to $Y_i$, respondent and nonrespondent outcome distributions may be similar on average, though a small realized respondent sample can still differ by chance. If response is related to $Y_i$, the outcome distribution among respondents can differ systematically from that among all eligible sampled units. Response may also vary with observed characteristics that are related to $Y_i$, changing the mix of respondents even when the relationship between response and outcome within groups is not known.

Key relationship: In a population where $Y$ is the outcome and $R$ indicates response, the respondent mean is $E[Y\mid R=1]$, provided the conditional expectation exists and $P(R=1)>0$. Its difference from the population mean is $$ E[Y\mid R=1]-E[Y]=\frac{\operatorname{Cov}(Y,R)}{P(R=1)}. $$ Thus response associated with the outcome can shift the respondent mean. A response rate alone gives $P(R=1)$, not the covariance needed to determine the direction or size of that shift.

The relationship follows directly from the definition of conditional expectation. Since $R$ is an indicator, $E[R]=P(R=1)$ and $E[YR]=P(R=1)E[Y\mid R=1]$. Also, $\operatorname{Cov}(Y,R)=E[YR]-E[Y]E[R]$. Substituting these expressions and dividing by $P(R=1)>0$ gives the displayed difference. This identity describes a potential source of distortion; it does not, on its own, identify the response relationship from the observed data.

Worked Example: Calculating a Unit Response Rate

Suppose a hypothetical survey draws 250 units classified as eligible. Under the survey’s prespecified rule, 200 units provide enough substantive information to count as unit respondents. Of those 200, 175 complete the full questionnaire. Calculate and interpret the response and completion rates.

The unit response rate uses all 250 eligible sampled units as its denominator and the 200 units meeting the response rule as its numerator:

\text{Unit response rate}=\frac{200}{250}=0.80=80\%.

The full-completion rate, using the same eligible-sample denominator, is:

\text{Full-completion rate}=\frac{175}{250}=0.70=70\%.

Thus, 80% of the eligible sampled units meet the stated unit-response rule, while 70% complete the full questionnaire. These are descriptive participation rates. Neither calculation establishes whether the 50 units not counted as respondents differ from respondents on an outcome of interest.

Worked Example: Item Nonresponse and an Observed Mean

In a hypothetical survey, 10 participants are asked monthly household spending in dollars. Eight provide usable answers: $2000, 2200, 2400, 2600, 2800, 3000, 3200,$ and $3800$. Two leave the item blank. Calculate the item response rate and the mean among those who answered.

All 10 participants were asked the item, so the item response rate is $8/10$. The sum of the eight reported values is $22{,}000$ dollars, giving:

\text{Item response rate}=\frac{8}{10}=80\%,\qquad \bar{x}_{\text{answered}}=\frac{22{,}000\text{ dollars}}{8}=2{,}750\text{ dollars}.

Suppose, for illustration only, that a separate validation source establishes that the two omitted values were $5{,}000$ and $6{,}000$ dollars. The mean for all 10 participants would then be:

\bar{x}_{\text{all 10}}= \frac{22{,}000+5{,}000+6{,}000}{10} =\frac{33{,}000}{10} =3{,}300\text{ dollars}.

In this hypothetical example, the mean among item answerers is 550 dollars below the mean across all 10 participants. That difference arises because the two omitted values are high relative to the reported values. Ordinarily, the omitted answers are not known, so this comparison cannot simply be calculated from a survey’s returned questionnaires. It illustrates why the observed answerer mean need not equal the participant mean.

Response Patterns as Evidence to Examine

A useful first diagnostic is to compare response rates across groups defined by characteristics known for the full sample, such as region, sampling stratum, or contact mode. If rates differ, the respondent composition may differ from the sampled composition. Such a comparison can reveal a participation pattern, but it does not establish the outcome values of nonrespondents. Even similar rates across groups do not guarantee similar outcomes among respondents and nonrespondents within those groups.

Worked Example: Different Participation Across Regions

A hypothetical survey samples 120 urban households and 80 rural households. Ninety urban households and 40 rural households respond. Suppose the respondent outcome is a binary indicator of having used a specified service in the past month. For illustration, assume 40% of urban respondents and 20% of rural respondents report use. Compare the sample and respondent composition, then calculate the respondent mean.

The sampled composition is 60% urban and 40% rural. Among the 130 respondents, the urban share is $90/130$, or 69.23%, and the rural share is $40/130$, or 30.77%. The regional response rates are $90/120=75\%$ for urban households and $40/80=50\%$ for rural households. Overall, $130/200=65\%$ of sampled households respond.

The respondent mean of the binary outcome is the respondent share reporting use. Using the stated group rates and respondent counts:

\bar{Y}_{R} =\frac{90(0.40)+40(0.20)}{130} =\frac{44}{130} \approx 0.3385.

The respondent mean is approximately 33.85%. To isolate the effect of the changed regional mix in this illustration, suppose the two stated regional outcome rates also apply to nonrespondents within their respective regions. The mean for the full 200-household sample would then be:

\bar{Y}_{S} =\frac{120(0.40)+80(0.20)}{200} =\frac{64}{200} =0.32.

Under that explicit assumption, the respondent mean is 1.85 percentage points higher than the full-sample mean because urban households, who have the higher assumed outcome rate, make up a larger share of respondents. Without the assumption about outcomes among nonrespondents, the regional response rates alone cannot determine the full-sample outcome mean or the amount of distortion.

Interpreting Participation Diagnostics

Participation records are most informative when they preserve distinctions: selected versus eligible units, reached versus unreached units, refusals versus other nonresponses, and unit response versus item answers. A useful report states how each status is defined and gives the denominator used for each rate. If eligibility is unresolved for some selected units, report how those cases were treated rather than implying that the denominator is unambiguous.

Comparisons of respondents with the original sample can be made for characteristics available for both groups. Differences are evidence that the observed respondent composition changed on those characteristics. They are not, by themselves, a complete assessment of outcome bias. Conversely, a high response rate is not proof of representativeness: the smaller group that does not respond could differ substantially on an outcome. A low response rate signals greater scope for concern but does not establish that a particular estimate is biased.

The distinction between unit and item nonresponse also matters for interpretation. A survey may have high unit response but substantial nonresponse to a sensitive income question. In that case, the participants may represent the sampled units reasonably on some observed features while the subset answering income is different. The appropriate descriptive denominator and comparison group depend on whether the target is all eligible sampled units, survey participants, or participants asked a particular item.

Common Mistake

Wrong reasoning: “The survey response rate is 80%, so the respondent estimates are 80% representative.” A percentage response rate measures the share meeting a response rule; it is not a percentage measure of representativeness, nor does it show which outcomes the nonrespondents would have reported.

Correction: State the response-rate definition, examine how participation varies across available sample characteristics and survey items, and explain what remains unknown about nonrespondents. Do not infer the size or direction of outcome distortion from the response rate alone. Any particular adjustment requires additional assumptions or information and is beyond the scope of this introduction to participation.

Key takeaway: Unit nonresponse concerns whether sampled units participate; item nonresponse concerns whether participants answer particular questions. Participation rates describe these processes, while representativeness depends on how response relates to the quantities and characteristics being studied.

Check Your Understanding

Use the definitions and distinctions in this tutorial to answer the following questions.

A sampled participant completes an interview but leaves the employment-status question blank. Is this unit nonresponse, item nonresponse, or both? Explain using the stated response-rule idea.
A survey has 300 eligible sampled units, of which 210 meet its unit-response rule. What is the unit response rate, and what does it describe?
A question is asked of 160 participants, and 136 provide usable answers. Calculate the item response rate and identify its denominator.
Why can response rates that differ by region change the respondent composition without revealing the full-sample outcome mean?
What additional relationship, beyond the overall response rate, is needed to determine whether response shifts a respondent mean relative to a population mean?