Why an Incomplete Frame Matters
A survey can select people carefully and still miss part of the population it is meant to describe. Imagine drawing a simple random sample from a well-maintained list of people with email addresses. The sample may represent that list very well, but it cannot directly represent people who are not on the list. The central question is not only how was the sample selected? It is also who could have been selected?
A sampling frame is the list, register, or other operational source from which a sample is drawn. The target population is the full group the study intends to describe. When some members of the target population are absent from the frame, or have substantially less chance of appearing in it, the survey has a coverage problem. This tutorial focuses on undercoverage: missing or poorly represented parts of the target population.
Undercoverage is a property of the link between the target population and the frame, not a description of whether a particular sample happened to look unusual. A random sample from an incomplete frame can be a sound probability sample of the people on that frame. The limitation is that the frame does not fully represent the population of interest.
How Undercoverage Can Affect an Estimate
Missing people do not automatically make every estimate wrong. The key issue is whether the omitted members differ from covered members on the survey outcome. If the groups have similar outcome values, missing coverage may have little effect on that particular estimate. If they differ, a frame-based estimate can systematically overstate or understate the target-population value.
The following derivation makes the relationship precise for a numerical outcome. Suppose the target population \(U\) is divided into covered members \(F\) and omitted members \(M\). Let \(c\) be the fraction of the target population represented in the frame, and let \(\mu_F\) and \(\mu_M\) be the mean outcome in the covered and omitted groups. The target-population mean is the weighted average of the two group means:
If a simple random sample is drawn from the frame, its sample mean \(\bar{x}\) is centered on \(\mu_F\), the frame mean. The difference between that frame mean and the target-population mean is therefore
This result has two parts. The fraction omitted, \(1-c\), determines how much of the population is missing. The difference \(\mu_F-\mu_M\) determines how different the covered and omitted groups are on the outcome. If either factor is zero, this particular mean has no coverage bias: either everyone is covered, or the two group means are equal. Otherwise, the frame mean differs from the target mean. For a categorical outcome, the same weighted-average reasoning applies to the population proportion \(p\).
Worked Examples of Undercoverage
Worked Example: An Online Survey of County Adults
A county health department wants to estimate the proportion of all county adults who say they can access a clinic when needed. It draws a simple random sample from an online survey platform’s adult-user list. For this hypothetical example, assume that 72% of county adults are represented on the list, 62% of those covered adults can access a clinic, and 38% of adults omitted from the list can do so. Assume, to isolate the coverage issue, that everyone selected from the list provides an answer.
Identify the coverage gap. Adults not represented on the platform-user list have no chance of selection from this frame. The list covers \(c=0.72\) of the target population, so the omitted fraction is \(1-0.72=0.28\).
Calculate the target proportion. Weight each group’s proportion by its share of all county adults:
The hypothetical population proportion is 55.28%, while the covered group’s proportion is 62%. A random sample from the platform list could estimate the covered-group proportion well, yet that estimate would overstate the target-population proportion by \(0.62-0.5528=0.0672\), or 6.72 percentage points. The cause is not random selection within the list; it is the lower access rate among the omitted group.
Worked Example: A Household Survey Using an Address List
Suppose a hypothetical survey aims to estimate the proportion of all households in a region that have a working smoke alarm. An address list covers 85% of households. Among covered households, 20% lack a working alarm; among omitted households, 50% lack one. Assume a probability sample is selected from the address list and that every selected household answers, so this example isolates coverage rather than other sources of error.
State the relevant outcome. Let the outcome be whether a household lacks a working alarm. The covered-group proportion is \(p_F=0.20\), and the omitted-group proportion is \(p_M=0.50\). The fraction omitted is \(1-0.85=0.15\).
Compute the target proportion:
The target-population proportion is 24.5%. Reporting the covered-frame value of 20% would underestimate the proportion of all households without a working alarm by \(0.245-0.20=0.045\), or 4.5 percentage points. The omitted households matter here because they are both 15% of the target population and more likely to lack an alarm.
Worked Example: A Campus Survey of Undergraduates
A university wants to estimate the proportion of all its undergraduates who commute by car. A hypothetical registrar’s list covers students at the main campus but omits some students studying at remote sites. The list covers 75% of undergraduates. Of covered students, 30% commute by car; of omitted students, 55% commute by car. Assume a random sample from the list is fully observed.
Find the population proportion. The omitted fraction is \(1-0.75=0.25\). Weight the two group proportions by their population shares:
The target-population proportion is 36.25%, compared with 30% among students represented on the list. A frame-based estimate would be lower by \(0.3625-0.30=0.0625\), or 6.25 percentage points. The frame represents the main-campus group, not the full undergraduate population the university intends to describe.
Diagnosing a Coverage Problem
A useful diagnosis begins by stating the target population in concrete terms: who belongs, and when? Next, identify exactly what records or access points make up the sampling frame. Then compare the two. Ask which eligible people do not appear, whether any groups appear only intermittently, and whether the missingness could be connected to the survey outcome.
Name the people or units the conclusion is meant to describe, using clear eligibility criteria and a relevant time period.
State which list, register, directory, or access source will be used to select the sample. Do not assume its label guarantees complete coverage.
Identify population members missing from the frame or represented less reliably, such as adults without platform accounts, households absent from an address list, or students at remote sites.
Ask whether the omitted groups might have different values of the variable being measured. A gap may have little effect on one outcome and substantial effect on another.
For example, an online frame may cover nearly all residents in a particular age group but poorly cover those without reliable internet access. That omission is especially concerning for an estimate of internet access itself, since the chance of appearing in the frame is directly related to the outcome. In a different survey, the same gap might have a smaller effect if covered and omitted residents have similar values on the measured variable.
A related but distinct issue occurs when eligible people are listed in the frame but do not provide answers. That is a response problem rather than undercoverage: those people could have been selected from the frame. Keeping the distinction clear helps locate the source of a survey’s limitation. Likewise, a complete frame does not by itself guarantee that a selection method avoids self-selection or other problems.
Reducing the Coverage Gap
The most direct remedy is to improve the frame before selecting the sample. A survey team might update an address list, add records for eligible people at remote sites, or use more than one source to reach groups absent from a single list. Any expanded approach needs clear rules for deciding who is eligible and for avoiding duplicate representation. Otherwise, a repair to one gap can introduce a new one.
If some groups remain difficult to cover, the limitation should be made explicit when describing the population to which results apply. Analysts may also compare available information about covered and omitted groups, but such comparisons cannot establish that the groups are alike on an unmeasured outcome. Adjustments based on known group sizes or outcomes require suitable information and assumptions; they are not a substitute for checking coverage at the design stage.
The practical goal is not to claim that every frame is perfect. It is to find out who can and cannot enter the sample, decide whether that gap threatens the intended conclusion, and report the limits honestly. A large sample from a narrow frame can produce a precise estimate of the wrong population.
Common Mistake
Wrong reasoning: “We used a simple random sample, so every member of the target population had a fair chance to be selected and the estimate is unbiased.”
Correction: A simple random sample gives members of the sampling frame equal selection chances. It gives the target population equal chances only if the frame adequately represents that population. If a group is absent, its chance of selection from that frame is zero. Randomness within the frame does not erase undercoverage.
Check Your Understanding
Use the distinction between the target population, the frame, and the outcome to answer each question.
- A survey aims to describe all adults in a town but samples only people listed in an online panel. What is the potential coverage problem?
- In the mean derivation, what does the difference \(\mu_F-\mu_M\) tell you? What happens to the coverage bias if that difference is zero?
- A campus frame omits 10% of students. Is a 10% omission alone enough to determine whether an estimate is biased? Explain.
- Households are listed in a frame, but some selected households do not answer. Is this undercoverage? Why or why not?
- A random sample is drawn from an incomplete household address list. Which group can the sample directly represent: all target households, or those represented on the list?