Why Random Selection Matters

Why the Selection Method Matters

In Population, Sampling Frame, and Sample, we distinguished the group a study wants to learn about, the list used to find people, and the people actually selected. Now consider how those people are selected. A sample can be large and still give a misleading picture if the selection method favors certain members of the population.

Suppose a community center wants to learn how often residents use its walking paths. Asking people who happen to be on a path would be convenient, but it would tend to reach path users and miss residents who rarely or never go there. The issue is not just sample size. The selection method has made some perspectives much easier to include than others.

Selection bias occurs when the way individuals enter a sample systematically favors some members or types of members over others. A chance-based selection process helps prevent a researcher from choosing only people who are easy to reach, seem typical, or are expected to give a particular answer. It does not make every possible sample equally informative, and it does not guarantee that the selected sample will match the population perfectly.

This distinction connects to Generalizing Results to a Population and Scope of Inference: Four Combinations. Random selection supports generalizing from a sample to the population from which the sample was selected. It is separate from random assignment, which is used in experiments and supports cause-and-effect conclusions. A random sample does not, by itself, show that one variable causes another.

Chance Reduces Favoritism, Not All Error

A chance-based selection method gives the selection process a rule that does not depend on a person choosing the individuals they prefer. For example, a school could use a random-number generator to select student IDs from an accurate roster, then invite the selected students to report their weekly screen time. The method is different from asking students who happen to be in the library or inviting students to answer an open online poll.

Random selection matters even when nobody intends to be unfair. People who are easiest to contact may have different experiences from people who are difficult to reach. A researcher might unknowingly choose people who are nearby, available, or familiar. Chance offers a systematic alternative to those judgment-based choices.

However, a random selection process does not promise a sample that mirrors every population feature. Different chance-based samples from the same population can include different people and produce different statistics. This ordinary sample-to-sample variation is sampling variability, as introduced in Sources of Variability in Collected Data. Random selection makes that variability part of a chance process rather than a result of deliberate or predictable favoritism.

A useful way to think about the benefit is that chance makes the selection process defensible. If the frame is suitable and the selection is genuinely random, the sample is not simply a group chosen because it was convenient or appealing. That supports generalization, but the conclusion must still be limited to the population the frame can represent.

Worked Example: A Random Sample of Households

Random Selection Cannot Repair a Poor Frame

As explained in Population, Sampling Frame, and Sample, the sampling frame is the list or source used to select individuals. A random selection from a frame can only choose from people or units that the frame includes. If some members of the population are missing, they have no chance of being selected from that frame.

This is called undercoverage: part of the population of interest is not adequately represented in the sampling frame. Randomly selecting from an undercovered frame does not make the missing people appear in the sample. In the same way, a chance-based sample from a list that includes ineligible entries can select those entries. First ask whether the frame fits the population; then consider how selection was carried out.

Worked Example: A Random Draw from an Incomplete List

Nonresponse Is Another Limit

Selection and response are separate steps. A person can be randomly selected but not answer the survey. As in Population, Sampling Frame, and Sample, the selected sample and the respondents are not always the same group. If response rates or willingness to answer are related to the variable being measured, the responses may systematically differ from what would have been learned from all selected people.

For instance, suppose a random sample of residents is selected for a survey about park use, but residents who rarely visit parks are less likely to respond. The initial selection was chance-based, yet the completed responses could overrepresent frequent visitors. Random selection supports generalization only when the selection process and the data collected allow a reasonable connection to the population. A researcher should report nonresponse rather than quietly treating respondents as if they were the entire selected sample.

Worked Example: Randomly Selected, but Not All Responding

What Random Selection Does—and Does Not—Support

A clear conclusion matches its scope to the way the sample was obtained. If a random sample is selected from a frame that represents a defined population and the response process does not seriously undermine that connection, sample findings can be generalized to that population. If the frame covers only a subgroup, the conclusion should be limited to that subgroup.

Random selection does not mean that every selected person will be typical, that the sample statistic equals the parameter, or that every possible source of bias has disappeared. It also does not turn an observational study into an experiment. As covered in Observational Studies and Their Limits and Association Versus Causation in Study Conclusions, generalizing a result and claiming a cause are different kinds of conclusions.

In the next tutorial, Simple Random Sample Defined, we will give a more specific name and definition to one chance-based sampling method. For now, the central point is that random selection is a way to let chance—not convenience or preference—guide who enters the sample.

Common Mistakes and Full-Credit Communication

• Assuming random selection guarantees a representative sample. Say that chance-based selection reduces selection bias and supports generalization, not that it guarantees the sample will match the population exactly. Acknowledge sampling variability.
• Confusing random selection with random assignment. Selection concerns who enters a study and can support generalizing to a population. Assignment concerns which treatment experimental units receive and can support a cause-and-effect conclusion.
• Using “random” to describe an arbitrary choice. A researcher choosing whoever is nearby has not used random selection. Full-credit wording identifies a chance-based method, such as a random-number draw from a stated frame.
• Ignoring the frame. A random sample from an incomplete list cannot represent people absent from that list merely because the draw itself was random. Name the frame and the population, then explain any mismatch.
• Forgetting nonrespondents. Report how many were selected and how many responded. If the response process may be related to the measured variable, explain why that could limit generalization.
• Using the wrong denominator. A sample percentage uses the sample size; a population percentage uses the population size. State both groups and counts when comparing the two.
• Claiming a cause from a random sample. Random selection does not establish cause and effect. A careful conclusion describes the population the results may represent and avoids unsupported causal language.

A strong short explanation might say: “The 50 households were selected by chance from a current list of households in the service area, so the selection process reduces the risk of staff choosing a preferred group and can support generalizing to households represented by that list. Because 15 selected households did not respond, the association should consider whether their garden use may differ from that of respondents.” This states what random selection contributes and identifies a specific limitation.