Probability Samples and Known Selection Chances

Why the Selection Process Matters

A sample can contain a large number of people and still provide a weak basis for describing a population. The key question is not only how many people were observed, but how they came to be included. If selection follows a known chance process, it is possible to describe how samples could vary and to connect the observed data to the population. If selection chances are unknown, that connection is much harder to justify.

The target population and sampling frame identify who the study aims to describe and the units from which selection can occur. Here the focus is the next question: does the sampling procedure assign each population unit a known chance of selection? A probability sample does; a nonprobability sample does not. Probability sampling does not guarantee that a particular sample will look representative, but it gives a principled basis for quantifying the effects of chance selection.

Inclusion Probabilities and Probability Samples

Definition: For population unit \(i\), the inclusion probability \(\pi_i\) is the probability that unit \(i\) is selected into the sample under the sampling procedure. A probability sample is selected using a chance mechanism for which the inclusion probabilities of the population units are known and positive: \(0 < \pi_i \leq 1\). A nonprobability sample is selected without a chance mechanism that gives known, positive inclusion probabilities for the population units.

The probability refers to the selection procedure, not to whether a unit’s characteristics are uncertain. In a design-based view, the population and its values are fixed for the purpose of the analysis; it is the sample that could differ if the chance selection were repeated. Knowing the procedure lets us reason about that variation.

“Known” does not mean that every unit must have the same chance. One unit might have an inclusion probability of \(0.10\), while another has \(0.40\), provided those chances are specified or can be calculated from the procedure. Nor does “positive” mean “large.” It means that no population unit is ruled out by the selection procedure. A unit with zero chance cannot be represented by the sample through that design.

The word probability is about the method, not a label for a sample that happens to look balanced. A sample chosen informally might resemble the population on a few visible characteristics, but that resemblance does not reveal the selection chances or establish how the sample might vary under repetition.

Worked Example: Classifying a Selection Procedure

A fictional university wants to estimate how many hours its 800 enrolled students studied last week. In Plan A, a computer uses a stated random procedure to select 80 students from a complete list, and every student has a known 0.10 chance of selection. In Plan B, staff ask students who happen to be in the library to complete the same survey. Classify each plan.

Plan A. Each of the 800 students has a known inclusion probability, \(\pi_i=0.10\), and that probability is greater than zero. The selection is made by a chance mechanism, so this is a probability sample.

Plan B. The chance that a particular student is included depends on factors such as whether the student visits the library while staff are present. The procedure does not specify known inclusion probabilities for all 800 students. It is a nonprobability sample, even if staff collect 80 completed surveys.

The plans differ because of how the students were selected, not because one has more responses or because the library sample might happen to include students from several faculties.

How Known Chances Support Inference

Suppose a finite population contains \(N\) units, with a value \(y_i\) for each unit. The population total is \(T=\sum_{i=1}^{N}y_i\). Let \(I_i=1\) if unit \(i\) is selected and \(I_i=0\) otherwise. Then \(P(I_i=1)=\pi_i\). If a unit is not selected, \(I_i y_i/\pi_i=0\); if it is selected, its contribution is adjusted by its inclusion probability.

This leads to an estimator of the population total that uses the known selection chances. The following justification shows a central reason those chances matter.

Result: For a probability sample with known \(\pi_i>0\), the estimator \(\hat{T}=\sum_{i\ \mathrm{in\ sample}}y_i/\pi_i\) has expected value \(T\) over the sampling procedure. In other words, its average over repeated samples generated by that procedure equals the population total.

To see why, write the estimator as a sum over all population units, using the indicator \(I_i\). Taking its expectation gives:

\begin{aligned} \hat{T} &= \sum_{i=1}^{N}\frac{I_i y_i}{\pi_i},\\ E(\hat{T}) &= \sum_{i=1}^{N}\frac{y_i E(I_i)}{\pi_i}\\ &= \sum_{i=1}^{N}\frac{y_i\pi_i}{\pi_i}\\ &= \sum_{i=1}^{N}y_i=T. \end{aligned}

The second line uses \(E(I_i)=P(I_i=1)=\pi_i\), and the next line uses \(\pi_i>0\). This argument does not require different units’ selections to be independent. The formula also shows why chance information is useful: it tells us how to adjust the observed contributions so their expected sum corresponds to the population total.

This result does not say every realized estimate equals the truth. A particular sample can produce an estimate above or below \(T\). Nor does it say the ordinary sample average is appropriate whenever selection chances are known. If chances differ across units, treating every observed unit equally may give disproportionate influence to units that were easier to select. The estimator must reflect the design and the quantity being estimated.

Worked Example: Using Unequal Inclusion Chances

Consider a hypothetical population of four units with fixed values \(y_1=10\), \(y_2=20\), \(y_3=30\), and \(y_4=40\). A specified random procedure includes each unit separately, with inclusion probabilities \(0.50\), \(0.25\), \(0.50\), and \(0.25\), respectively. The population total is \(10+20+30+40=100\). Suppose one possible sample contains units 1 and 3.

For this sample, the weighted estimate of the total is:

\hat{T}=\frac{10}{0.50}+\frac{30}{0.50}=20+60=80.

This particular estimate is 80, not the true total of 100. That is possible: unbiasedness refers to the average estimate across repeated samples, not to a guarantee about an individual sample. For each unit \(i\), the expected contribution is \(\pi_i(y_i/\pi_i)=y_i\). Thus the expected total estimate is \(10+20+30+40=100\).

The different probabilities matter. A selected unit with inclusion probability \(0.25\) receives a larger adjustment than a selected unit with probability \(0.50\), because the lower-probability unit represents more units, on average, under this procedure. In practice, the population values are not all known; the example isolates how the selection probabilities enter the calculation.

What Probability Sampling Does—and Does Not—Establish

Known, positive inclusion probabilities create a formal link between the sample and the defined population. They make it possible to use the selection procedure to justify estimates and assess sampling uncertainty, provided the estimator and uncertainty calculation match the design. For inference about a population mean, for example, one can estimate the total and divide by the known population size \(N\); how to quantify uncertainty depends on further details of the sampling procedure.

Knowing each unit’s inclusion probability is not always enough to calculate the precision of an estimate. How units’ selections relate to one another can affect how much estimates vary across samples. For example, two procedures can give the same individual inclusion probabilities yet produce different patterns of samples. A complete uncertainty calculation must use the relevant features of the sampling design.

Probability sampling also does not automatically remove other problems. A frame may omit some members of the target population. Selected people may not respond, and recorded answers may not accurately measure the intended variable. These problems can create bias or limit the population to which results apply, even when the initial selection was random. Random selection addresses the selection mechanism; it does not repair every stage of data collection.

Worked Example: Random Invitations and Missing Responses

A fictional clinic has a list of 1,000 eligible patients. It uses a chance procedure that gives each patient a known 0.20 chance of being invited to answer a survey about appointment access. Of the 200 invited patients, 130 respond. What can the clinic conclude about the sampling process?

For the invitations: every eligible patient had a known, positive inclusion probability of 0.20. The invited group is a probability sample of the listed patients.

For the respondents: the 130 people who answered are a subset of those invited. Their chances of responding may differ according to their experiences, schedules, or other characteristics. Unless response chances are known or otherwise handled using defensible assumptions, the respondent group cannot simply be treated as a probability sample of all 1,000 patients with the original 0.20 inclusion probability.

The clinic can describe the 130 responses. Generalizing from them to all eligible patients requires attention to nonresponse and the relationship between response and appointment access. Randomly choosing whom to invite does not make the decision to respond random.

Common Mistakes

“The sample is big, so it must represent the population.” Size alone does not reveal how the sample was selected. A large nonprobability sample can systematically miss groups.
“A sample is representative because its observed group percentages look plausible.” Similarity on a few characteristics does not establish known inclusion probabilities or account for unobserved differences.
“Random selection guarantees no bias.” Random selection supports inference about selection variation, but undercoverage, nonresponse, and measurement problems can remain.
“Known probabilities must be equal.” Probability samples can have unequal inclusion probabilities. The analysis may need to account for those differences.
“An unbiased estimator always gives the correct answer.” Unbiasedness describes a long-run average over repeated samples. A particular estimate can still differ from the population value.

Key takeaway: A probability sample is produced by a chance mechanism with known, positive inclusion probabilities for the population units. Those probabilities make design-based population inference possible, but valid conclusions still depend on using an appropriate analysis and addressing coverage, response, and measurement limitations.

Check Your Understanding

Use the distinction between selection procedure and observed sample to answer the questions.

A company randomly selects workers from a complete employee list using a documented procedure that gives each worker a known positive chance of selection. Is this a probability sample? Explain.
A website invites anyone who visits to answer a poll. Why is the poll’s response count not enough to establish known inclusion probabilities for the website’s target population?
In a probability sample, one unit has an inclusion probability of 0.10 and another has an inclusion probability of 0.40. Is unequal selection chance compatible with probability sampling? Why?
What does it mean to say an estimator is unbiased over repeated samples? Does that guarantee the estimate from one sample equals the population value?
A random group is invited to a survey, but many invitees do not respond. Why might conclusions about all invited people still require additional care?