Why Inclusion Probabilities Matter
A sample is not simply a smaller version of a population. The method used to select it determines which units could be observed and how likely each unit was to be included. Those chances matter when interpreting a sample statistic as evidence about a population: a sample that deliberately includes many rare units may be useful, but its raw average need not describe the population average.
The key is to describe the sampling design before judging the sample. Inclusion probabilities summarize how that design treats individual units. They also help separate two questions that are often confused: Was the sample selected in a way that supports inference about the target population? And, for a particular variable, does the realized sample provide a good account of the population quantity of interest?
The target population is the set of units about which the study aims to make an inference. The design is the rule, including any random selection, that produces the sample. An inclusion probability therefore depends on both: changing the target population or the selection procedure can change \(\pi_i\). A probability of selection is not the same thing as a probability of participation after selection; participation is a separate issue.
Equal, Unequal, and Zero Probabilities
In a simple random sample of a fixed size \(n\) from a population of \(N\) units, every unit has inclusion probability \(n/N\). Other designs intentionally give units different chances. For example, a survey might sample a larger fraction of a small region to obtain enough observations there, or use a size-related selection rule that gives larger units higher chances. These are not automatically flaws. The important question is whether the design and subsequent analysis are appropriate for the target quantity.
A unit has a zero inclusion probability if the design makes it impossible for that unit to enter the sample. This can happen when a target-population unit is absent from the sampling frame, or when the design excludes a group. A zero probability differs fundamentally from a small positive probability: collecting a larger sample under the same design still cannot select a unit with \(\pi_i=0\).
Positive but unequal inclusion probabilities mean that some observations represent more population units than others in a design-based calculation. When probabilities are known, inverse-probability weights \(1/\pi_i\) can account for this unequal selection. These weights do not make every realized sample identical to the population, nor do they repair zero inclusion probabilities without additional assumptions.
Worked Example: Unequal Chances and a Weighted Estimate
Suppose a hypothetical target population has 1,000 households: 800 in metropolitan areas and 200 in remote areas. A design selects 160 metropolitan households and 100 remote households by simple random sampling within each group. Find the group-specific inclusion probabilities and compare the unweighted sample mean of a hypothetical income measure with a population-share-weighted estimate.
Within the metropolitan group, each household’s probability is \(160/800=0.20\). Within the remote group, it is \(100/200=0.50\). Thus remote households are sampled at a higher rate. The sample contains \(160/260\), or about 61.54%, metropolitan households, even though they make up 80% of the target population.
Suppose the observed sample mean income is 52 thousand dollars in metropolitan areas and 40 thousand dollars in remote areas. The unweighted mean across the 260 sampled households is
To estimate the target-population mean while respecting the population group sizes, use the sample group means with the population shares:
The value 49.60 thousand dollars is a design-weighted estimate, not a known population mean: the group means came from samples and can differ from the actual group means. The example shows why the raw sample mean can give disproportionate influence to a group that was intentionally sampled at a higher rate.
What Representativeness Means
“Representative” is not a single numerical property that a sample either has or lacks. It is a judgment relative to a target population, a particular feature or estimand, and an intended use. A sample can match the population’s regional composition but differ substantially in income composition. It can estimate a population mean of one variable well while giving a poor estimate of another.
Selection design and realized sample are related but not identical. Inclusion probabilities describe the design before the sample is drawn. Even under a sound probability design, a particular sample can differ from the population by chance. Conversely, one realized sample may happen to match a population feature closely even if the design offers weak grounds for expecting that match in repeated samples.
Representativeness also does not mean that each individual in the sample resembles a typical individual. It concerns whether the sample and its analysis support the intended population description. For example, a sample deliberately balanced across regions may not reproduce the population’s regional proportions, yet it can still support a population estimate if the analysis correctly accounts for that design.
Worked Example: A Zero-Probability Group
Suppose a hypothetical study targets 1,000 households, but its frame contains only 800 registered households. A simple random sample of 80 registered households is drawn, and 90% of those sampled report having a particular service. The 200 households outside the frame have zero inclusion probability under this design. For illustration, suppose an analyst uses 10% as an assumed service rate for those unregistered households. What overall rate does this information suggest, and what does it establish?
Each registered household has inclusion probability \(80/800=0.10\). The observed 90% is the sample rate among the registered households; it is not automatically their true population rate. Combining that sample rate with the assumed 10% rate outside the frame gives the sample-based estimate
Thus 74% is an estimate under the stated calculation, not an established actual rate for all households. The sample rate among registered households is subject to sampling variation, and the 10% rate for unregistered households is an assumption here. Because those 200 households have zero inclusion probability, the design alone cannot estimate their service rate. Generalizing to the full target population requires additional information or assumptions about them.
Variable-Specific Representativeness
A sample’s composition is one feature of its representativeness, but it is not a substitute for examining the variable or estimand that matters. The next example contrasts two measured variables in the same hypothetical sample. It assumes that population values are available for illustration; in an actual study, researchers generally do not know those population summaries and must estimate them.
Worked Example: Good Balance for One Variable, Not Another
Consider a hypothetical population of 100 people, half in group A and half in group B. A sample includes 10 people from each group, so its group composition matches the population. The following table gives the population means and the realized sample means for two measured variables, \(Y\) and \(Z\).
| Group | Population count | Sample count | Population mean of \(Y\) | Sample mean of \(Y\) | Population mean of \(Z\) | Sample mean of \(Z\) |
|---|---|---|---|---|---|---|
| A | 50 | 10 | 4 | 4 | 2 | 8 |
| B | 50 | 10 | 6 | 6 | 2 | 2 |
The population mean of \(Y\) is \([50(4)+50(6)]/100=5\). The sample mean of \(Y\) is \([10(4)+10(6)]/20=5\), so this realized sample matches the population mean of \(Y\). For \(Z\), the population mean is \([50(2)+50(2)]/100=2\), whereas the sample mean is \([10(8)+10(2)]/20=5\). Matching the population’s group proportions has not made the sample mean representative for \(Z\).
The example does not show that the sample design must fail. A sample can match one variable and not another, by chance or because selection relates to characteristics associated with a measured variable. It shows why a claim of representativeness should specify the feature being discussed rather than treating it as a universal label.
A Design-Based Result for Positive Probabilities
For a finite population with fixed values \(y_i\), let \(I_i\) equal 1 if unit \(i\) is sampled and 0 otherwise. If every unit has a known positive inclusion probability, the inverse-probability estimator of the population total is \(\hat{T}=\sum_{i\in U} I_i y_i/\pi_i\). Its key property follows directly from the definition of inclusion probability.
To verify the result, take expectation over repeated samples generated by the design. Since \(E(I_i)=P(i\in S)=\pi_i\), linearity of expectation gives
This is a repeated-sampling statement: averaging the estimator over samples drawn under the design gives the finite-population total. It does not say that a particular estimate equals the total, or that the observed sample is balanced on every variable. The positivity condition is essential. If \(\pi_i=0\), the term \(I_i y_i/\pi_i\) is undefined, and no design-based weighting of the observed sample can recover that unit’s contribution without further information or assumptions.
Common Mistakes and Practical Checks
- Calling unequal probabilities automatically biased. Unequal chances are a feature of some valid designs. The concern is whether estimation accounts for them when targeting population quantities.
- Treating a weighted estimate as the known population truth. A design-weighted estimate uses observed sample values and remains subject to sampling variability.
- Assuming a large sample solves zero coverage. If a group has \(\pi_i=0\), increasing the sample size under the same design still cannot select it.
- Calling a sample representative without naming the target. State the target population and the variable, feature, or estimand for which representativeness is being considered.
- Assuming demographic balance guarantees balance on every outcome. Matching group shares does not ensure that the sample matches population values of income, employment, or other variables within those groups.
When evaluating a population claim, identify the target units, describe how the sample was selected, and ask whether every target unit had a positive chance of selection. Then specify the variable or estimand at issue and distinguish observed sample summaries from population quantities. These checks clarify both what the design supports and where inference depends on assumptions.
Check Your Understanding
Answer each question by referring to the target population and the sampling design.
- A design selects 30 of 150 units in one stratum and 20 of 40 in another by simple random sampling within strata. What is each stratum’s inclusion probability, and which stratum is sampled at the higher rate?
- In your own words, why does a zero inclusion probability differ from a small positive inclusion probability?
- A sample has the same regional proportions as the target population. Does that establish representativeness for every measured variable? Explain.
- What does design-unbiasedness of an inverse-probability estimate say, and what does it not say about a particular realized sample?
- A sample rate is 90% for a covered group, and an analyst assumes a 10% rate for an uncovered group. Why should the resulting overall rate be described as an estimate rather than an established population fact?