Sampling Designs And Inclusion Probabilities

Why the Sampling Design Matters

A predictive dataset may contain only a fraction of the records in the population it is intended to describe. How that fraction was selected affects what can be inferred from the sample. If every eligible record had the same known chance of selection, the sample can support one kind of population estimate; if records were selected at different rates, the analysis needs to account for those differences. If the selection chances are unknown, a design-based population estimate may not be available at all.

This tutorial focuses on the route from population records to the observed sample: the sampling design and each record’s inclusion probability. It does not assume that the sampled records automatically represent the target population. Instead, it makes explicit which records could be selected and how likely they were to be included. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

The target population and sampling frame need not be identical. For example, a hypothetical analysis might target all active policies during a year but draw its sample from a system extract that omits policies administered on a separate platform. Sampling probabilities describe selection from the frame; they cannot, by themselves, recover records that never entered the frame. That is why defining the population and frame precisely is part of specifying a sampling design.

A probability sample uses a specified random procedure so that the relevant inclusion probabilities can be calculated. In a nonprobability sample, such as one assembled from volunteers or convenient records without a known random selection procedure, those probabilities generally cannot be calculated from the design. A large sample size does not, on its own, supply the missing probabilities.

Common Probability Sampling Designs

Different designs give records different routes into the sample. The central calculation is always the chance that a particular record is selected, but the calculation depends on the actual stages and rules of selection.

Strata are groups defined before sampling, with sampling performed within each group. The number sampled in each stratum can be proportional to its population size or deliberately disproportionate. In the latter case, records in some strata have higher inclusion probabilities than others. Cluster sampling instead selects groups as the first sampling stage. If the design then selects records within each chosen cluster, a record must pass both selection stages to be included.

For systematic sampling, the statement about probability \(1/k\) assumes a frame size divisible by \(k\), a uniformly selected random start among the \(k\) possible starting positions, and selection every \(k\)th record. The ordering of the frame still matters: a pattern in the ordering that aligns with the interval can make the realized sample unusual. The inclusion probability alone does not describe every feature of a design.

Using Inclusion Probabilities to Estimate a Total

Suppose a finite population contains \(N\) records with fixed values \(y_1,\ldots,y_N\), and the sampling design gives every record a known positive inclusion probability \(\pi_i\). Let \(I_i=1\) if record \(i\) is included and \(I_i=0\) otherwise. The inverse-probability estimator of the population total is the Horvitz–Thompson estimator:

The weight \(1/\pi_i\) is larger for a record that was less likely to be selected. The estimator adds each observed value after scaling it to account for its selection chance. When the population size \(N\) is known, dividing \(\widehat{T}_{HT}\) by \(N\) estimates the population mean. These design-based results use the specified random selection mechanism; they do not require the fixed record values themselves to follow a particular probability distribution.

The expectation is taken over repeated samples drawn using the same design, while the population values \(y_i\) remain fixed. Therefore,

This establishes that the estimator is unbiased over repeated samples from the design. It does not mean that a particular sample’s estimate must equal the true total. Nor does it establish that a sample drawn from an incomplete frame estimates a target population outside that frame without additional assumptions. The proof depends on each target record being represented in the frame and having a known positive probability of selection.

For uncertainty calculations, the probabilities that two records are included together can also matter. Define \(\pi_{ij}=P(i\text{ and }j\text{ are both included})\). In general, records’ inclusion events need not be independent, so knowing only the individual \(\pi_i\) values is not enough to describe all the sampling dependence or calculate every design-based variance.

Multistage Selection and Unknown Probabilities

In a multistage design, a record’s inclusion probability is the probability of passing every selection stage required for inclusion. For example, if a clinic is selected first and a patient is then sampled within that clinic, the patient’s probability is the clinic-selection probability multiplied by the conditional probability of patient selection, provided that the second-stage probability is defined conditional on the clinic being selected. Omitting a stage gives the wrong record-level probability and therefore the wrong weight.

The next example illustrates this calculation. Its payment amounts and population are hypothetical.

A convenience sample or a sample assembled through volunteers is different: unless a known selection mechanism supplies the inclusion probabilities, weights cannot be calculated by simply taking their reciprocals. A post hoc guess at each record’s selection chance is not the same as a probability specified by the sampling design. Claims about population totals based on such a sample require other methods and assumptions; the design-based calculation above cannot be justified by the sample size alone.